mds: add new performance and subvolume utilization metrics by salieri11 · Pull Request #66551 · ceph/ceph

salieri11 · 2025-12-08T12:58:29Z

Summary

This PR adds new observability metrics for CephFS MDS:

MDS Rank Performance Metrics - Per-rank CPU utilization and open requests count
Subvolume Utilization Metrics - Quota limit and current space usage for subvolumes

Fixes: https://tracker.ceph.com/issues/73700
Fixes: https://tracker.ceph.com/issues/74135

Changes

New per-rank counters exposed via mds_rank_perf labeled perf counters:

Metric	Description
`cpu_usage`	Sum of per-core CPU utilization (100 = one fully saturated core; can exceed 100 on multi-core)
`open_requests`	Number of metadata requests currently in flight

These metrics are sampled periodically and aggregated by rank 0 for cluster-wide visibility.

Subvolume Utilization Metrics

New fields added to subvolume metrics:

Metric	Description
`quota_bytes`	Configured quota limit in bytes (0 if no quota/unlimited)
`used_bytes`	Current space usage based on recursive inode statistics (`rstat.rbytes`)

Key implementation details:

Quota info is cached in MetricsHandler::subvolume_quota map, updated via MDCache::broadcast_quota_to_client
Cache entries are evicted after 2 × subv_metrics_window_interval of inactivity to prevent unbounded memory growth
used_bytes is sourced from cached broadcast values with fallback to dynamic rstat fetch

Important behavioral note:

Only data I/O (reads/writes to file contents) triggers metric updates
Metadata operations (mkdir, rmdir, unlink, rename, chmod, etc.) do NOT generate I/O metrics
After file deletions, used_bytes updates on next data I/O or quota broadcast

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins test classic perf Jenkins Job | Jenkins Job Definition
jenkins test crimson perf Jenkins Job | Jenkins Job Definition
jenkins test signed Jenkins Job | Jenkins Job Definition
jenkins test make check Jenkins Job | Jenkins Job Definition
jenkins test make check arm64 Jenkins Job | Jenkins Job Definition
jenkins test submodules Jenkins Job | Jenkins Job Definition
jenkins test dashboard Jenkins Job | Jenkins Job Definition
jenkins test dashboard cephadm Jenkins Job | Jenkins Job Definition
jenkins test api Jenkins Job | Jenkins Job Definition
jenkins test docs ReadTheDocs | Github Workflow Definition
jenkins test ceph-volume all Jenkins Jobs | Jenkins Jobs Definition
jenkins test windows Jenkins Job | Jenkins Job Definition
jenkins test rook e2e Jenkins Job | Jenkins Job Definition

You must only issue one Jenkins command per-comment. Jenkins does not understand
comments with more than one command.

salieri11 · 2025-12-08T18:23:56Z

jenkins test make check

salieri11 · 2025-12-09T08:00:38Z

jenkins test make check arm64

doc/cephfs/metrics.rst

salieri11 · 2025-12-22T12:41:08Z

jenkins test windows

salieri11 · 2025-12-30T10:14:14Z

jenkins test windows

salieri11 · 2025-12-30T14:03:27Z

jenkins test api

salieri11 · 2026-01-14T12:37:18Z

jenkins test api

doc/cephfs/metrics.rst

src/common/TrackedOp.cc

src/mds/MDSRank.cc

src/mds/MDBalancer.cc

src/mds/MDCache.cc

vshankar

Dropping some more comments. I'm still reviewing this closely and would be done by tomorrow.

src/mds/MDCache.cc

src/mds/MetricsHandler.cc

src/mds/MetricAggregator.cc

vshankar

Minor nits. Otherwise LGTM and would be ready to run through QA tests once the nits are addressed. Nice work @salieri11

src/mds/MDBalancer.cc

src/mds/MDCache.cc

qa/tasks/cephfs/test_mds_metrics.py

vshankar · 2026-02-19T13:10:35Z

jenkins retest this please

vshankar · 2026-02-20T08:51:13Z

jenkins test make check

anthonyeleven · 2026-02-20T13:37:49Z

Nice. Can we expect a backport to Tentacle or Squid?

vshankar · 2026-02-22T10:18:00Z

Nice. Can we expect a backport to Tentacle or Squid?

Sorry! This is based on a biggish feature that will be available for Umbrella.

vshankar · 2026-02-22T10:19:31Z

This PR is under test in https://tracker.ceph.com/issues/75073.

Perf merics: CPU% and number of open requests Subvolume utilization metrics: quota info and current size Signed-off-by: Igor Golikov <igolikov@redhat.com> Fixes: https://tracker.ceph.com/issues/74135 Fixes: https://tracker.ceph.com/issues/73700

Signed-off-by: Igor Golikov <igolikov@redhat.com> Fixes: https://tracker.ceph.com/issues/74135

test for CPU utilizationa and number of open requests Signed-off-by: Igor Golikov <igolikov@redhat.com> Fixes: https://tracker.ceph.com/issues/73700

Add comperehensive tests to validate correct quota and current size metrics for subvolumes Signed-off-by: Igor Golikov <igolikov@redhat.com> Fixes: https://tracker.ceph.com/issues/74135

docs for subvolume utilization and MDS perf metrics Signed-off-by: Igor Golikov <igolikov@redhat.com> Fixes: https://tracker.ceph.com/issues/74135 Fixes: https://tracker.ceph.com/issues/73700

…ifecycle() Signed-off-by: Venky Shankar <vshankar@redhat.com>

* refs/pull/66551/head:

vshankar · 2026-02-25T07:06:32Z

added a fix to use safe_while to avoid racy check and squashed commits.

vshankar · 2026-02-25T07:58:42Z

jenkins test make check arm64

vshankar · 2026-02-25T09:15:22Z

jenkins test make check

vshankar · 2026-02-25T12:28:33Z

force pushed -- no changes

vshankar

https://tracker.ceph.com/projects/cephfs/wiki/QA_main_--_2026#wip-vshankar-testing-20260226041846

github-actions · 2026-03-03T10:57:25Z

This is an automated message by src/script/redmine-upkeep.py.

I have resolved the following tracker ticket due to the merge of this PR:

https://tracker.ceph.com/issues/74135

No backports are pending for the ticket. If this is incorrect, please update the tracker
ticket and reset to Pending Backport state.

Update Log: https://github.com/ceph/ceph/actions/runs/22619802892

github-actions · 2026-03-03T10:57:50Z

This is an automated message by src/script/redmine-upkeep.py.

I have resolved the following tracker ticket due to the merge of this PR:

https://tracker.ceph.com/issues/73700

No backports are pending for the ticket. If this is incorrect, please update the tracker
ticket and reset to Pending Backport state.

Update Log: https://github.com/ceph/ceph/actions/runs/22619802892

github-actions bot added cephfs Ceph File System common core documentation mgr tests labels Dec 8, 2025

salieri11 marked this pull request as ready for review December 8, 2025 13:00

salieri11 requested review from a team as code owners December 8, 2025 13:00

salieri11 force-pushed the wip-igolikov-new-metrics-74135-73700 branch from a298bcc to 0e296e4 Compare December 9, 2025 09:34

anthonyeleven reviewed Dec 9, 2025

View reviewed changes

doc/cephfs/metrics.rst Outdated Show resolved Hide resolved

sumabai added the wip-suma-testing upstream new feature testing label Dec 16, 2025

kdhaduk81 added the wip-kdhaduk-testing-2 label Dec 16, 2025

salieri11 force-pushed the wip-igolikov-new-metrics-74135-73700 branch from 0e296e4 to a589dcf Compare December 17, 2025 13:33

salieri11 force-pushed the wip-igolikov-new-metrics-74135-73700 branch from a589dcf to 9e64628 Compare December 30, 2025 17:46

salieri11 force-pushed the wip-igolikov-new-metrics-74135-73700 branch 2 times, most recently from 1589259 to 6ec5eb9 Compare January 14, 2026 10:57

cloudbehl reviewed Jan 23, 2026

View reviewed changes

doc/cephfs/metrics.rst Show resolved Hide resolved

vshankar assigned vshankar and kotreshhr Feb 3, 2026

vshankar reviewed Feb 3, 2026

View reviewed changes

src/common/TrackedOp.cc Show resolved Hide resolved

vshankar reviewed Feb 9, 2026

View reviewed changes

src/mds/MDSRank.cc Outdated Show resolved Hide resolved

vshankar reviewed Feb 9, 2026

View reviewed changes

src/mds/MDBalancer.cc Outdated Show resolved Hide resolved

vshankar reviewed Feb 9, 2026

View reviewed changes

src/mds/MDCache.cc Show resolved Hide resolved

cloudbehl mentioned this pull request Feb 9, 2026

monitoring: Add subvolume dashboard #66802

Merged

14 tasks

vshankar requested changes Feb 9, 2026

View reviewed changes

src/mds/MDCache.cc Outdated Show resolved Hide resolved

src/mds/MetricsHandler.cc Show resolved Hide resolved

src/mds/MetricsHandler.cc Outdated Show resolved Hide resolved

src/mds/MetricAggregator.cc Outdated Show resolved Hide resolved

vshankar reviewed Feb 17, 2026

View reviewed changes

src/mds/MDBalancer.cc Outdated Show resolved Hide resolved

src/mds/MDCache.cc Outdated Show resolved Hide resolved

qa/tasks/cephfs/test_mds_metrics.py Outdated Show resolved Hide resolved

qa/tasks/cephfs/test_mds_metrics.py Outdated Show resolved Hide resolved

vshankar added the wip-vshankar-testing3 label Feb 19, 2026

Igor Golikov and others added 6 commits February 24, 2026 20:28

mgr: add subvolume quota metrics to the manager

c8f2964

Signed-off-by: Igor Golikov <igolikov@redhat.com> Fixes: https://tracker.ceph.com/issues/74135

test: add test for new MDS perf metrics

90cfbfd

test for CPU utilizationa and number of open requests Signed-off-by: Igor Golikov <igolikov@redhat.com> Fixes: https://tracker.ceph.com/issues/73700

test: add tests for subvolume utilization metrics

b48a923

Add comperehensive tests to validate correct quota and current size metrics for subvolumes Signed-off-by: Igor Golikov <igolikov@redhat.com> Fixes: https://tracker.ceph.com/issues/74135

docs: update docs with new metrics description

89e3ab0

docs for subvolume utilization and MDS perf metrics Signed-off-by: Igor Golikov <igolikov@redhat.com> Fixes: https://tracker.ceph.com/issues/74135 Fixes: https://tracker.ceph.com/issues/73700

qa: use safe_while() to avoid racy checks in test_subvolume_metrics_l…

2850b3f

…ifecycle() Signed-off-by: Venky Shankar <vshankar@redhat.com>

vshankar added a commit to vshankar/ceph that referenced this pull request Feb 25, 2026

Merge PR ceph#66551 into wip-vshankar-testing-20260222.101816

b1cce97

* refs/pull/66551/head:

vshankar force-pushed the wip-igolikov-new-metrics-74135-73700 branch from a09dbb5 to 8b9eddf Compare February 25, 2026 07:06

vshankar force-pushed the wip-igolikov-new-metrics-74135-73700 branch from 8b9eddf to 2850b3f Compare February 25, 2026 12:28

vshankar approved these changes Mar 3, 2026

View reviewed changes

vshankar merged commit d962d5e into ceph:main Mar 3, 2026
13 checks passed

Conversation

salieri11 commented Dec 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Subvolume Utilization Metrics

Contribution Guidelines

Checklist

Uh oh!

salieri11 commented Dec 8, 2025

Uh oh!

salieri11 commented Dec 9, 2025

Uh oh!

Uh oh!

salieri11 commented Dec 22, 2025

Uh oh!

salieri11 commented Dec 30, 2025

Uh oh!

salieri11 commented Dec 30, 2025

Uh oh!

salieri11 commented Jan 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vshankar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vshankar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vshankar commented Feb 19, 2026

Uh oh!

vshankar commented Feb 20, 2026

Uh oh!

anthonyeleven commented Feb 20, 2026

Uh oh!

vshankar commented Feb 22, 2026

Uh oh!

vshankar commented Feb 22, 2026

Uh oh!

vshankar commented Feb 25, 2026

Uh oh!

vshankar commented Feb 25, 2026

Uh oh!

vshankar commented Feb 25, 2026

Uh oh!

vshankar commented Feb 25, 2026

Uh oh!

vshankar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Mar 3, 2026

Uh oh!

github-actions bot commented Mar 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

salieri11 commented Dec 8, 2025 •

edited

Loading