Labeled Perf Counters #48657

alimaredia · 2022-10-28T12:38:37Z

Labeled Perf Counters initial work.

This PR adds labels to perf counters allowing for multi-dimensional perf counters to be stored and dumped in the form

"rgw": {
    "labels": {
        "Bucket: "bkt1",
        "User: "user1",
    },
    "put_b": 1048576,
    "put_initial_lat": {
        "avgcount": 1,
        "sum": 0.013000200,
        "avgtime": 0.013000200
    }
},

Signed-off-by: Ali Maredia amaredia@redhat.com

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

src/vstart.sh

cbodley · 2022-12-08T16:29:56Z

@alimaredia setting aside the cache part for now, can you explain the strategy for converting existing counters to use labels?

for example, rgw's multisite sync counters put the source-zone in the counter name at https://github.com/ceph/ceph/blob/bc01d2ed/src/rgw/rgw_rados.cc#L478:

      counters(sync_counters::build(store->ctx(), std::string("data-sync-from-") + source_zone->name)),

how would we change that to use a "source={zone name}" label?

src/common/perf_counters.cc

src/common/perf_counters.h

src/common/perf_counters_cache_key.cc

src/common/ceph_context.cc

alimaredia · 2022-12-08T21:17:00Z

@cbodley you would just need to change the string argument std::string("data-sync-from-") + source_zone->name to one in the labeled format like what you get in perf_counters_cache_key(). The PerfCounter created here goes into the same PerfCountersCollection the rest of labeled perf counters do.

You would also need to need to pass true into the last argument of the PerfCountersBuilder() constructor that is called in build() to mark the perf counter as labeled. That being said, we talked earlier about having all PerfCounters being labeled, so if that differentiation goes away then that bool would go away.

github-actions · 2022-12-14T20:21:30Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

alimaredia · 2023-01-05T20:07:07Z

jenkins test make check

alimaredia · 2023-01-06T03:50:27Z

jenkins test make check

alimaredia · 2023-02-15T20:46:41Z

@ljflores @yuriw: Did this PR get run as part of a job yet?

ljflores · 2023-02-15T23:04:30Z

@alimaredia not yet, but I expect to get it in soon. All of Yuri's labels are currently taken up (he prefers to have up to 12 batches going at a time). But we just approved a batch so will have space to start a new batch soon

ljflores · 2023-02-15T23:05:14Z

You'll see a "wip-yuri*-testing" label added to the PR when it's been batched.

ljflores · 2023-02-16T16:25:41Z

@alimaredia it's going into testing now

alimaredia · 2023-02-17T21:04:13Z

@ljflores awesome let me know how it goes!

ljflores · 2023-02-17T21:51:49Z

@alimaredia I'll have results for you next week

ljflores · 2023-02-22T17:09:33Z

Update on testing: A different PR in the testing batch caused a regression. To verify, I am dropping it from the batch and rerunning a few tests. This PR did not seem at all related, so I am working to unblock it.

github-actions · 2023-02-23T13:14:13Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

a flat representation of a set of prometheus labels, returned as a std::string. this string can either be used for sorting an ordered container of perf counters, or for hashing an unordered container Signed-off-by: Casey Bodley <cbodley@redhat.com>

Add the ability to dump labeled perf counters for a daemon. Labeled perf counters are stored in a CephContext's PerfCountersCollection. Labeled and unlabeled perf counters are dumped to the admin socket via `counters dump` command. The schema for labeled and unlabeled perf counters are dumped to the admin socket via `counters schema` command. This commit includes docs and additional unit tests Signed-off-by: Ali Maredia <amaredia@redhat.com>

idryomov

A rebase was needed due to Filestore removal. src/os/filestore/FileStore.h hunk was dropped as the file is gone entirely:

diff --git a/src/os/filestore/FileStore.h b/src/os/filestore/FileStore.h
index 67caeaec0a01e..e8a1714217be5 100644
--- a/src/os/filestore/FileStore.h
+++ b/src/os/filestore/FileStore.h
@@ -509,7 +509,7 @@ class FileStore : public JournalingObjectStore,
 
   void dump_perf_counters(ceph::Formatter *f) override {
     f->open_object_section("perf_counters");
-    logger->dump_formatted(f, false);
+    logger->dump_formatted(f, false, false);
     f->close_section();
   }

alimaredia · 2023-02-27T17:34:15Z

@idryomov thank you for the clarification around my recent rebase

ljflores · 2023-02-27T18:50:04Z

Rados suite review: https://pulpito.ceph.com/yuriw-2023-02-16_22:44:43-rados-wip-yuri-testing-2023-02-16-0839-distro-default-smithi
https://pulpito.ceph.com/lflores-2023-02-20_21:22:20-rados-wip-yuri-testing-2023-02-16-0839-distro-default-smithi
https://pulpito.ceph.com/yuriw-2023-02-23_16:42:54-rados-wip-yuri-testing-2023-02-22-2037-distro-default-smithi
https://pulpito.ceph.com/lflores-2023-02-23_17:54:36-rados-wip-yuri-testing-2023-02-22-2037-distro-default-smithi

Failures, unrelated:
1. https://tracker.ceph.com/issues/58585
2. https://tracker.ceph.com/issues/58560
3. https://tracker.ceph.com/issues/58496
4. https://tracker.ceph.com/issues/49961
5. https://tracker.ceph.com/issues/58861
6. https://tracker.ceph.com/issues/58797
7. https://tracker.ceph.com/issues/49428

Details:
1. rook: failed to pull kubelet image - Ceph - Orchestrator
2. test_envlibrados_for_rocksdb.sh failed to subscribe to repo - Infrastructure
3. osd/PeeringState: FAILED ceph_assert(!acting_recovery_backfill.empty()) - Ceph - RADOS
4. scrub/osd-recovery-scrub.sh: TEST_recovery_scrub_1 failed - Ceph - RADOS
5. OSError: cephadm config file not found - Ceph - Orchestrator
6. scrub/osd-scrub-dump.sh: TEST_recover_unexpected fails from "ERROR: Unexpectedly low amount of scrub reservations seen during test" - Ceph - RADOS
7. ceph_test_rados_api_snapshots fails with "rados_mon_command osd pool create failed with error -22" - Ceph - RADOS

ljflores · 2023-02-27T18:51:01Z

From RADOS point of view, it looks good.

ljflores · 2023-02-27T18:53:05Z

@idryomov @cbodley good to merge?

idryomov · 2023-02-27T19:05:42Z

@idryomov @cbodley good to merge?

I'd say so, thanks for handling the RADOS suite!

ljflores · 2023-02-27T19:20:08Z

Thanks for waiting while we got lab issues sorted!

idryomov · 2023-03-31T08:45:47Z

@jmolmo This is already in reef and not getting backported to quincy.

github-actions bot added build/ops common rgw tests labels Oct 28, 2022

alimaredia added the do-not-merge label Oct 28, 2022

alimaredia changed the title ~~Wip rgw labeled perf counters cache~~ [DRAFT] Labeled Perf Counters Oct 28, 2022

alimaredia commented Nov 9, 2022

View reviewed changes

src/vstart.sh Outdated Show resolved Hide resolved

avanthakkar added the wip-athakkar-testing label Dec 1, 2022

alimaredia changed the title ~~[DRAFT] Labeled Perf Counters~~ Labeled Perf Counters Dec 2, 2022

alimaredia requested a review from a team as a code owner December 4, 2022 20:50

github-actions bot added the documentation label Dec 4, 2022

alimaredia force-pushed the wip-rgw-labeled-perf-counters-cache branch 2 times, most recently from cc8a42a to 9f1a01c Compare December 6, 2022 21:45

jmolmo requested review from cbodley and removed request for cbodley December 7, 2022 14:54

cbodley reviewed Dec 8, 2022

View reviewed changes

alimaredia requested a review from a team as a code owner December 13, 2022 21:20

github-actions bot added crimson needs-rebase labels Dec 13, 2022

avanthakkar removed the wip-athakkar-testing label Dec 15, 2022

alimaredia force-pushed the wip-rgw-labeled-perf-counters-cache branch from 7657a4b to 846c420 Compare December 20, 2022 16:03

github-actions bot removed the needs-rebase label Dec 20, 2022

alimaredia force-pushed the wip-rgw-labeled-perf-counters-cache branch from afb51b4 to 38107e9 Compare December 20, 2022 16:50

alimaredia removed the do-not-merge label Jan 6, 2023

alimaredia force-pushed the wip-rgw-labeled-perf-counters-cache branch from 72b4e76 to c8920c1 Compare January 6, 2023 04:35

ljflores added the wip-yuri-testing label Feb 16, 2023

github-actions bot added the needs-rebase label Feb 23, 2023

alimaredia force-pushed the wip-rgw-labeled-perf-counters-cache branch from 93c8675 to 56e2a10 Compare February 23, 2023 17:03

github-actions bot removed the needs-rebase label Feb 23, 2023

cbodley and others added 2 commits February 23, 2023 12:04

idryomov approved these changes Feb 26, 2023

View reviewed changes

idryomov merged commit b487144 into ceph:main Feb 27, 2023

jmolmo added needs-reef-backport needs-quincy-backport backport required for quincy labels Mar 27, 2023

idryomov removed needs-quincy-backport backport required for quincy needs-reef-backport labels Mar 31, 2023

This was referenced Mar 31, 2023

quincy: rbd-mirror: switch to labeled perf counters #50689

Closed

exporter: add support for exposing labeled perf counters #50369

Merged

rbd-mirror: switch to labeled perf counters #50302

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Labeled Perf Counters #48657

Labeled Perf Counters #48657

alimaredia commented Oct 28, 2022 •

edited

Loading

cbodley commented Dec 8, 2022

alimaredia commented Dec 8, 2022

github-actions bot commented Dec 14, 2022

alimaredia commented Jan 5, 2023

alimaredia commented Jan 6, 2023

alimaredia commented Feb 15, 2023

ljflores commented Feb 15, 2023

ljflores commented Feb 15, 2023

ljflores commented Feb 16, 2023

alimaredia commented Feb 17, 2023

ljflores commented Feb 17, 2023

ljflores commented Feb 22, 2023 •

edited

Loading

github-actions bot commented Feb 23, 2023

idryomov left a comment

alimaredia commented Feb 27, 2023

ljflores commented Feb 27, 2023

ljflores commented Feb 27, 2023

ljflores commented Feb 27, 2023

idryomov commented Feb 27, 2023

ljflores commented Feb 27, 2023

idryomov commented Mar 31, 2023

Labeled Perf Counters #48657

Labeled Perf Counters #48657

Conversation

alimaredia commented Oct 28, 2022 • edited Loading

Contribution Guidelines

Checklist

cbodley commented Dec 8, 2022

alimaredia commented Dec 8, 2022

github-actions bot commented Dec 14, 2022

alimaredia commented Jan 5, 2023

alimaredia commented Jan 6, 2023

alimaredia commented Feb 15, 2023

ljflores commented Feb 15, 2023

ljflores commented Feb 15, 2023

ljflores commented Feb 16, 2023

alimaredia commented Feb 17, 2023

ljflores commented Feb 17, 2023

ljflores commented Feb 22, 2023 • edited Loading

github-actions bot commented Feb 23, 2023

idryomov left a comment

Choose a reason for hiding this comment

alimaredia commented Feb 27, 2023

ljflores commented Feb 27, 2023

ljflores commented Feb 27, 2023

ljflores commented Feb 27, 2023

idryomov commented Feb 27, 2023

ljflores commented Feb 27, 2023

idryomov commented Mar 31, 2023

alimaredia commented Oct 28, 2022 •

edited

Loading

ljflores commented Feb 22, 2023 •

edited

Loading