Feat/cluster metadata manifest age metric #17404
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
andijcr
force-pushed
the
feat/cluster_metadata_manifest_age_metric
branch
from
0e97f85
to
6561ddd
Compare
|
new failures in https://buildkite.com/redpanda/redpanda/builds/46874#018e7fd6-7ef4-41a6-8262-7230c841fe62: new failures in https://buildkite.com/redpanda/redpanda/builds/46874#018e7fd6-7ef9-4b86-bda0-1ba5447f32ed: new failures in https://buildkite.com/redpanda/redpanda/builds/46874#018e7fd6-7efc-49a9-b41f-1d6b69d03a5b: new failures in https://buildkite.com/redpanda/redpanda/builds/46874#018e7fd6-7ef7-4b95-8ec4-03159352ac39: new failures in https://buildkite.com/redpanda/redpanda/builds/46874#018e7fe8-c3f8-419c-acbf-8fd97656ed7a: new failures in https://buildkite.com/redpanda/redpanda/builds/46874#018e7fe8-c3f5-4b5d-9ed6-473939561df9: new failures in https://buildkite.com/redpanda/redpanda/builds/46874#018e7fe8-c3fb-4b08-bb65-53b6c166ee34: new failures in https://buildkite.com/redpanda/redpanda/builds/46874#018e7fe8-c3fe-493c-8257-62a4f1562008: new failures in https://buildkite.com/redpanda/redpanda/builds/46914#018e82ae-5f15-413b-b330-46f460aa8919: |
|
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46874#018e7fe8-c3fb-4b08-bb65-53b6c166ee34 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46914#018e82ae-5f15-413b-b330-46f460aa8919 |
andrwng
reviewed
Mar 27, 2024
src/v/cluster/controller_probe.cc
Outdated
| auto maybe_manifest_ref | ||
| = _controller.metadata_uploader().manifest(); | ||
| if (!maybe_manifest_ref.has_value()) { | ||
| return int64_t{-1}; |
There was a problem hiding this comment.
I think 0 will be a more natural value to aggregate over.
There was a problem hiding this comment.
Let's also check if the manifest's timestamp is 0, and return 0 then too. It indicates we've never uploaded.
src/v/cluster/controller_probe.cc
Outdated
Comment on lines
116
to
119
|
|
||
| auto age_s = std::chrono::duration_cast<std::chrono::seconds>( | ||
| now_ts - manifest.upload_time_since_epoch); | ||
| return int64_t{age_s.count()}; |
There was a problem hiding this comment.
If you're looking for a place to test this, I think cluster/cloud_metadata/tests/uploader_test.cc would be a good place. The full application is in that test, so we should have metrics.
field _metadata_uploader is an unique_ptr that gets initialized with a value only if cloud storage is initialized. change the return type to reflect the fact that the uploader might be null
andijcr
force-pushed
the
feat/cluster_metadata_manifest_age_metric
branch
from
6561ddd
to
e866d80
Compare
expose the how long ago (in seconds) the manifest was uploaded. the metric is added only when the metadata_uploader is available
andijcr
force-pushed
the
feat/cluster_metadata_manifest_age_metric
branch
from
e866d80
to
97b4131
Compare
andrwng
approved these changes
Mar 27, 2024
dotnwat
approved these changes
Mar 27, 2024
| @@ -102,6 +104,41 @@ void controller_probe::setup_metrics() { | |||
| "Number of partitions that lack quorum among replicants")) | |||
| .aggregate({sm::shard_label}), | |||
| }); | |||
|
|
|||
| if (auto maybe_uploader = _controller.metadata_uploader()) { | |||
There was a problem hiding this comment.
please prefer if (auto x ...; x) instead of implicit cast to bool behavior.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
define a new metric
redpanda_cluster_latest_cluster_metadata_manifest_ageTo tracks the age (in seconds) of the cluster_metadata_manifest saved in cloud storage.
It's updated by the controller, and should result in a Sawtooth pattern, raising steadily and dropping to 0 when a new
cluster_metadata_manifestin uploaded.Fixes https://github.com/redpanda-data/core-internal/issues/1207
Backports Required
Release Notes
Feature
redpanda_cluster_latest_cluster_metadata_manifest_ageto track the age of the cluster_metadata_manifest in cloud storage