New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
release-23.1: kvserver: add metrics for raft (re-)proposals #113153
Conversation
Thanks for opening a backport. Please check the backport criteria before merging:
If some of the basic criteria cannot be satisfied, ensure that the exceptional criteria are satisfied within.
Add a brief release justification to the body of your PR to justify this backport. Some other things to consider:
|
It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR? 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approving based on:
critical gaps in an operators' ability to detect and troubleshoot serious issues in their clusters
Will need an additional non-KV TL review, as well as a post in #db-backports-point-releases, as per the backport policy.
72628d2
to
6f2ede0
Compare
@jbowens Could you please take a look if this is good to backport to 23.1? This change backports a few metrics useful for L2. |
This commit adds new metrics: - `raft.commands.proposed`: commands proposed to Raft by leaseholders - `raft.commands.reproposed.unchanged`: commads retried/reproposed to Raft because they take too long to apply (so they might be dropped) - `raft.commands.reproposed.new-lai`: commands retried/reproposed to Raft because they were committed to Raft out of order (failed the LAI or closed timestamp check) The `proposed` metric includes both `reproposed` metrics. Epic: none Release note (ops change): added metrics for raft proposals and reproposals
6f2ede0
to
078d0e8
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
Backport 1/1 commits from #111272.
/cc @cockroachdb/release
This commit adds new metrics:
raft.commands.proposed
: commands proposed to Raft by leaseholdersraft.commands.reproposed.unchanged
: commads retried/reproposed to Raft because they take too long to apply (so they might be dropped)raft.commands.reproposed.new-lai
: commands retried/reproposed to Raft because they were committed to Raft out of order (failed the LAI or closed timestamp check)The
proposed
metric includes bothreproposed
metrics.Resolves #105199
Epic: none
Release note (ops change): added metrics for raft proposals and reproposals
Release justification: observability improvement for support escalations