Fix some invalid metrics definitions #4465
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dnr
approved these changes
Jun 12, 2023
common/metrics/metric_defs.go
Outdated
| @@ -1470,10 +1470,10 @@ var ( | |||
| ReplicationTasksRecvBacklog = NewDimensionlessHistogramDef("replication_tasks_recv_backlog") | |||
| ReplicationTasksApplied = NewCounterDef("replication_tasks_applied") | |||
| ReplicationTasksFailed = NewCounterDef("replication_tasks_failed") | |||
| ReplicationTasksLag = NewTimerDef("replication_tasks_lag") | |||
| ReplicationTasksLag = NewDimensionlessHistogramDef("replication_tasks_lag") | |||
There was a problem hiding this comment.
It looks like this will change the histogram buckets used? It looks like NewTimerDef would cause it to use Milliseconds buckets? But the default for Dimensionless is the same as Milliseconds up to 100,000, so that's probably fine
There was a problem hiding this comment.
Oh whoops, yeah that's a good catch. I think it could cause issues for people who overwrote the bucket boundaries originally. I'd at least like to put a comment in the release notes--making a v2 of this metric is probably overkill though.
MichaelSnowden
force-pushed
the
snowden/broken-metrics-types
branch
from
f5c9b53
to
b1c21b0
Compare
MichaelSnowden
force-pushed
the
snowden/broken-metrics-types
branch
from
b1c21b0
to
d14f9aa
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changed?
These metrics were defined incorrectly: the type of metric actually emitted does not match the metric type in their definition. I found this out when I was working on another change here: 05c26a8
There is still an issue in the archiver metrics code, but it's trickier to fix because, there, not only does the emitted type not match the defined type, but the emitted type is probably not what we want (we probably do want a bytes histogram instead of a counter there), so I'm saving it for a separate PR: 05c26a8#diff-9a79cb5619e22e12ac9cc70a86968b3fce9735922fc7a8f0394664e0ef4bca50L200-L201
Why?
I'm working on fixing this metrics interface, and one of the required changes will be to ensure that metrics definitions actually match the emission type, i.e. you can't do
.Counter(NewGaugeDef(...).GetMetricName())How did you test it?
I made a commit that verifies, at compile time, that the types of metrics definitions provided to the metrics.Handler methods, match the type expected by those methods, and that commit builds (at least locally): 7a0ccd1
Potential risks
Is hotfix candidate?