KAFKA-19888: Clamp negative values in coordinator histograms #20986
+8
−4
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Clamp negative values in coordinator histograms, instead of throwing an exception.
github-actions
bot
added
triage
squah-confluent
commented
Nov 25, 2025
| assertEquals(highestTrackableValue, hdrHistogram.max(now)); | ||
|
|
||
| hdrHistogram.record(-50L); | ||
| assertEquals(0, hdrHistogram.max(now + 1000L)); |
Contributor
Author
There was a problem hiding this comment.
There's no histogram min method. We can add one, but it'd be only used in this one test.
Member
|
@squah-confluent Could you please explain the issue and the fix in the description? This must be back ported to 4.2, 4.1 and 4.0. |
Contributor
Author
|
@dajac Thanks for taking a look. I rewrote the description. Let me know your thoughts. |
dajac
pushed a commit
that referenced
this pull request
Nov 26, 2025
The coordinator runtime and group coordinator currently use wall clock time to measure durations for metrics. When the system clock goes backwards due to time adjustments, we attempt to record negative durations for metrics, which throws an ArrayIndexOutOfBoundsException exception. This causes request processing and partition loading to fail while the clock is being adjusted. If partition loading fails, the group the coordinator for that partition becomes unavailable until the broker is restarted or leadership changes again. To address this, we clamp negative durations to zero in histograms instead of throwing ArrayIndexOutOfBoundsExceptions. We will move towards using a monotonic clock for metrics in future work. Reviewers: David Jacot <djacot@confluent.io>
dajac
pushed a commit
that referenced
this pull request
Nov 26, 2025
The coordinator runtime and group coordinator currently use wall clock time to measure durations for metrics. When the system clock goes backwards due to time adjustments, we attempt to record negative durations for metrics, which throws an ArrayIndexOutOfBoundsException exception. This causes request processing and partition loading to fail while the clock is being adjusted. If partition loading fails, the group the coordinator for that partition becomes unavailable until the broker is restarted or leadership changes again. To address this, we clamp negative durations to zero in histograms instead of throwing ArrayIndexOutOfBoundsExceptions. We will move towards using a monotonic clock for metrics in future work. Reviewers: David Jacot <djacot@confluent.io>
dajac
pushed a commit
that referenced
this pull request
Nov 26, 2025
The coordinator runtime and group coordinator currently use wall clock time to measure durations for metrics. When the system clock goes backwards due to time adjustments, we attempt to record negative durations for metrics, which throws an ArrayIndexOutOfBoundsException exception. This causes request processing and partition loading to fail while the clock is being adjusted. If partition loading fails, the group the coordinator for that partition becomes unavailable until the broker is restarted or leadership changes again. To address this, we clamp negative durations to zero in histograms instead of throwing ArrayIndexOutOfBoundsExceptions. We will move towards using a monotonic clock for metrics in future work. Reviewers: David Jacot <djacot@confluent.io>
Member
|
Merged to trunk, 4.2, 4.1 and 4.0. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The coordinator runtime and group coordinator currently use wall clock
time to measure durations for metrics. When the system clock goes
backwards due to time adjustments, we attempt to record negative
durations for metrics, which throws an ArrayIndexOutOfBoundsException
exception. This causes request processing and partition loading to fail
while the clock is being adjusted. If partition loading fails, the group
the coordinator for that partition becomes unavailable until the broker
is restarted or leadership changes again.
To address this, we clamp negative durations to zero in histograms
instead of throwing ArrayIndexOutOfBoundsExceptions. We will move
towards using a monotonic clock for metrics in future work.
Reviewers: David Jacot djacot@confluent.io