Ignore a deferred un-assign superseded by a later rebalance - #2818
Draft
delthas wants to merge 1 commit into
Draft
Ignore a deferred un-assign superseded by a later rebalance#2818delthas wants to merge 1 commit into
delthas wants to merge 1 commit into
Conversation
Contributor
Hello delthas,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
... and 3 files with indirect coverage changes
@@ Coverage Diff @@
## development/9.5 #2818 +/- ##
===================================================
- Coverage 75.76% 75.59% -0.17%
===================================================
Files 200 200
Lines 13922 13937 +15
===================================================
- Hits 10548 10536 -12
- Misses 3364 3391 +27
Partials 10 10
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Contributor
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
|
On ERR__REVOKE_PARTITIONS the un-assign is deferred until the processing queue and the offset ledger have drained. If the next rebalance granted the partitions back before that happened, the deferred callback still ran and un-assigned them: the consumer then owned partitions at the broker with no local assignment, and since group membership had not changed, nothing triggered another rebalance to rescue it. Track a rebalance id, bumped on every rebalance event, and give up on a deferred un-assign whose id no longer matches. The check runs again before un-assigning, as publishing offsets to zookeeper in between is asynchronous and leaves a second window for the partitions to come back. The drain watchdog is now cleared on every rebalance rather than only on assignment, so a timer armed by a superseded revoke can no longer disconnect a consumer that is not stuck. Issue: BB-835
delthas
force-pushed
the
improvement/BB-835/rebalance-guard
branch
from
August 19, 2026 17:47
de184b1 to
fa04a32
Compare
Contributor
Request integration branchesWaiting for integration branch creation to be requested by the user. To request integration branches, please comment on this pull request with the following command: Alternatively, the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On
ERR__REVOKE_PARTITIONSthe un-assign is deferred until in-flight work has drained, so offsets can be committed before the partitions are released. If the next rebalance grants those partitions back before the drain finishes, the deferred callback still fires and un-assigns them.sequenceDiagram participant K as Kafka participant C as BackbeatConsumer K->>C: revoke [0-4] Note right of C: work in flight,<br/>un-assign deferred K->>C: assign [0-4] C->>C: assign() applied Note right of C: drain completes C->>K: unassign() — discards the new assignment Note right of C: owns partitions at the broker,<br/>no local assignment, no rebalance to recoverMembership has not changed, so nothing triggers another rebalance and the consumer is stuck until restarted, while looking healthy on every liveness signal. Observed in a CTST census run: idle for 2 h 35 min with 188 of 199 messages stranded.
Changes
_rebalanceId, bumped on every rebalance event. A deferred un-assign gives up when its id no longer matches — checked before touching the drain signals and watchdog, which by then belong to a later rebalance, and again beforeunassign(), since publishing offsets to zookeeper in between is asynchronous and leaves a second window.SUPERSEDEDvalue on the existing rebalance counter, so the race is visible in metrics. Nothing downstream enumerates these values.Tests
Seven unit tests in
tests/unit/backbeatConsumer.js, no broker needed (existingBackbeatConsumerMock). Each guard was verified by reverting it and confirming exactly one test fails.Note
A superseded cycle does not emit
'unassign', so a shutdown racing an assignment still leavesclose()waiting. That is pre-existing and owned by BB-833: emitting the event without having un-assigned would pushclose()intodisconnect()while still holding partitions, which hangs.Issue: BB-835