Leave the consumer group explicitly on shutdown - #2819
Draft
delthas wants to merge 1 commit into
Draft
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
... and 3 files with indirect coverage changes
@@ Coverage Diff @@
## improvement/BB-835/rebalance-guard #2819 +/- ##
======================================================================
+ Coverage 75.59% 75.61% +0.01%
======================================================================
Files 200 200
Lines 13937 13978 +41
======================================================================
+ Hits 10536 10569 +33
- Misses 3391 3399 +8
Partials 10 10
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
delthas
force-pushed
the
improvement/BB-833/leave-group-on-shutdown
branch
4 times, most recently
from
August 20, 2026 15:23
27cd1e8 to
882ec92
Compare
close() unsubscribed and then waited for the rebalance callback to un-assign before disconnecting. librdkafka delivers no such callback when the consumer holds no assignment, and postpones the unsubscribe outright while a rebalance is in progress, so close() never returned: the pod was SIGKILLed at the end of its grace period with the member still registered at the broker. The coordinator then kept the group waiting for a process that no longer existed, and could elect it leader of the next generation, leaving every member without partitions until the session timeout evicted it. Leave the group from close() itself, in the order the group state machine needs: commit, unsubscribe, un-assign, disconnect. Un-assigning does not clear the group assignment, so unsubscribing first is what parks the protocol on an un-assign, and our own un-assign then completes it and sends the LeaveGroup, before disconnect() and without depending on a revoke callback reaching us mid-close. In-flight work is still drained first so its offsets are committed, with the bound it already had through the revoke path, and skipped once the client is disconnected, when there is nothing left to commit and no partitions to give back. A revoke arriving during the shutdown is left for close() to answer rather than releasing the partitions early, which would cut that drain short and strand the offsets it exists to commit. Deferred un-assigns from before the shutdown are superseded for the same reason, and a partition grant arriving mid-close no longer tears down the drain close() is waiting on. An arbitrary rebalance error now synchronises the assignment, as librdkafka requires: the event supersedes any pending revoke on its way past, so leaving the assignment alone would stand the consumer down with no un-assign coming and no watchdog left to force one. close() is also answerable more than once: a second call used to install a second drain wait and strand the first caller until its own timeout, where the services install SIGTERM handlers that can run twice. Also stop waiting on an in-flight backlog publish, which could keep close() rescheduling itself indefinitely, and bound the final disconnect so a client that will not finish tearing down cannot hold the process after the group has already been left. Issue: BB-833
delthas
force-pushed
the
improvement/BB-833/leave-group-on-shutdown
branch
from
August 20, 2026 15:48
882ec92 to
765fc71
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
close()unsubscribed and then waited for the rebalance callback to un-assign before disconnecting. librdkafka delivers no such callback when the consumer holds no assignment, and postpones the unsubscribe outright while a rebalance is in progress — soclose()never returned and the pod was SIGKILLed with the member still registered at the broker.sequenceDiagram participant C as BackbeatConsumer participant K as Kafka Note over C: SIGTERM during a rebalance C->>K: unsubscribe() Note right of K: postponed — a rebalance<br/>is already in progress C->>C: wait for 'unassign' … forever Note over C: SIGKILL at the grace period,<br/>no LeaveGroup ever sent Note over K: member still registered,<br/>may be elected leader of the next<br/>generation and never SyncGroupThe group then holds zero partitions until
session.timeout.ms(45 s) evicts the member. During a rolling update a rebalance is in progress essentially by construction, since the new pod joins before the old one is told to stop.Changes
Release the partitions and drop the subscription before closing, so the close path has nothing to hand back to us and the
LeaveGroupgoes out whatever state the group is in:The bracketed steps only ran if librdkafka delivered a revoke callback. It delivers none when the consumer holds no assignment, and postpones the unsubscribe outright while a rebalance is in progress — in both cases the wait never ends. The same steps now run unconditionally, in
close()itself.The order of the last two matters.
rd_kafka_cgrp_unassign()does not clearrkcg_group_assignment, so un-assigning first is inert as far as the group state machine is concerned;unsubscribe()then fires a revoke at us and parks inWAIT_UNASSIGN_CALL, leaving theLeaveGroupgated on a callback round trip thatdisconnect()is simultaneously blocking on. Unsubscribing first puts us in the one join-state whereunassign()is meaningful, so our own call completes it and sends theLeaveGroupbeforedisconnect()is reached.close()to answer. Releasing the partitions there would cut short the drainclose()is waiting on and strand the offsets that drain exists to commit (BB-758).close()installed. Both previously leftclose()waiting on a callback that could never fire.close()no longer waits on an in-flight backlog publish, which could otherwise keep it rescheduling itself every second indefinitely.In-flight work is still drained before the partitions are released, so offsets are committed exactly as before, and that wait keeps the bound it already had through the revoke path (
max.poll.interval.ms - 1000) — a wedged task delays the departure no longer than it does today. Draining is skipped once the client is disconnected, since there is then nothing to commit and no partitions to give back.That bound is inherited, not chosen: at the default
max.poll.interval.msit is ~299 s, far longer than a pod's grace period, so a wedged task is still killed rather than departing cleanly. Replacing it with a deadline derived from the grace period is the budget work, deliberately left out here — BB-854 shortens the drain first.Verification
Unit tests for the call ordering, completion with no assignment held, the drain wait, the offset-publish skip, watchdog cleanup, and a revoke arriving mid-drain. Each was checked against the previous implementation to confirm it fails there.
Beyond that, a pod-level harness against a real broker (Kafka 3.9.0), terminating consumers in the two states that matter — waiting to rejoin after a revoke, and mid-drain with work in flight:
LeaveGroupreached the coordinator in 11/12 iterations. One R3 leave took 38 s — not this path: an orphaned member id (BB-843) held the group inPreparingRebalancefirst, which is the other half of the same stall.The remaining mid-drain hang is the drain wait itself: the harness sees five tasks whose callbacks never fire, so
close()waits out its bound. That bound is ~299 s at the defaultmax.poll.interval.ms, far longer than a pod's grace period, so such a shutdown is still killed as a ghost. Whether those tasks are genuinely stuck or an artefact of the harness is unresolved — but either way it is the drain budget, not the departure, that is exposed, and this is the strongest argument yet for doing that work next.Issue: BB-833