New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rebalance.timeout.ms support (KIP-62) #1039
Comments
Yes, librdkafka does all control plane stuff in the background and the application doesn't need to worry. |
Did you mean "doesn't" need to worry?
Hmm... According to KIP-62, it looks like the rebalance timeout is actually the new limit for per-message processing time. Am I misreading the KIP? |
Yes :) You are right about KIP-62, so while librdkafka performs heartbeats in the background - which solves the initial problem - it does not yet support KIP-62 protocol changes - the rebalance timeout / max processing time. |
Thanks for the update. Looking forward to when support for KIP-62 / rebalance timeout is added. |
@edenhill I'm interested in tackling the implementation of KIP-62 but it will be a bit of a challenge without context. Could you give me some pointers on what/where needs to be changed? Any tips on how to test this would be greatly appreciated. |
Since librdkafka already has a background thread (or a bunch) that takes care of all the actual broker communication, including heartbeats, there are only a couple of things that needs to be done in to support KIP-62:
|
This is scheduled for v1.0.0 |
Changed defaults: * session.timeout.ms = 10000
Changed defaults: * session.timeout.ms = 10000
Changed defaults: * session.timeout.ms = 10000
Now on master |
Does librdkafka support heartbeats in a background thread? (KIP-62)
Trying to minimize risk of a spinning consumer group if a message unexpectedly takes too long to process.
This landed in Kafka 0.10.1.0 as it required a protocol change to pass the rebalance timeout around.
The text was updated successfully, but these errors were encountered: