-
-
Notifications
You must be signed in to change notification settings - Fork 522
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add way to commit offsets with Consumer #436
Conversation
With the first steps made to explore how this would look, there's a couple of things to mention:
|
Looking at the code, one thing I'm worried about given the current implementation is that I'll have to dig deeper to see if that's the effect it's having, see if I can write a test to verify it either way. You could say that could be a separate issue and PR, but as this concerns a public API, changing it's behaviour would incur a breaking change, so it's probably something we want to figure out right from the beginning. |
Having looked closer at how |
Some work remains to be done on the documentation side, but apart from that, I think this is looking pretty good and ready for review. |
This PR looks good, but I think it is still missing something. What happens if you try to commit outside of the run loop and you receive a protocol error, for example, and this should be the same for the isolated WDYT? |
You make a good point, something I hadn't considered thus far. Thinking out loud for a bit: Technically speaking, you'd say that both calls to commit offsets (or send heartbeats) happen in the context of the Looking at where these protocol errors are handled now, it seems to be at the Runner level. I guess that has the benefit that any protocol errors inside a @tulios not having really looked at the reality of it in code, my recommendation would be to handle protocol errors that concern the consumer group inside |
Looking at the actual code, things seem a bit more complicated, especially if we want to avoid an all-out refactor. The |
I took the approach of adding a From the perspective of code organisation, I can imagine the same working for The thing I'm wondering if retrying after a rebalance is actually the behaviour we want to see from committing offsets. A commit would be sensitive to the assignment context it's made in, so if that context changes, isn't the correct thing to bail on the attempt instead? A new assignment would then pick up the previously committed offset and "retry" through processing the messages again. To achieve exactly-once, transactions should be used. |
…s, but do trigger a rejoin in the background
@tulios I think that covers it, ready for another look from you! |
@JaapRood sorry for the delay, the changes should work. Final question 😄 |
@tulios no worries, that's what async working is for 👍 . Fetching and committing do not have to be in sync, as far as I've been able to gather through experience, my exploration above in this PR and #395. They just happen to be most of the time. It should be through the use of Hopefully that holds water, happy to further discuss those semantics if need be! |
Nice, that's also my understanding. I just wanted to make sure we were aligned. |
What's the timeline for this reaching a release? (I'd been looking at the documentation & thought it was already released) |
Hi @ThisIsMissEm, we usually run all the changes in production before we release to the public. We merged the feature recently so it might take at least a week for us to release, in the meanwhile, you can depend on the commit hash ( The current setup is suboptimal, and we are already addressing the situation, once #441 is merged every merge to master will create a pre-release version on GitHub with the change. |
@tulios okay, thanks! |
Closes #378.