-
Notifications
You must be signed in to change notification settings - Fork 133
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Documents usage of transactional producer #440
Conversation
@iravid I retained the structure of the other example. There is however a way to commit offsets that is slightly more efficient:
This runs committing on another fiber then producing the messages. So delays in committing does not affect producing messages. (BTW, it would be great if the needed transducer would become part of the library as well.) WDYT? |
Hi @vigoo, can you validate the documentation this PR adds please? |
So we needed a few weeks to find out that this PR is quite wrong. This documentation is very much needed! |
Sorry I somehow missed this PR before. I will try to add an example of how we use transactions with zio-kafka. |
@vigoo We have something that works but it is quite ugly ('work' pending #469 😉). There are almost 2 pages of control code while we only need to consume, map and produce/commit. I keep thinking we should be able to totally hide the generations from the user of the library. Since it is possible the user's code combines messages from multiple generations (and commit them together), the only safe way to do this is by not offering messages from a new generation from the consumer, until all messages from the previous generation have been committed. Roughly like this: WDYT? |
Here is a short summary of how we are using it. I will try to find time and document it properly :)
I could not come up with anything that makes this simpler in the library, but of course it would be nice to reduce the complexity. |
What's the status of this? |
Unless @vigoo wants to spend some more time on this I propose we close this issue. The text I had written is not correct. In addition, I am following @svroonland 's work (helping out where I can, which unfortunately is not much, this code is very complex) that later can be expanded to make this problem go away completely. |
Lacking proper documentation for now, the transactional test added in #644 can be used for learning purposes. |
We have something very similar to the added example code running in production.