Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
Transactions: Add offsets to be committed directly after `producer.send` #752
The transaction flow could produce duplicates (as the test cases showed). By adding the offsets to be committed directly after
The included test-case where multiple transactional source-sink flows are being restarted while moving messages from one topic to another. This test-case showed produced duplicates before the change.
The restarted transactional producers should be fine. They provide transactional id which then is used to retrieve the producer id. This, together with calling
Still investigating the root cause of this. Currently looking into the effects of rebalancing to the transactional consumer, where the duplicates are coming from.
referenced this pull request
Mar 22, 2019
Added a commit where consumer actor is only initialized when the consumer stage gets demand. This makes sure that
However I still noticed some duplicates (but less) after this change.
Here is another commit that builds onto this, where it checks for duplicate messages in the producer. This would catch any duplicates arriving to the same instance of producer. However there does not seem to be any when running the test-case, meaning that duplication occurs between different instances of the producer.
I also published jars with this fix for anymore that would like to try this out:
You will need to add Alpakka Kafka snapshot repository to your build to resolve it: https://doc.akka.io/docs/alpakka-kafka/current/snapshots.html