Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
Synchronous partition assignment handler (internal) #761
For now this PR just introduces the synchronous partition assignment handler internally without a user-facing API as the use-cases for this need to be understood better.
Introduce a new callback for changes in partitions assigned to an Alpakka Kafka consumer. This allows executing code before a rebalance continues, and querying the underlying Kafka consumer.
These callbacks are much more powerful (and dangerous) than the current rebalance listener actor. The callbacks are executed on the same thread that called
This allows for new ways to handle offsets which might show more efficient for certain use-cases. The tests show an example for external offset storage and one example which commits on rebalance only.
It has been discussed in #539 that the lack of blocking user-defined call-backs to react on Kafka's partition rebalancing makes certain use cases impossible with Alpakka Kafka.
In particular, it is impossible to combine Kafka's partition assignment with external offset storage.
This looks really promising @ennru . When I was considering an API I was thinking about providing strategies that users could opt in with (i.e. "commit on revoked partitions"), and then provide a super user interface which just exposed the consumer itself, but I like the idea behind