Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Synchronous partition assignment handler (internal) #761

Open
wants to merge 19 commits into
base: master
from

Conversation

Projects
None yet
3 participants
@ennru
Copy link
Member

commented Mar 29, 2019

Purpose

For now this PR just introduces the synchronous partition assignment handler internally without a user-facing API as the use-cases for this need to be understood better.

Introduce a new callback for changes in partitions assigned to an Alpakka Kafka consumer. This allows executing code before a rebalance continues, and querying the underlying Kafka consumer.

These callbacks are much more powerful (and dangerous) than the current rebalance listener actor. The callbacks are executed on the same thread that called poll() (the actor thread) so they are allowed to call into the Kafka consumer eg. to query current offsets, commit offsets or to seek to an offset position. The callbacks add to the time poll() is executed and block the internal Kafka consumer actor from doing any other work. As this can easily make influence other Kafka consumer actors running on the same dispatcher, the time the callbacks take is checked and warnings are issued to the logs if they pass a configurable threshold.

This allows for new ways to handle offsets which might show more efficient for certain use-cases. The tests show an example for external offset storage and one example which commits on rebalance only.

Background Context

It has been discussed in #539 that the lack of blocking user-defined call-backs to react on Kafka's partition rebalancing makes certain use cases impossible with Alpakka Kafka.

In particular, it is impossible to combine Kafka's partition assignment with external offset storage.

References

References #539
Follow up #841

@ennru ennru changed the title Synchronous partition assignment handler WIP: Synchronous partition assignment handler Mar 29, 2019

@seglo

This comment has been minimized.

Copy link
Contributor

commented Mar 30, 2019

This looks really promising @ennru . When I was considering an API I was thinking about providing strategies that users could opt in with (i.e. "commit on revoked partitions"), and then provide a super user interface which just exposed the consumer itself, but I like the idea behind RestrictedConsumer better.

@2m 2m added this to the 1.1.0 milestone May 27, 2019

@ennru ennru referenced this pull request Jun 17, 2019

Closed

Alpakka team sprint plan 2019-06-17 #106

10 of 15 tasks complete

@ennru ennru force-pushed the ennru:synchronous-partition-handling branch from 1bcbe5d to 244e8fa Jun 20, 2019

Show resolved Hide resolved build.sbt Outdated
@2m

This comment has been minimized.

Copy link
Member

commented Jun 26, 2019

Looking good. I like how TransactionalSource was able to use this without any code changes.

@ennru ennru force-pushed the ennru:synchronous-partition-handling branch from 2632f94 to 7f7db12 Jun 28, 2019

@ennru ennru force-pushed the ennru:synchronous-partition-handling branch from 8730228 to 352c50a Jul 2, 2019

@ennru ennru modified the milestones: 1.1.0, 1.0.5 Jul 2, 2019

@ennru ennru force-pushed the ennru:synchronous-partition-handling branch from 352c50a to 349a67b Jul 2, 2019

@ennru

This comment has been minimized.

Copy link
Member Author

commented Jul 2, 2019

For now, I removed the user-facing API so that the new partition assignment handler is used internally, only.
The plan is to continue to explore the scenarios this kind of partition assignment handler improves as eg. #841 and #843

@ennru ennru changed the title WIP: Synchronous partition assignment handler Synchronous partition assignment handler (internal) Jul 2, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.