Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kafka cluster sharding support with external sharding allocation #1067

Merged
merged 20 commits into from
Mar 6, 2020

Conversation

seglo
Copy link
Member

@seglo seglo commented Feb 26, 2020

Summary

Provides several convenience setup functions to use a Kafka consumer group aware Akka cluster external sharding allocation strategy. What this means in simple terms is that Akka cluster entity shards will be co-located on akka cluster nodes with a Kafka Consumer's assigned partitions.

Fixes: #1074

Features

  • Retrieve an Akka cluster sharding ShardingMessageExtractor for enveloped or non-enveloped entities.
  • Automatically determine the number of shards to use in the ShardingMessageExtractor create given a topic.
  • Manually assign the number of partitions in the ShardingMessageExtractor.
  • Use the same Apache Kafka Murmur2 hashing function that the KafkaProducer DefaultPartitioner uses to make sure consumed messages are routed to the correct entity.
  • Retrieve an Alpakka Kafka rebalance listener actor that can be used to automatically update the akka cluster external shard allocation mappings when a Kafka consumer group rebalance occurs.

For a sample usage see this WIP Akka Samples PR akka/akka-samples#192

TODO

  • implementation
  • docs

build.sbt Outdated Show resolved Hide resolved
build.sbt Outdated Show resolved Hide resolved
build.sbt Show resolved Hide resolved
Copy link
Member

@ennru ennru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking forward to the docs, how should they integrate the Akka Sample?

build.sbt Outdated Show resolved Hide resolved
Copy link
Member

@chbatey chbatey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. My only comment is regarding APIs that combine classic and typed akka

build.sbt Outdated Show resolved Hide resolved
docs/src/main/paradox/cluster-sharding.md Outdated Show resolved Hide resolved
docs/src/main/paradox/cluster-sharding.md Outdated Show resolved Hide resolved
docs/src/main/paradox/cluster-sharding.md Outdated Show resolved Hide resolved
docs/src/main/paradox/cluster-sharding.md Outdated Show resolved Hide resolved
docs/src/main/paradox/cluster-sharding.md Outdated Show resolved Hide resolved
docs/src/main/paradox/cluster-sharding.md Outdated Show resolved Hide resolved
docs/src/main/paradox/cluster-sharding.md Outdated Show resolved Hide resolved
@seglo seglo force-pushed the seglo/kafka-cluster-sharding branch from da7d3eb to 458b41f Compare March 6, 2020 17:55
@seglo seglo changed the title Kafka cluster sharding support with akka cluster external shard allocation Kafka cluster sharding support with external sharding allocation Mar 6, 2020
@seglo seglo merged commit 907c187 into akka:master Mar 6, 2020
@seglo seglo deleted the seglo/kafka-cluster-sharding branch March 6, 2020 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Kafka cluster sharding support with akka cluster external shard allocation
3 participants