Skip to content

Kafka cluster sharding support with external sharding allocation#1067

Merged
seglo merged 20 commits intoakka:masterfrom
seglo:seglo/kafka-cluster-sharding
Mar 6, 2020
Merged

Kafka cluster sharding support with external sharding allocation#1067
seglo merged 20 commits intoakka:masterfrom
seglo:seglo/kafka-cluster-sharding

Conversation

@seglo
Copy link
Contributor

@seglo seglo commented Feb 26, 2020

Summary

Provides several convenience setup functions to use a Kafka consumer group aware Akka cluster external sharding allocation strategy. What this means in simple terms is that Akka cluster entity shards will be co-located on akka cluster nodes with a Kafka Consumer's assigned partitions.

Fixes: #1074

Features

  • Retrieve an Akka cluster sharding ShardingMessageExtractor for enveloped or non-enveloped entities.
  • Automatically determine the number of shards to use in the ShardingMessageExtractor create given a topic.
  • Manually assign the number of partitions in the ShardingMessageExtractor.
  • Use the same Apache Kafka Murmur2 hashing function that the KafkaProducer DefaultPartitioner uses to make sure consumed messages are routed to the correct entity.
  • Retrieve an Alpakka Kafka rebalance listener actor that can be used to automatically update the akka cluster external shard allocation mappings when a Kafka consumer group rebalance occurs.

For a sample usage see this WIP Akka Samples PR akka/akka-samples#192

TODO

  • implementation
  • docs

@seglo seglo requested a review from chbatey February 26, 2020 22:08
Copy link
Contributor

@ennru ennru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking forward to the docs, how should they integrate the Akka Sample?

Copy link
Contributor

@chbatey chbatey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. My only comment is regarding APIs that combine classic and typed akka

@seglo seglo force-pushed the seglo/kafka-cluster-sharding branch from da7d3eb to 458b41f Compare March 6, 2020 17:55
@seglo seglo changed the title Kafka cluster sharding support with akka cluster external shard allocation Kafka cluster sharding support with external sharding allocation Mar 6, 2020
@seglo seglo merged commit 907c187 into akka:master Mar 6, 2020
@seglo seglo deleted the seglo/kafka-cluster-sharding branch March 6, 2020 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Kafka cluster sharding support with akka cluster external shard allocation

3 participants