Scaling out with Kafka Consumer Groups
Latest commit 80a1d7a Nov 28, 2016 @abhirockzz committed on GitHub Update
Failed to load latest commit information.

Scaling out with Kafka Consumer Groups

A simple example to demonstrate how Kafka consumers are designed for distributed, scale-out architectures


  • single node cluster (keeping things simple)
  • 4 partitions
  • start with 1 consumer and bump up to 4 consumers (increment by 1)

Next ....

  • start Kafka
  • download and setup code mvn clean install
  • start the producer (we'll keep one producer for simplicity) - DOWNLOAD_DIR/target/java -jar kafka-scale.jar producer
  • start all consumers one by one and keep track of the logs in order to figure out partition load distribution - DOWNLOAD_DIR/target/java -jar kafka-scale.jar consumer

Check out the blog post for more details