Question consumer groups

[holybit]

I notice there is no Zookeeper support which I assume means consumer groups are not possible?

The problem I am trying to solve is we have a Kafka topic with 24 partitions. I would like to build a Perl consumer using this library but am not certain how I will scale the consumers. Is it possible to do any of the following given the scenario above with this Kafka Perl library:

• Have 12 consumers with each consumer reading from 2 partitions?
• Have 24 consumers with each consumer reading 1 partition?
• Have 48 consumers with each consumer reading 1 partition?

[vladm]

Kafka package does not implement High Level Consumer API, however you can try to attach multiple consumer to the same topic by multiplexing their access - allowing only a single consumer be working with the same topic at any given time. Here is the solution/suggestion offered in Kafka documentation:

"By having a notion of parallelism—the partition—within the topics, Kafka is able to provide both ordering guarantees and load balancing over a pool of consumer processes. This is achieved by assigning the partitions in the topic to the consumers in the consumer group so that each partition is consumed by exactly one consumer in the group. By doing this we ensure that the consumer is the only reader of that partition and consumes the data in order. Since there are many partitions this still balances the load over many consumer instances. Note however that there cannot be more consumer instances than partitions.

Kafka only provides a total order over messages within a partition, not between different partitions in a topic. Per-partition ordering combined with the ability to partition data by key is sufficient for most applications. However, if you require a total order over messages this can be achieved with a topic that has only one partition, though this will mean only one consumer process."