New partitions are not seen by existing consumers

[bataras]

Using these tools:

• saram-cluter-cli
• kafka manager UI
• a simple producer writing every second to a topic, logging which topic/partition it went to

Create a single, 1 partition topic.
Run the producer. Then run the CLI tool like this..

./sarama-cluster-cli -brokers <a broker addr) -group test -topics <a topic> -offset oldest

The cli tool happily consumes messages. But if you go to the Kafka Manager and add a partition to it, the Producer code sees a rebalance and starts writing alternately to both partitions, but the CLI consumer does not see the new partition and thus starts missing half the messages.

If you stop and rerun the CLI, it sees both partitions and catches up.

Problem is the sarama-cluster consumer needs to be able to automatically pick up new partitions

[bataras]

This issue is about a consumer not picking up new partitions on an existing topic, while a produce does. It's not about picking up new topics

[Preylien]

My bad, I misread your post, and assumed it was related to the problem I was running into. I'll remove it to avoid confusion. 👍

[dim]

@Preylien please see my comment on #170 (comment).

[Preylien]

@dim I don't think this issue is closed. I misread the problem bataras was having, so I believe it should be reopened and looked into separately. Sorry about any confusion.

[bataras]

Right. This issue is about a live consumer group not recognizing when a new partition is added to a topic

[FelixChong]

May I know if any updates on this issue?

[dim]

@FelixChong no updates, sarama-cluster is now officially deprecated since IBM/sarama#1099 was merged and released. The native sarama implementation makes it easy to watch topics and change subscriptions on rebalances.