Improve balance in the RoundRobinAssigner.

[Ben-M]

The RoundRobinAssigner now use the same round robin to assign partitions across all topics.
Previously the RoundRobinAssigner would round robin partitions within each topic.
If the consumer was subscribed to many topics with a single partition, this would lead to a
situation in which all partitions of all topics were consumed by a single consumer.

[Nevon]

This seems like a reasonable change. Is there any concern during deployment of this version into a consumer group with the old version still running? I don't see any reason why there would, but I just want to be sure.

[Ben-M]

@Nevon my understanding is that only that only the leader runs the assigner, so I think we should be OK:
https://github.com/tulios/kafkajs/blob/master/src/consumer/consumerGroup.js#L131-L142

In general one thing I'm curious about (which applies equally to the original Assigner), is what happens if different consumers in the groups are subscribed to different topics.

[Nevon]

In general one thing I'm curious about (which applies equally to the original Assigner), is what happens if different consumers in the groups are subscribed to different topics.

That's an interesting question. Should be possible to write a test that explores what would happen. I'm pretty sure that the leader will only assign topic partitions for topics that it is subscribed to, but I'm not 100% sure.

That being said, I would say subscribing is something you do on the consumer group, rather than on each individual node, from a logical perspective. The only time you would have consumers in a group with mismatching subscriptions is during a rolling deploy or something, in which case they will eventually settle on a set of subscribed topics.

[tulios]

Everything looks good, can you use utils/flatten instead of [].concat(...)?

[Ben-M]

Gladly!

[Ben-M]

Done

[tulios]

In general, one thing I'm curious about (which applies equally to the original Assigner), is what happens if different consumers in the groups are subscribed to different topics.

The leader will decide based on its assignment, and I agree with @Nevon about how the subscriptions work.

That being said, I would say subscribing is something you do on the consumer group, rather than on each individual node, from a logical perspective. The only time you would have consumers in a group with mismatching subscriptions is during a rolling deploy or something, in which case they will eventually settle on a set of subscribed topics.

Do you have this use case today?

[Ben-M]

I agree on subscriptions, and I don't have the use case today. It's just something that gets explicit handling in the Java assigner. I don't think we need it.

[tulios]

This is a legit failure: https://dev.azure.com/tulios/kafkajs/_build/results?buildId=647&view=logs&j=8acd0940-73cb-5c19-95b3-8544b52a2cd2&t=9ffa329e-81ad-5607-4135-a537d31f23d0

I wonder if this was caused by my suggestion of switching [].concat(...topicsPartionArrays) with flatten, maybe I miss understood the code.

[tulios]

This is a legit failure: https://dev.azure.com/tulios/kafkajs/_build/results?buildId=647&view=logs&j=8acd0940-73cb-5c19-95b3-8544b52a2cd2&t=9ffa329e-81ad-5607-4135-a537d31f23d0

I wonder if this was caused by my suggestion of switching [].concat(...topicsPartionArrays) with flatten, maybe I miss understood the code.

I tested locally, it's the same. I will try to make it more stable.