-
-
Notifications
You must be signed in to change notification settings - Fork 522
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improve balance in the RoundRobinAssigner. #635
Conversation
17da7ba
to
a06199f
Compare
a06199f
to
8f1d80b
Compare
This seems like a reasonable change. Is there any concern during deployment of this version into a consumer group with the old version still running? I don't see any reason why there would, but I just want to be sure. |
@Nevon my understanding is that only that only the leader runs the assigner, so I think we should be OK: In general one thing I'm curious about (which applies equally to the original Assigner), is what happens if different consumers in the groups are subscribed to different topics. |
That's an interesting question. Should be possible to write a test that explores what would happen. I'm pretty sure that the leader will only assign topic partitions for topics that it is subscribed to, but I'm not 100% sure. That being said, I would say subscribing is something you do on the consumer group, rather than on each individual node, from a logical perspective. The only time you would have consumers in a group with mismatching subscriptions is during a rolling deploy or something, in which case they will eventually settle on a set of subscribed topics. |
}) | ||
const topicsPartitions = [].concat(...topicsPartionArrays) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Everything looks good, can you use utils/flatten
instead of [].concat(...)
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Gladly!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
The leader will decide based on its assignment, and I agree with @Nevon about how the subscriptions work.
Do you have this use case today? |
I agree on subscriptions, and I don't have the use case today. It's just something that gets explicit handling in the Java assigner. I don't think we need it. |
The RoundRobinAssigner now use the same round robin to assign partitions across all topics. Previously the RoundRobinAssigner would round robin partitions within each topic. If the consumer was subscribed to many topics with a single partition, this would lead to a situation in which all partitions of all topics were consumed by a single consumer.
8f1d80b
to
52f4238
Compare
This is a legit failure: https://dev.azure.com/tulios/kafkajs/_build/results?buildId=647&view=logs&j=8acd0940-73cb-5c19-95b3-8544b52a2cd2&t=9ffa329e-81ad-5607-4135-a537d31f23d0 I wonder if this was caused by my suggestion of switching |
I tested locally, it's the same. I will try to make it more stable. |
The RoundRobinAssigner now use the same round robin to assign partitions across all topics.
Previously the RoundRobinAssigner would round robin partitions within each topic.
If the consumer was subscribed to many topics with a single partition, this would lead to a
situation in which all partitions of all topics were consumed by a single consumer.