ISSUE-525 / 764: ruby-kafka does not support different topic subscriptions in the same consumer group

[eduardopoleoflipp]

There are 2 issues that currently prevent supporting this feature:

1. The current RoundRobinAssignmentStrategy assumes all subscriptions are equal

We need a more generalized algorithm to perform assignments now that we have different subscriptions across the consumer group. This PR introduces a new additional strategy heavily inspired in the java kafka client RoundRobinAssignor that properly handle this case.

2. The leader consumer thread assumes that all consumers subscription are equivalent

If different consumers were to have different topic subscription we can't rely on the in-memory @topics variable to produce correct assignments. Instead, we can leverage the subscription information provided by the cluster, and stored in@members, to determine the correct consumer subscriptions for a particular consumer group. Once we have the correct list of topics the downstream code in the assignor can then fetch the corresponding partitions from the cluster in preparation for the assignments.

[mrnonz]

Can't wait!!

[dasch]

Would there be a reason to not always use the new assignment strategy? I'd prefer to have fewer options to confuse people.

[dasch]

Looks good! I'd like to see if we could perhaps merge this with the existing strategy rather than adding a new one, if this is purely additional in terms of functionality and reliability.

[eduardopoleoflipp]

I just wanted to be cautious since I'm not all too familiarized with the project but yes, I can merge this with the original strategy. It should be easy enough since this already passes all the original strategy unit tests. Thanks for taking time to review.

[eduardopoleoflipp]

Hey @dasch ! I incorporated your suggestions. There are a few failing tests but I have a strong suspicious that these are flaky tests. There are all of the type of:

Kafka::ConnectionError:
       Could not connect to any of the seed brokers:
       - kafka://localhost:9092: Connection error IOError: stream closed in another thread
      
Kafka::UnknownError:
       Kafka::UnknownError

Kafka::DeliveryFailed:
       Could not connect to any of the seed brokers:
       - kafka://localhost:9092: Connection error IOError: stream closed in another thread

I've pushed already 3 times and I'm always getting similar errors but for different specs. Also I find it hard to believe that my changes would break the connection to the broker.

[dasch]

Yeah, unfortunately the test suite is really flake – I would love to have someone improve that :D

[dasch]

This looks good! Can you add an entry to the changelog?

[eduardopoleoflipp]

Great! just added the entry to the changelog. Thank you so much!

[dasch]

Awesome! I'd like to cut an alpha release and have you test that out in production, if that's okay?

[eduardopoleoflipp]

Great yes of course, we will!

[eduardopoleoflipp]

@dasch I'll ping you on the dev slack channel once we've ascertained that everything is looking good on our system with the new changes.

[dasch]

Great!