KAFKA-2435: Fair consumer partition assignment strategy

[noslowerdna]

No description provided.

[asfbot]

kafka-trunk-git-pr #158 FAILURE
Looks like there's a problem with this pull request

[asfbot]

kafka-trunk-git-pr #170 FAILURE
Looks like there's a problem with this pull request

[noslowerdna]

I don't believe that the @asfbot test failure is at all related to this change. I've built the project successfully locally several times.

kafka.api.QuotasTest > testThrottledProducerConsumer FAILED
    junit.framework.AssertionFailedError: Should have been throttled
        at junit.framework.Assert.fail(Assert.java:47)
        at junit.framework.Assert.assertTrue(Assert.java:20)
        at kafka.api.QuotasTest.testThrottledProducerConsumer(QuotasTest.scala:136)

[noslowerdna]

Looks like #160 may address this failed test...

[hachikuji]

@noslowerdna Out of curiosity, do you have any use cases where the consumers are subscribing to different sets of topics? The only such case I can think of is during a rolling upgrade.

[noslowerdna]

@hachikuji Our primary use case is similar to a rolling upgrade - the members of a consumer group independently retrieve configuration on a regular interval about what topics to subscribe to, and close/refresh their consumer stream whenever the configured set of topics changes. A config change should eventually propagate to all consumers within a few minutes, but we want to still be processing optimally and keep the number of fetcher threads as evenly balanced as possible during the transition window.

[hachikuji]

@noslowerdna Since the consumer supports differing subscriptions, it may make sense for us to include an assignor which takes that into account (even if there don't seem to be any use cases which call for differing subscriptions in a steady state). That said, this change does impact the client configuration (even if in a compatible way), so I wonder if we need a KIP. Also, the new consumer allows users to provide their own assignor implementation, so if the use case is not too common, it may be better to leave it out of core Kafka. What do you think?

[noslowerdna]

@hachikuji I will go ahead and create the KIP documentation for this. It's a minor change but still worth discussing with a wider audience.

I implemented an equivalent "fair" assignor for the new consumer last night (#979). For that pull request I added a new test (testMultipleConsumersUnbalancedSubscriptions) for each of the 3 implementations, that illustrates the skewed assignments for the range and roundrobin assignors with a borderline worst-case scenario.

For the new consumer we could certainly include a custom implementation in our own code if the KIP gets rejected. In that case I think we should at least document somewhere that the roundrobin assignor can give somewhat suboptimal results if the subscriptions are not identical (which is/will be allowed as of https://issues.apache.org/jira/browse/KAFKA-2172).

[noslowerdna]

@hachikuji It seems that I do not have sufficient permissions to create the KIP wiki page. Here's an initial draft: https://gist.github.com/noslowerdna/577e25182d69306532d1

[hachikuji]

@noslowerdna What's your apache id? I can ask someone to give you access.

[noslowerdna]

@hachikuji Thanks, I'm "noslowerdna" there also.

[hachikuji]

@noslowerdna I asked @gwenshap to give you wiki permissions.

[noslowerdna]

KIP link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-49+-+Fair+Partition+Assignment+Strategy

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/442/
Test PASSed (JDK 8 and Scala 2.12).

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/443/
Test PASSed (JDK 8 and Scala 2.11).

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/441/
Test PASSed (JDK 7 and Scala 2.10).

[ghost]

Is this change set still active?

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/2373/
Test PASSed (JDK 7 and Scala 2.10).

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/2373/
Test FAILed (JDK 8 and Scala 2.12).

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/2377/
Test FAILed (JDK 8 and Scala 2.11).

[hachikuji]

Closing this since the JIRA has been resolved as "won't fix."