KAFKA-1543: Change replication factor during partition map generation

[llamahunter]

If the topic-to-move-json-file contains a new replication-factor for
a topic, it is used when assigning partitions to brokers. If missing,
the existing replication-factor of the topic is maintained.

[llamahunter]

Starting the conversation with this pull request. It seems to work well, and is minimally invasive to the code and command line tool API. I think this is a better approach than putting replication-factor changes in kafka-topics.sh --alter because the replication re-assignment is likely going to require a lot of partition copying, and it's probably better to review the effect rather than it being hidden.

I bumped the 'version' on the topics to move json file, but it turns out no one looks at the version number, anyway. It's backwards compatible with version 1, and old code getting a version 2 file will just ignore the new fields.

[llamahunter]

It's unclear what that failed check is about. I can't see how this change could have affected consumer pattern subscription. I suspect fragile tests.

[llamahunter]

Looked at a few other recent pull requests. Even documentation-only changes are causing the consumer tests to fail. Looks like broken tests.

[tcrayford]

@llamahunter try rebasing against trunk? My patch for 3933 has had passing tests just fine.

[guozhangwang]

@hachikuji Could you take a look at this PR? @llamahunter Could you change the title to KAFKA-1543: Change... so that the JIRA can be linked?

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/324/
Test FAILed (JDK 8 and Scala 2.11).

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/322/
Test FAILed (JDK 7 and Scala 2.10).

[asfbot]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/323/
Test PASSed (JDK 8 and Scala 2.12).

[llamahunter]

Hi... back from a lengthy vacation. What happened with this pull request? Did you go with the kafka-topics.sh approach rather than kafka-reassign-partitions.sh?

[guozhangwang]

@llamahunter I don't think there is a decision on which script to augment yet. Although @jkreps once suggested doing it in kafka-topics.sh, your response on the JIRA and this PR is still targeted at reassign-partition script.

[llamahunter]

I disagree with Jay in putting it in kafka-topics.sh for the reasons I stated above. Changing replication factor is a heavy-weight operation.

[cloudyday]

This feature is still interesting to me. Just checking in.

[msilb]

I see the parent issue has been open since 2014. Any idea when this finally is going to be implemented?

[jamiealquiza]

Found this issue while browsing for something else, but fwiw my topicmappr cli tool does exactly this: topicmappr rebuild --topics <your-topic> --brokers -1 --replication n. Creates a compatible file that can now be assigned or otherwise worked with.

[wlarch]

I have been looking to increase the replicationFactor for the topics of our application because we added brokers to the Kafka cluster. It appears to be possible using the following documentation :

https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor