KAFKA-19623: Implement KIP-1147 for console producer/consumer/share-consumer. #20479

ShivsundarR · 2025-09-04T12:46:07Z

What
https://issues.apache.org/jira/browse/KAFKA-19623

The PR implements KIP-1147
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-1147%3A+Improve+consistency+of+command-line+arguments)
for the console tools i.e. ConsoleProducer, ConsoleConsumer and
ConsoleShareConsumer.
Currently the previous names for the options are still usable but
there will be warning message stating those are deprecated and will be
removed in a future version.
I have added unit tests and also manually verified using the console
tools that things are working as expected.

Reviewers: Andrew Schofield aschofield@confluent.io, Jhen-Yung Hsu
jhenyunghsu@gmail.com, Jimmy Wang
48462172+JimmyWang6@users.noreply.github.com

JimmyWang6

@ShivsundarR Thanks for the patch. I think perhaps console_consumer.py and console_share_consumer.py might need to be changed as well.

AndrewJSchofield

Thanks for the PR. There are some more changes required too.

--property is deprecated and replaced by --formatter-property in the two consumer tools
There are a few docs references to the old names
The docker example references the old names

Yunyung

Thanks for the PR. Partial review only. Please also update the e2e tests and ensure compatibility with older versions.

tools/src/main/java/org/apache/kafka/tools/ConsoleProducer.java

tools/src/main/java/org/apache/kafka/tools/consumer/ConsoleShareConsumerOptions.java

tools/src/main/java/org/apache/kafka/tools/consumer/ConsoleConsumerOptions.java

tools/src/test/java/org/apache/kafka/tools/consumer/ConsoleShareConsumerOptionsTest.java

tools/src/test/java/org/apache/kafka/tools/consumer/ConsoleConsumerOptionsTest.java

AndrewJSchofield · 2025-09-16T17:01:20Z

@ShivsundarR Please fix the check style failure. I'll continue with reviewing the PR.

AndrewJSchofield

You need to look at docker/test/docker_sanity_test.py.

docker/examples/README.md

AndrewJSchofield

A couple of minor additional comments. Please can you run the docker_sanity_test.py also.

docker/examples/README.md

ShivsundarR · 2025-09-17T08:09:34Z

Thanks @AndrewJSchofield for the review. I ran the docker-sanity-test locally and it ran successfully. I have attached the results below.
results_docker_sanity_test.txt

Also there was a bug in docker_build_test.py that would not allow running this test using a local image. I had to modify it temporarily to accept --kafka-archive for tests and then it ran.
I have created an AK issue here - https://issues.apache.org/jira/browse/KAFKA-19718.

docker/test/docker_sanity_test.py

ShivsundarR · 2025-09-17T12:23:09Z

I ran one of the tests in test_console_consumer.py with an older version (4.1) and latest version (4.2) and confirmed that the tests pass.
The older one uses the older parameters(consumer.config) and the tests running latest version uses the new parameters (command-config)
test_4_1.txt
test_4_2.txt

AndrewJSchofield

Looks good to me.

chia7712

@ShivsundarR Thanks for the patch. Just two small suggestions remain, which could be addressed in a follow-up

chia7712 · 2025-09-17T16:05:34Z

tools/src/main/java/org/apache/kafka/tools/consumer/ConsoleConsumerOptions.java

+                            " print.value=true|false\n" +
+                            " key.separator=<key.separator>\n" +
+                            " line.separator=<line.separator>\n" +
+                            " headers.separator=<line.separator>\n" +


typo: line.separator -> headers.separator

chia7712 · 2025-09-17T16:05:42Z

tools/src/main/java/org/apache/kafka/tools/consumer/ConsoleShareConsumerOptions.java

+                                " print.value=true|false\n" +
+                                " key.separator=<key.separator>\n" +
+                                " line.separator=<line.separator>\n" +
+                                " headers.separator=<line.separator>\n" +


I'll fix this.

Yunyung · 2025-09-19T07:25:41Z

tools/src/test/java/org/apache/kafka/tools/consumer/ConsoleShareConsumerOptionsTest.java

The new --formatter-property is not tested in the UT and still uses --property in the test.

Thanks @Yunyung, I had missed them, my bad.
I have raised a minor followup to add the unit tests here - #20560.

Yunyung · 2025-09-19T07:29:02Z

tools/src/test/java/org/apache/kafka/tools/consumer/ConsoleConsumerOptionsTest.java

Thanks, I have raised follow-up fix here - #20560.

…onsumer. (apache#20479) *What* https://issues.apache.org/jira/browse/KAFKA-19623 - The PR implements KIP-1147 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-1147%3A+Improve+consistency+of+command-line+arguments) for the console tools i.e. `ConsoleProducer`, `ConsoleConsumer` and `ConsoleShareConsumer`. - Currently the previous names for the options are still usable but there will be warning message stating those are deprecated and will be removed in a future version. - I have added unit tests and also manually verified using the console tools that things are working as expected. Reviewers: Andrew Schofield <aschofield@confluent.io>, Jhen-Yung Hsu <jhenyunghsu@gmail.com>, Jimmy Wang <48462172+JimmyWang6@users.noreply.github.com>

Implement KIP-1147 for console tools

90632f6

github-actions bot added triage PRs from the community tools labels Sep 4, 2025

ShivsundarR added ci-approved and removed triage PRs from the community labels Sep 4, 2025

ShivsundarR requested a review from AndrewJSchofield September 4, 2025 12:47

JimmyWang6 reviewed Sep 5, 2025

View reviewed changes

AndrewJSchofield requested changes Sep 5, 2025

View reviewed changes

Yunyung reviewed Sep 8, 2025

View reviewed changes

ShivsundarR added 2 commits September 11, 2025 18:33

Merge remote-tracking branch 'upstream/trunk' into KAFKA-19623

be274af

Address review comments

9fb6aeb

github-actions bot added the docker Official Docker image label Sep 16, 2025

AndrewJSchofield requested changes Sep 16, 2025

View reviewed changes

docker/examples/README.md Outdated Show resolved Hide resolved

docker/examples/README.md Outdated Show resolved Hide resolved

ShivsundarR added 3 commits September 16, 2025 23:11

Address comments + checkstyle

bb419ae

Merge remote-tracking branch 'upstream/trunk' into KAFKA-19623

ed9922f

Fix checkstyle

c33f5ce

AndrewJSchofield requested changes Sep 16, 2025

View reviewed changes

docker/examples/README.md Outdated Show resolved Hide resolved

docker/examples/README.md Outdated Show resolved Hide resolved

ShivsundarR added 3 commits September 17, 2025 13:23

Remove 4.2 warning documentation

96b8e0f

Merge remote-tracking branch 'upstream/trunk' into KAFKA-19623

e383d94

Remove local file

0d19656

Use formatter-property in integ tests

15b8f68

github-actions bot added the streams label Sep 17, 2025

Merge remote-tracking branch 'upstream/trunk' into KAFKA-19623

b959032

AndrewJSchofield reviewed Sep 17, 2025

View reviewed changes

docker/test/docker_sanity_test.py Show resolved Hide resolved

docker/test/docker_sanity_test.py Show resolved Hide resolved

Fix test

d935026

AndrewJSchofield approved these changes Sep 17, 2025

View reviewed changes

AndrewJSchofield merged commit 3bc50f9 into apache:trunk Sep 17, 2025
22 checks passed

chia7712 reviewed Sep 17, 2025

View reviewed changes

Yunyung reviewed Sep 19, 2025

View reviewed changes

KAFKA-19623: Implement KIP-1147 for console producer/consumer/share-consumer. #20479

KAFKA-19623: Implement KIP-1147 for console producer/consumer/share-consumer. #20479

Conversation

ShivsundarR commented Sep 4, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JimmyWang6 left a comment

Choose a reason for hiding this comment

Uh oh!

AndrewJSchofield left a comment

Choose a reason for hiding this comment

Uh oh!

Yunyung left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AndrewJSchofield commented Sep 16, 2025

Uh oh!

AndrewJSchofield left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

AndrewJSchofield left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ShivsundarR commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ShivsundarR commented Sep 17, 2025

Uh oh!

AndrewJSchofield left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chia7712 left a comment

Choose a reason for hiding this comment

Uh oh!

chia7712 Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

chia7712 Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

AndrewJSchofield Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

Yunyung Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ShivsundarR Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

Yunyung Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

ShivsundarR Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ShivsundarR commented Sep 4, 2025 •

edited by github-actions bot

Loading

ShivsundarR commented Sep 17, 2025 •

edited

Loading

Yunyung Sep 19, 2025 •

edited

Loading