KAFKA-14582: Move JmxTool to tools #13136

fvaleri · 2023-01-20T16:56:22Z

This PR is based on #13131.

Output example:

$ bin/kafka-run-class.sh org.apache.kafka.tools.JmxTool \
  --jmx-url service:jmx:rmi:///jndi/rmi://:9990/jmxrmi \
  --object-name kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec \
  --attributes FifteenMinuteRate,FiveMinuteRate \
  --date-format "yyyyMMdd-hh:mm:ss" \
  --reporting-interval 1000 \
  --report-format tsv
Trying to connect to JMX url: service:jmx:rmi:///jndi/rmi://:9999/jmxrmi
time    20230120-06:23:14
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec:FifteenMinuteRate    0.0
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec:FiveMinuteRate	0.0
time	20230120-06:23:15
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec:FifteenMinuteRate	0.0
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec:FiveMinuteRate	0.0
time	20230120-06:23:16
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec:FifteenMinuteRate	0.0
kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec:FiveMinuteRate	0.0
^C

System test using it:

$ TC_PATHS="tests/kafkatest/tests/core/throttling_test.py::ThrottlingTest.test_throttled_reassignment" \
  _DUCKTAPE_OPTIONS='--parameters '\''{"bounce_brokers":"false"}'\' \
    bash tests/docker/run_tests.sh
docker exec ducker01 bash -c "cd /opt/kafka-dev && ducktape --cluster-file /opt/kafka-dev/tests/docker/build/cluster.json  ./tests/kafkatest/tests/core/throttling_test.py::ThrottlingTest.test_throttled_reassignment --parameters '{"bounce_brokers":"false"}'"
/usr/local/lib/python3.9/dist-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
[INFO:2023-01-21 09:21:39,367]: starting test run with session id 2023-01-21--017...
[INFO:2023-01-21 09:21:39,368]: running 1 tests...
[INFO:2023-01-21 09:21:39,368]: Triggering test 1 of 1...
[INFO:2023-01-21 09:21:39,373]: RunnerClient: Loading test {'directory': '/opt/kafka-dev/tests/kafkatest/tests/core', 'file_name': 'throttling_test.py', 'cls_name': 'ThrottlingTest', 'method_name': 'test_throttled_reassignment', 'injected_args': {'bounce_brokers': 'false'}}
[INFO:2023-01-21 09:21:39,375]: RunnerClient: kafkatest.tests.core.throttling_test.ThrottlingTest.test_throttled_reassignment.bounce_brokers=false: on run 1/1
[INFO:2023-01-21 09:21:39,376]: RunnerClient: kafkatest.tests.core.throttling_test.ThrottlingTest.test_throttled_reassignment.bounce_brokers=false: Setting up...
[INFO:2023-01-21 09:21:41,701]: RunnerClient: kafkatest.tests.core.throttling_test.ThrottlingTest.test_throttled_reassignment.bounce_brokers=false: Running...
[INFO:2023-01-21 09:26:23,770]: RunnerClient: kafkatest.tests.core.throttling_test.ThrottlingTest.test_throttled_reassignment.bounce_brokers=false: Tearing down...
[WARNING - 2023-01-21 09:26:24,090 - service_registry - stop_all - lineno:53]: Error stopping service <ProducerPerformanceService-0-139773137726720: num_nodes: 1, nodes: ['ducker09']>: 1
[INFO:2023-01-21 09:27:12,415]: RunnerClient: kafkatest.tests.core.throttling_test.ThrottlingTest.test_throttled_reassignment.bounce_brokers=false: PASS
[INFO:2023-01-21 09:27:12,415]: RunnerClient: kafkatest.tests.core.throttling_test.ThrottlingTest.test_throttled_reassignment.bounce_brokers=false: Data: None
================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.11.3
session_id:       2023-01-21--017
run time:         5 minutes 33.062 seconds
tests run:        1
passed:           1
flaky:            0
failed:           0
ignored:          0
================================================================================
test_id:    kafkatest.tests.core.throttling_test.ThrottlingTest.test_throttled_reassignment.bounce_brokers=false
status:     PASS
run time:   5 minutes 33.040 seconds
--------------------------------------------------------------------------------

fvaleri · 2023-01-24T08:21:32Z

@mimaison @clolov @vamossagar12 this is ready for review if you have some time.

mimaison

Thanks @fvaleri for the PR. I left a few comments. I think we should use this opportunity to add a few tests too. Ideally write them against the previous tool first to ensure the new version works the same way.

checkstyle/import-control.xml

core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala

tools/src/main/java/org/apache/kafka/tools/JmxCommand.java

fvaleri · 2023-01-25T14:31:01Z

Thanks @mimaison.

I addressed all your comments and now working on the test suite.

…lasses These classes are required by most commands, so they must be migrated first. Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

fvaleri · 2023-01-27T10:35:57Z

@mimaison I've rebased and added some tests.

This is now ready for another review. Thanks.

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

mimaison

Thanks for the PR. I left a few questions.

checkstyle/import-control.xml

core/src/main/scala/kafka/Kafka.scala

core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala

tools/src/test/java/org/apache/kafka/tools/JmxCommandTest.java

tools/src/test/java/org/apache/kafka/tools/JmxToolTest.java

tools/src/main/java/org/apache/kafka/tools/JmxTool.java

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

tools/src/test/java/org/apache/kafka/tools/JmxToolTest.java

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

mimaison

LGTM

ijuma · 2023-02-11T17:33:43Z

Looks like this caused some system test failures: http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/trunk/2023-02-11--001.system-test-kafka-trunk--1676126329--confluentinc--master--d0d9e3f297/report.html

TimeoutError('ubuntu@worker9: Jmx tool took too long to start')
Traceback (most recent call last):
File "/home/jenkins/workspace/system-test-kafka_trunk/kafka/venv/lib/python3.7/site-packages/ducktape-0.8.17-py3.7.egg/ducktape/tests/runner_client.py", line 184, in _do_run
data = self.run_test()
File "/home/jenkins/workspace/system-test-kafka_trunk/kafka/venv/lib/python3.7/site-packages/ducktape-0.8.17-py3.7.egg/ducktape/tests/runner_client.py", line 262, in run_test
return self.test_context.function(self.test)
File "/home/jenkins/workspace/system-test-kafka_trunk/kafka/venv/lib/python3.7/site-packages/ducktape-0.8.17-py3.7.egg/ducktape/mark/_mark.py", line 433, in wrapper
return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
File "/home/jenkins/workspace/system-test-kafka_trunk/kafka/tests/kafkatest/tests/client/quota_test.py", line 139, in test_quota
self.kafka.start()
File "/home/jenkins/workspace/system-test-kafka_trunk/kafka/tests/kafkatest/services/kafka/kafka.py", line 654, in start
self.wait_for_start(node, monitor, timeout_sec)
File "/home/jenkins/workspace/system-test-kafka_trunk/kafka/tests/kafkatest/services/kafka/kafka.py", line 890, in wait_for_start
self.start_jmx_tool(self.idx(node), node)
File "/home/jenkins/workspace/system-test-kafka_trunk/kafka/tests/kafkatest/services/monitor/jmx.py", line 90, in start_jmx_tool
wait_until(lambda: self._jmx_has_output(node), timeout_sec=30, backoff_sec=.5, err_msg="%s: Jmx tool took too long to start" % node.account)
File "/home/jenkins/workspace/system-test-kafka_trunk/kafka/venv/lib/python3.7/site-packages/ducktape-0.8.17-py3.7.egg/ducktape/utils/util.py", line 58, in wait_until
raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: ubuntu@worker9: Jmx tool took too long to start

fvaleri marked this pull request as draft January 20, 2023 16:56

fvaleri force-pushed the move-jmx-tool branch 2 times, most recently from 2e97c4a to 16b90df Compare January 20, 2023 17:31

fvaleri marked this pull request as ready for review January 21, 2023 14:54

fvaleri force-pushed the move-jmx-tool branch 2 times, most recently from 1b927c6 to b7daf79 Compare January 23, 2023 13:12

mimaison reviewed Jan 24, 2023

View reviewed changes

fvaleri force-pushed the move-jmx-tool branch from 28d9f91 to cbdb6ea Compare January 24, 2023 18:22

fvaleri added 3 commits January 26, 2023 20:47

KAFKA-14628: Move CommandLineUtils and CommandDefaultOptions shared c…

b648e19

…lasses These classes are required by most commands, so they must be migrated first. Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

KAFKA-14582: Move JmxTool to tools

e40010d

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

Add MM feedback

a167801

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

mimaison mentioned this pull request Jan 27, 2023

KAFKA-14559: Fix JMX tool to handle the object names with wild cards and optional attributes #13060

Merged

3 tasks

Add integration test

37b998a

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

fvaleri force-pushed the move-jmx-tool branch from f05b104 to 37b998a Compare January 27, 2023 10:29

fvaleri added 2 commits January 27, 2023 16:39

Switch back to the original class name

4213449

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

Minor improvements

607ff24

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

mimaison reviewed Jan 31, 2023

View reviewed changes

Add changes from Mickael feedback

f825f87

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

mimaison reviewed Feb 1, 2023

View reviewed changes

tools/src/test/java/org/apache/kafka/tools/JmxToolTest.java Outdated Show resolved Hide resolved

fvaleri and others added 2 commits February 1, 2023 18:12

Make Idea JUnit pluging happy

d86dc35

Signed-off-by: Federico Valeri <fedevaleri@gmail.com>

Remove printStackTrace()

aa99752

mimaison approved these changes Feb 2, 2023

View reviewed changes

mimaison merged commit 50e0e3c into apache:trunk Feb 2, 2023

fvaleri deleted the move-jmx-tool branch February 2, 2023 10:23

chia7712 mentioned this pull request Feb 11, 2023

MINOR: the package of JmxTool is incorrect when running quota_test.py #13233

Merged

3 tasks

fvaleri added the tools label May 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-14582: Move JmxTool to tools #13136

KAFKA-14582: Move JmxTool to tools #13136

fvaleri commented Jan 20, 2023 •

edited

fvaleri commented Jan 24, 2023

mimaison left a comment

fvaleri commented Jan 25, 2023

fvaleri commented Jan 27, 2023

mimaison left a comment

mimaison left a comment

ijuma commented Feb 11, 2023 •

edited

KAFKA-14582: Move JmxTool to tools #13136

KAFKA-14582: Move JmxTool to tools #13136

Conversation

fvaleri commented Jan 20, 2023 • edited

fvaleri commented Jan 24, 2023

mimaison left a comment

Choose a reason for hiding this comment

fvaleri commented Jan 25, 2023

fvaleri commented Jan 27, 2023

mimaison left a comment

Choose a reason for hiding this comment

mimaison left a comment

Choose a reason for hiding this comment

ijuma commented Feb 11, 2023 • edited

fvaleri commented Jan 20, 2023 •

edited

ijuma commented Feb 11, 2023 •

edited