KAFKA-14578: Move ConsumerPerformance to tools

[fvaleri]

=== test data
bin/kafka-producer-perf-test.sh --num-records 30000 --record-size 1000 --throughput -1 --topic my-topic --producer-props bootstrap.servers=:9092 linger.ms=0 batch.size=16384
...
30000 records sent, 199.973337 records/sec (0.19 MB/sec), 1.26 ms avg latency, 205.00 ms max latency, 1 ms 50th, 2 ms 95th, 2 ms 99th, 14 ms 99.9th.


=== test 1
bin/kafka-consumer-perf-test.sh --bootstrap-server :9092 --messages 3000 --timeout 10000 --fetch-size 1048576 --topic my-topic --group my-group
=== old
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2023-02-07 17:53:25:127, 2023-02-07 17:53:28:418, 3.2864, 0.9986, 3446, 1047.0981, 3222, 69, 47.6284, 49942.0290
=== new
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2023-02-23 11:00:27:431, 2023-02-23 11:00:31:063, 3.2949, 0.9072, 3455, 951.2665, 3557, 75, 43.9326, 46066.6667


=== test 2
bin/kafka-consumer-perf-test.sh --bootstrap-server :9092 --messages 3000 --timeout 10000 --fetch-size 100 --topic my-topic --group my-group --show-detailed-stats --reporting-interval 100
=== old
2023-02-07 17:58:09:633, 0, 0.0010, 0.0003, 1, 0.3089, 3212, 25, 0.0381, 40.0000
2023-02-07 17:58:09:733, 0, 0.2279, 2.2697, 239, 2380.0000, 0, 100, 2.2697, 2380.0000
2023-02-07 17:58:09:833, 0, 0.5789, 3.5095, 607, 3680.0000, 0, 100, 3.5095, 3680.0000
2023-02-07 17:58:09:933, 0, 1.0824, 5.0354, 1135, 5280.0000, 0, 100, 5.0354, 5280.0000
2023-02-07 17:58:10:033, 0, 1.8015, 7.1907, 1889, 7540.0000, 0, 100, 7.1907, 7540.0000
2023-02-07 17:58:10:133, 0, 2.8172, 10.1566, 2954, 10650.0000, 0, 100, 10.1566, 10650.0000
=== new
2023-02-23 11:23:35:617, 0, 0.0010, 0.0003, 1, 0.3075, 3217, 35, 0.0272, 28.5714
2023-02-23 11:23:35:717, 0, 0.2356, 2.3460, 247, 2460.0000, 0, 100, 2.3460, 2460.0000
2023-02-23 11:23:35:817, 0, 0.5245, 2.8896, 550, 3030.0000, 0, 100, 2.8896, 3030.0000
2023-02-23 11:23:35:917, 0, 0.9947, 4.7016, 1043, 4930.0000, 0, 100, 4.7016, 4930.0000
2023-02-23 11:23:36:017, 0, 1.6460, 6.5136, 1726, 6830.0000, 0, 100, 6.5136, 6830.0000
2023-02-23 11:23:36:117, 0, 2.5759, 9.2983, 2701, 9750.0000, 0, 100, 9.2983, 9750.0000


=== test 3
bin/kafka-consumer-perf-test.sh --bootstrap-server :9092 --messages 3000 --timeout 10000 --fetch-size 100 --topic my-topic --group my-group &
bin/kafka-consumer-perf-test.sh --bootstrap-server :9092 --messages 3000 --timeout 10000 --fetch-size 100 --topic my-topic --group my-group &
wait
=== old
2023-02-07 18:10:21:826, 2023-02-07 18:10:25:780, 2.8610, 0.7236, 3000, 758.7253, 3238, 716, 3.9958, 4189.9441
2023-02-07 18:10:21:824, 2023-02-07 18:10:25:988, 2.8610, 0.6871, 3000, 720.4611, 3239, 925, 3.0930, 3243.2432
=== new
2023-02-23 11:07:55:051, 2023-02-23 11:07:59:421, 2.8610, 0.6547, 3000, 686.4989, 3276, 1094, 2.6152, 2742.2303
2023-02-23 11:07:55:105, 2023-02-23 11:07:59:864, 2.8610, 0.6012, 3000, 630.3845, 3225, 1534, 1.8651, 1955.6714


=== system test
TC_PATHS="tests/kafkatest/benchmarks/core/benchmark_test.py::Benchmark.test_producer_and_consumer" bash tests/docker/run_tests.sh
=== old
SESSION REPORT (ALL TESTS)
ducktape version: 0.11.3
session_id:       2023-02-07--006
run time:         7 minutes 12.212 seconds
tests run:        6
passed:           6
flaky:            0
failed:           0
ignored:          0
=== new
SESSION REPORT (ALL TESTS)
ducktape version: 0.11.3
session_id:       2023-02-23--002
run time:         7 minutes 30.679 seconds
tests run:        6
passed:           6
flaky:            0
failed:           0
ignored:          0

[fvaleri]

@mimaison @OmniaGM @clolov

[mimaison]

Thanks for the PR. I left a few comments.

[fvaleri]

@mimaison thanks for the review.

I fixed the rebalance listener and re-ran all the tests in the description.

[mimaison]

LGTM