KAFKA-19306: Migrate LogCompactionTester to tools module #19905

yunchipang · 2025-06-04T20:50:48Z

jira: KAFKA-19306

log

Producing 1000000 messages..to topics
log-cleaner-test--1726257157227544436-0
Logging produce requests to
/tmp/kafka-log-cleaner-produced-4277333202437782857.txt
Sleeping for 20seconds...
Consuming messages...
Logging consumed messages to
/tmp/kafka-log-cleaner-consumed-5119667275442624588.txt
1000000 rows of data produced, 120194 rows of data consumed (88.0%
reduction).
De-duplicating and validating output files...
Validated 90157 values, 0 mismatches.
Data verification is completed

result

================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.12.0
session_id:       2025-06-27--001
run time:         1 minute 3.622 seconds
tests run:        1
passed:           1
flaky:            0
failed:           0
ignored:          0
================================================================================
test_id:
kafkatest.tests.tools.log_compaction_test.LogCompactionTest.test_log_compaction.metadata_quorum=ISOLATED_KRAFT
status:     PASS
run time:   1 minute 3.307 seconds
--------------------------------------------------------------------------------

github-actions · 2025-06-12T03:28:05Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

github-actions · 2025-06-15T03:33:08Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

github-actions · 2025-06-18T03:28:43Z

A label of 'needs-attention' was automatically added to this PR in order to raise the
attention of the committers. Once this issue has been triaged, the triage label
should be removed to prevent this automation from happening again.

chia7712 · 2025-06-22T19:25:45Z

tools/src/test/java/org/apache/kafka/tools/LogCompactionTester.java

+ * Then we compare the final message in both logs for each key. If this final message is not the same for all keys we
+ * print an error and exit with exit code 1, otherwise we print the size reduction and exit with exit code 0.
+ */
+public class LogCompactionTester {


the e2e don't include the test jar of tools module, which results in no class error when running log_compaction_test.py. Given that ClientCompatibilityTest is in production code, it should be fine to move LogCompactionTester to production code also.

…s correctly

…nal captured

yunchipang · 2025-06-25T00:29:54Z

@chia7712 can you have a look? thanks!

TaiJuWu · 2025-06-25T03:24:48Z

@yunchipang Please merge trunk and fix the build error.

…e-LogCompactionTester-to-tools-module

tools/src/main/java/org/apache/kafka/tools/LogCompactionTester.java

chia7712

@yunchipang thanks for this patch. It seems to me splitting the scala file to 4 java files are unnecessary.

chia7712 · 2025-07-04T16:15:10Z

tools/src/main/java/org/apache/kafka/tools/TestRecordUtils.java

+import java.util.Spliterators;
+import java.util.function.Consumer;
+
+public class TestRecordUtils {


Could you please consider merging all helpers to LogCompactionTester?

chia7712 · 2025-07-04T16:16:48Z

tools/src/main/java/org/apache/kafka/tools/TestRecord.java

+ */
+package org.apache.kafka.tools;
+
+public class TestRecord {


Please use record type to streamline it and then inline it to LogCompactionTester

chia7712 · 2025-07-04T16:17:11Z

tools/src/main/java/org/apache/kafka/tools/LogCompactionTesterOptions.java

+import joptsimple.OptionParser;
+import joptsimple.OptionSpec;
+
+public class LogCompactionTesterOptions {


ditto. Move it to LogCompactionTesterOptions and then rename it to Options

(wip) migrate LogCompactionTester

f30bf5d

github-actions bot added triage tools build labels Jun 4, 2025

move files to correct locations

d12c864

github-actions bot added core tests labels Jun 5, 2025

call new Java class in log_compaction_tester.py

387446e

github-actions bot added the needs-attention label Jun 12, 2025

add test description comment; add async stderr reader to externalSort

1b5f04f

github-actions bot removed the needs-attention label Jun 14, 2025

github-actions bot added the needs-attention label Jun 15, 2025

yunchipang force-pushed the KAFKA-19306-migrate-LogCompactionTester-to-tools-module branch from c68abe6 to c0ec2b9 Compare June 17, 2025 02:25

github-actions bot removed the needs-attention label Jun 17, 2025

yunchipang force-pushed the KAFKA-19306-migrate-LogCompactionTester-to-tools-module branch 2 times, most recently from 715b34a to bba76b2 Compare June 18, 2025 02:07

github-actions bot added the needs-attention label Jun 18, 2025

chia7712 reviewed Jun 22, 2025

View reviewed changes

github-actions bot removed needs-attention triage labels Jun 23, 2025

move LogCompactionTester to production code

267ac01

yunchipang force-pushed the KAFKA-19306-migrate-LogCompactionTester-to-tools-module branch from c1fbd3a to 267ac01 Compare June 24, 2025 04:32

yunchipang added 4 commits June 23, 2025 22:37

add TestUtils dependency

8e6e10d

run spotbugsMain

b3a7c98

override equals() and hashCode() in TestRecord to compare test record…

2fc08b8

…s correctly

add end of line char to validate output message, to have complete sig…

7bf0cc4

…nal captured

yunchipang marked this pull request as ready for review June 25, 2025 00:27

TaiJuWu added the ci-approved label Jun 25, 2025

Merge remote-tracking branch 'upstream/trunk' into KAFKA-19306-migrat…

ca04b4a

…e-LogCompactionTester-to-tools-module

yunchipang commented Jun 25, 2025

View reviewed changes

tools/src/main/java/org/apache/kafka/tools/LogCompactionTester.java Show resolved Hide resolved

yunchipang added 2 commits June 27, 2025 10:15

add waitUntilTrue helper in LogCompactionTester

b3d74a7

fix checkStyleMain

c9a2d74

yunchipang requested a review from chia7712 June 28, 2025 17:54

chia7712 reviewed Jul 4, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

KAFKA-19306: Migrate LogCompactionTester to tools module #19905

KAFKA-19306: Migrate LogCompactionTester to tools module #19905

Uh oh!

yunchipang commented Jun 4, 2025 •

edited by github-actions bot

Loading

Uh oh!

github-actions bot commented Jun 12, 2025

Uh oh!

github-actions bot commented Jun 15, 2025

Uh oh!

github-actions bot commented Jun 18, 2025

Uh oh!

chia7712 Jun 22, 2025

Uh oh!

yunchipang commented Jun 25, 2025

Uh oh!

TaiJuWu commented Jun 25, 2025 •

edited

Loading

Uh oh!

Uh oh!

chia7712 left a comment

Uh oh!

chia7712 Jul 4, 2025

Uh oh!

chia7712 Jul 4, 2025

Uh oh!

chia7712 Jul 4, 2025

Uh oh!

Uh oh!

KAFKA-19306: Migrate LogCompactionTester to tools module #19905

Are you sure you want to change the base?

KAFKA-19306: Migrate LogCompactionTester to tools module #19905

Uh oh!

Conversation

yunchipang commented Jun 4, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jun 12, 2025

Uh oh!

github-actions bot commented Jun 15, 2025

Uh oh!

github-actions bot commented Jun 18, 2025

Uh oh!

chia7712 Jun 22, 2025

Choose a reason for hiding this comment

Uh oh!

yunchipang commented Jun 25, 2025

Uh oh!

TaiJuWu commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

chia7712 left a comment

Choose a reason for hiding this comment

Uh oh!

chia7712 Jul 4, 2025

Choose a reason for hiding this comment

Uh oh!

chia7712 Jul 4, 2025

Choose a reason for hiding this comment

Uh oh!

chia7712 Jul 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yunchipang commented Jun 4, 2025 •

edited by github-actions bot

Loading

TaiJuWu commented Jun 25, 2025 •

edited

Loading