MINOR: Various cleanups in clients tests #15877

mimaison · 2024-05-06T14:49:43Z

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

chia7712

@mimaison nice cleanup!

chia7712 · 2024-05-07T01:02:59Z

clients/src/test/java/org/apache/kafka/test/Microbenchmarks.java

@@ -78,34 +78,30 @@ public static void main(String[] args) throws Exception {
        final Time time = Time.SYSTEM;
        final AtomicBoolean done = new AtomicBoolean(false);
        final Object lock = new Object();


Maybe we should just delete this old class ...

I'm not sure whether anyone uses this. I still think we should check before deleting it.

chia7712 · 2024-05-07T01:09:29Z

...nts/src/test/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkThreadTest.java

@@ -106,10 +103,9 @@ public void setup() {
        commitRequestManager = testBuilder.commitRequestManager.orElseThrow(IllegalStateException::new);
        offsetsRequestManager = testBuilder.offsetsRequestManager;
        coordinatorRequestManager = testBuilder.coordinatorRequestManager.orElseThrow(IllegalStateException::new);
-        heartbeatRequestManager = testBuilder.heartbeatRequestManager.orElseThrow(IllegalStateException::new);
-        memberhipsManager = testBuilder.membershipManager.orElseThrow(IllegalStateException::new);
+        HeartbeatRequestManager heartbeatRequestManager = testBuilder.heartbeatRequestManager.orElseThrow(IllegalStateException::new);


The variables heartbeatRequestManager and membershipManager are unused. Are they used to test the existence of heartbeatRequestManager and membershipManager? If so, could we rewrite them by assertTrue? For example: assertTrue(testBuilder.heartbeatRequestManager.isPresent());

I don't think they are used to test the existence of the managers here, I would say they were just left unused so we should remove them. Managers are retrieved in this way in many other tests (ex HeartbeatRequestManagerTest), but only when needed.

btw, removing all those unused might help us remove the suppression ClassDataAbstractionCoupling, worth checking

Right, these can be removed and as @lianetm pointed this allows getting rid of the suppression. Thanks

chia7712 · 2024-05-07T01:42:06Z

clients/src/test/java/org/apache/kafka/clients/MetadataTest.java

@@ -1323,7 +1323,7 @@ public void testConcurrentUpdateAndFetchForSnapshotAndCluster() throws Interrupt
                } else { // Thread to read metadata snapshot, once its updated
                    try {
                        if (!atleastMetadataUpdatedOnceLatch.await(5, TimeUnit.MINUTES)) {


How about using assertDoesNotThrow? For example:

assertTrue(assertDoesNotThrow(() -> atleastMetadataUpdatedOnceLatch.await(5, TimeUnit.MINUTES)), "Test had to wait more than 5 minutes, something went wrong.");

A bit unusual to chain assert calls but I'll accept it

chia7712 · 2024-05-07T01:43:06Z

clients/src/test/java/org/apache/kafka/clients/MetadataTest.java

@@ -1335,7 +1335,7 @@ public void testConcurrentUpdateAndFetchForSnapshotAndCluster() throws Interrupt
            });
        }
        if (!allThreadsDoneLatch.await(5, TimeUnit.MINUTES)) {


How about using assertTrue?

assertTrue(allThreadsDoneLatch.await(5, TimeUnit.MINUTES), "Test had to wait more than 5 minutes, something went wrong.");

Yes that's even better!

chia7712 · 2024-05-07T01:48:04Z

clients/src/test/java/org/apache/kafka/common/header/internals/RecordHeadersTest.java

-        Iterator<Header> headerIterator = headers.iterator();
-        while (headerIterator.hasNext()) {
-            headerIterator.next();
+        for (Header ignore : headers) {


How about using toArray? for example: headers.toArray().length

Nice suggestion

lianetm · 2024-05-08T14:38:52Z

clients/src/test/java/org/apache/kafka/clients/consumer/KafkaConsumerTest.java

            consumer.subscribe(Collections.singleton(topic));
            fail("Expected an InvalidGroupIdException");
        } catch (InvalidGroupIdException e) {
            // OK, expected
        }


Since we're cleaning up this, maybe just assertThrows? (same for the following 3 try)

Yes, I was able to simplify a few of these. Thanks!

lianetm · 2024-05-08T15:04:46Z

clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java

@@ -3232,7 +3231,7 @@ public void testProducerBatchRetriesWhenPartitionLeaderChanges() throws Exceptio
            assertTrue(client.hasInFlightRequests());
            client.respond(produceResponse(tp0, -1, Errors.NOT_LEADER_OR_FOLLOWER, 0));
            sender.runOnce(); // receive produce response, batch scheduled for retry
-            assertTrue(!futureIsProduced.isDone(), "Produce request is yet not done.");
+            assertFalse(futureIsProduced.isDone(), "Produce request is yet not done.");


Looks to me this error message is inverted too right? it's stating the desired state, not the failure: if the assert fails, it's because the produce request is done when it shouldn't (not because "is yet not done"). Maybe just "Produce request shouldn't complete yet"...

Good catch, I updated the messages

lianetm · 2024-05-08T15:05:11Z

clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java

@@ -3253,13 +3252,13 @@ public void testProducerBatchRetriesWhenPartitionLeaderChanges() throws Exceptio
            assertTrue(client.hasInFlightRequests());
            client.respond(produceResponse(tp0, -1, Errors.NOT_LEADER_OR_FOLLOWER, 0));
            sender.runOnce(); // receive produce response, schedule batch for retry.
-            assertTrue(!futureIsProduced.isDone(), "Produce request is yet not done.");
+            assertFalse(futureIsProduced.isDone(), "Produce request is yet not done.");


lianetm · 2024-05-08T15:14:42Z

clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java

@@ -3492,9 +3491,9 @@ private void verifyErrorMessage(ProduceResponse response, String expectedMessage
        assertEquals(expectedMessage, e1.getCause().getMessage());
    }

-    class AssertEndTxnRequestMatcher implements MockClient.RequestMatcher {
+    static class AssertEndTxnRequestMatcher implements MockClient.RequestMatcher {


seems could be private too

lianetm · 2024-05-08T15:27:49Z

clients/src/test/java/org/apache/kafka/common/utils/ImplicitLinkedHashMultiCollectionTest.java

-        for (int i = 0; i < testElements.length; i++) {
-            assertTrue(multiSet.add(testElements[i]));
+        for (TestElement testElement : testElements) {
+            assertTrue(multiSet.add(testElement));
        }
-        for (int i = 0; i < testElements.length; i++) {
-            assertFalse(multiSet.add(testElements[i]));
+        for (TestElement testElement : testElements) {
+            assertFalse(multiSet.add(testElement));


could this be simplified to a single loop with the 2 asserts in it? we would achieve the same, validate an elem can be added only if new, for all elems.

Agreed, we can merge both loops

lianetm · 2024-05-13T13:36:20Z

clients/src/test/java/org/apache/kafka/clients/consumer/KafkaConsumerTest.java

-        } catch (Throwable e) {
-            assertEquals(partitionRevoked + singleTopicPartition, e.getCause().getCause().getMessage());
-        }
+        t = assertThrows(Throwable.class, () -> consumer.close(Duration.ofMillis(0)));


Not introduced by this PR, but here we could be more specific and assertThrows(KafkaException.class, ...). Regardless of the type of exception thrown in the rebalance callbacks, a KafkaException is what's always propagated (legacy and new consumer).

Btw, reviewing this PR I noticed that this KafkaConsumerTest has lots of tests not enabled for the new consumer (with lots of TODOs about it). I created KAFKA-16737 to make sure we address it, enable the tests that apply and clean up the code.

Good call, I pushed an update.
Thanks for raising KAFKA-16737

chia7712

LGTM

mimaison · 2024-05-14T08:16:55Z

Thanks for the reviews! Test failures are not related, merging to trunk

Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Lianet Magrans <lianetmr@gmail.com>

MINOR: Various cleanups in clients tests

8a3dc21

chia7712 reviewed May 7, 2024

View reviewed changes

lianetm reviewed May 8, 2024

View reviewed changes

Address review comments

388e277

lianetm reviewed May 13, 2024

View reviewed changes

Throwable -> KafkaException

cad9e27

chia7712 approved these changes May 14, 2024

View reviewed changes

mimaison merged commit 0587a9a into apache:trunk May 14, 2024
1 check failed

mimaison deleted the cleanups-clients-test branch May 14, 2024 08:19

chia7712 mentioned this pull request May 23, 2024

[MINOR] Code Cleanup #16021

Closed

TaiJuWu pushed a commit to TaiJuWu/kafka that referenced this pull request Jun 8, 2024

MINOR: Various cleanups in clients tests (apache#15877)

cef8a26

Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Lianet Magrans <lianetmr@gmail.com>

gongxuanzhang pushed a commit to gongxuanzhang/kafka that referenced this pull request Jun 12, 2024

MINOR: Various cleanups in clients tests (apache#15877)

2f17fd8

Reviewers: Chia-Ping Tsai <chia7712@gmail.com>, Lianet Magrans <lianetmr@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MINOR: Various cleanups in clients tests #15877

MINOR: Various cleanups in clients tests #15877

mimaison commented May 6, 2024

chia7712 left a comment

chia7712 May 7, 2024

mimaison May 13, 2024

chia7712 May 7, 2024

lianetm May 7, 2024

lianetm May 7, 2024

mimaison May 13, 2024

chia7712 May 7, 2024

mimaison May 13, 2024

chia7712 May 7, 2024

mimaison May 13, 2024

chia7712 May 7, 2024

mimaison May 13, 2024

lianetm May 8, 2024

mimaison May 13, 2024

lianetm May 8, 2024

mimaison May 13, 2024

lianetm May 8, 2024

lianetm May 8, 2024

lianetm May 8, 2024

mimaison May 13, 2024

lianetm May 13, 2024

mimaison May 13, 2024

chia7712 left a comment

mimaison commented May 14, 2024

MINOR: Various cleanups in clients tests #15877

MINOR: Various cleanups in clients tests #15877

Conversation

mimaison commented May 6, 2024

Committer Checklist (excluded from commit message)

chia7712 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chia7712 left a comment

Choose a reason for hiding this comment

mimaison commented May 14, 2024