KAFKA-13794: Follow up to fix comparator #12006

ddrid · 2022-04-07T01:39:34Z

Follow up to original PR #11991

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

ddrid · 2022-04-07T01:43:08Z

Hi @artemlivshits, please take a look if you have time

ddrid · 2022-04-07T01:48:42Z

also cc @hachikuji

showuon

@ddrid , thanks for the PR. Left a comment.

showuon · 2022-04-07T04:04:54Z

clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java

@@ -187,7 +187,7 @@ private void startSequencesAtBeginning(TopicPartition topicPartition, ProducerId
        private static final Comparator<ProducerBatch> PRODUCER_BATCH_COMPARATOR = (b1, b2) -> {
            if (b1.baseSequence() < b2.baseSequence()) return -1;
            else if (b1.baseSequence() > b2.baseSequence()) return 1;
-            else return b1.equals(b2) ? 0 : 1;
+            else return b1.equals(b2) ? 0 : Integer.compare(b1.hashCode(), b2.hashCode());


Do you think we should override the equals and hashcode method in ProducerBatch class? Otherwise, the hashcode result could have possibility to be identical and makes the original issue happened again.

Hi @showuon, thanks for you comment. I found some trouble when adding equals method in ProducerBatch. The only two final fields I can use is createdMs and topicPartition, I don't think it make sense if I use these two fields since they usually have duplicates. Do you have any suggestions? Thanks

ProducerBatch doesn't override hashCode, so a default implementation is used. It's not fully specified what the default implementation of hashCode has to return, but it looks like the suggested implementation is to return a value based on the object address, so it should be different for different objects.
BTW, we could probably just do

else return Integer.compare(b1.hashCode(), b2.hashCode());

if the objects are equal, their hash code must be equal too.

@artemlivshits done, thanks for your suggestion

Make sense. Thanks for the update.

But why do we have to limit ourselves to these 3 options? We can do the equals check and then do the hashCode thing if unequal.

If equals is true, then hash codes must be equal too, so comparing hashCode for equal objects would return 0 in that case. If equals is false, then we do hashCode comparison. If we fold these 2 cases, then if we just do hashCode comparison, the result is the same as comparing equals first.

That's true, we can't rely on hashcode if equals is false either. I'll take a look at the code a bit later, but it should be possible to avoid the latent bug here.

From the release perspective, it improves over the previous release, so for now we could take this as is and file a bug to work on a more precise solution in a future release. We could either add a unique long id (for every new object just increment a static atomic and get a new value) or instead of a producer batch use a HashSet of batches.

It turns out that there are a couple of simple solutions, see #12096.

artemlivshits · 2022-04-08T17:15:35Z

LGTM, thank you for the fix.

showuon

LGTM!

showuon · 2022-04-09T02:56:07Z

Failed tests are unrelated

Build / ARM / kafka.network.ConnectionQuotasTest.testListenerConnectionRateLimitWhenActualRateAboveLimit()

In comparator, objects that are not equal need to have a stable order otherwise, binary search may not find the objects. Improve the producer batch comparator Reviewers: Luke Chen <showuon@gmail.com>

showuon · 2022-04-09T03:05:15Z

Cherry-pick back to 3.2, 3.1, 3.0 as #11991 did.

dajac · 2022-04-09T09:46:34Z

@tombentley We might want to get this one into 3.1.1 to fix the previous fix.

ijuma · 2022-04-14T01:00:08Z

clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java

@@ -187,7 +187,7 @@ private static class TopicPartitionEntry {
        private static final Comparator<ProducerBatch> PRODUCER_BATCH_COMPARATOR = (b1, b2) -> {
            if (b1.baseSequence() < b2.baseSequence()) return -1;
            else if (b1.baseSequence() > b2.baseSequence()) return 1;
-            else return b1.equals(b2) ? 0 : 1;
+            else return Integer.compare(b1.hashCode(), b2.hashCode());


Hmm, hashCode can have collisions. This doesn't seem right either.

I had the same question like you. But then, I think since we can't have a perfect way to compare two ProducerBatch object, the hashCode could be an alternative approach. That's my thought. Thanks.

I replied to the original discussion clarifying that some of the assumptions there are incorrect. We should not return true for the case where hashCode is true and equals is false.

Replied to the other comment as well. I wouldn't say the assumptions are incorrect -- the assumption is that hashCode would provide enough differentiation to significantly improve over original logic, without introducing regressions or requiring more complex implementation.

The assumption about object addresses and hashCode is what I was referring to.

tombentley · 2022-04-20T17:25:33Z

@ijuma did you have any ideas about how to fix this without the problem with hash collisions?

ijuma · 2022-04-21T17:15:03Z

@tombentley I had a chat with @hachikuji about this and it seems that the behavior we want is slightly differently. I'll see if I can submit a PR today for discussion.

ijuma · 2022-04-28T13:25:06Z

@tombentley See #12096.

In comparator, objects that are not equal need to have a stable order otherwise, binary search may not find the objects. Improve the producer batch comparator Reviewers: Luke Chen <showuon@gmail.com>

…cs-11-may-2022 * apache-kafka/3.1: (51 commits) MINOR: reload4j build dependency fixes (apache#12144) KAFKA-13255: Use config.properties.exclude when mirroring topics (apache#11401) KAFKA-13794: Fix comparator of inflightBatchesBySequence in TransactionsManager (round 3) (apache#12096) KAFKA-13794: Follow up to fix producer batch comparator (apache#12006) fix: make sliding window works without grace period (#kafka-13739) (apache#11980) 3.1.1 release notes (apache#12001) KAFKA-13794; Fix comparator of `inflightBatchesBySequence` in `TransactionManager` (apache#11991) KAFKA-13782; Ensure correct partition added to txn after abort on full batch (apache#11995) KAFKA-13748: Do not include file stream connectors in Connect's CLASSPATH and plugin.path by default (apache#11908) KAFKA-13775: CVE-2020-36518 - Upgrade jackson-databind to 2.12.6.1 (apache#11962) ...

…cs-12-may-2022 * apache-kafka/3.0: (14 commits) fix: make sliding window works without grace period (#kafka-13739) (apache#11980) KAFKA-13794: Follow up to fix producer batch comparator (apache#12006) KAFKA-13794; Fix comparator of `inflightBatchesBySequence` in `TransactionManager` (apache#11991) KAFKA-13748: Do not include file stream connectors in Connect's CLASSPATH and plugin.path by default (apache#11908) KAFKA-13418: Support key updates with TLS 1.3 (apache#11966) KAFKA-13770: Restore compatibility with KafkaBasedLog using older Kafka brokers (apache#11946) KAFKA-13761: KafkaLog4jAppender deadlocks when idempotence is enabled (apache#11939) KAFKA-13759: Disable idempotence by default in producers instantiated by Connect (apache#11933) MINOR: Fix `ConsumerConfig.ISOLATION_LEVEL_DOC` (apache#11915) KAFKA-13750; Client Compatability KafkaTest uses invalid idempotency configs (apache#11909) ...

sql888 · 2023-04-10T19:17:53Z

Failed tests are unrelated

Build / ARM / kafka.network.ConnectionQuotasTest.testListenerConnectionRateLimitWhenActualRateAboveLimit()

How to fix this failed test? getting this in M1 ARM Macbook. Thanks

kafka.network.ConnectionQuotasTest.testListenerConnectionRateLimitWhenActualRateAboveLimit() failed, log available in /Users/gli/git/qaas-kafka/core/build/reports/testOutput/kafka.network.ConnectionQuotasTest.testListenerConnectionRateLimitWhenActualRateAboveLimit().test.stdout

ConnectionQuotasTest > testListenerConnectionRateLimitWhenActualRateAboveLimit() FAILED
    java.util.concurrent.ExecutionException: org.opentest4j.AssertionFailedError: Expected rate (30 +- 7), but got 37.15860531368056 (600 connections / 16.147 sec) ==> expected: <30.0> but was: <37.15860531368056>
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
        at kafka.network.ConnectionQuotasTest.$anonfun$testListenerConnectionRateLimitWhenActualRateAboveLimit$3(ConnectionQuotasTest.scala:411)
        at scala.collection.immutable.List.foreach(List.scala:333)
        at kafka.network.ConnectionQuotasTest.testListenerConnectionRateLimitWhenActualRateAboveLimit(ConnectionQuotasTest.scala:411)

        Caused by:
        org.opentest4j.AssertionFailedError: Expected rate (30 +- 7), but got 37.15860531368056 (600 connections / 16.147 sec) ==> expected: <30.0> but was: <37.15860531368056>
            at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)
            at org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)
            at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:86)
            at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1003)
            at kafka.network.ConnectionQuotasTest.acceptConnectionsAndVerifyRate(ConnectionQuotasTest.scala:903)
            at kafka.network.ConnectionQuotasTest.$anonfun$testListenerConnectionRateLimitWhenActualRateAboveLimit$2(ConnectionQuotasTest.scala:409)

KAFKA-13794: Follow up to fix comparator

619c70e

showuon reviewed Apr 7, 2022

View reviewed changes

address comment

cb202a1

ddrid requested a review from showuon April 8, 2022 01:36

showuon approved these changes Apr 9, 2022

View reviewed changes

showuon merged commit 9596c7b into apache:trunk Apr 9, 2022

ijuma reviewed Apr 14, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-13794: Follow up to fix comparator #12006

KAFKA-13794: Follow up to fix comparator #12006

ddrid commented Apr 7, 2022

ddrid commented Apr 7, 2022

ddrid commented Apr 7, 2022

showuon left a comment

showuon Apr 7, 2022

ddrid Apr 7, 2022 •

edited

Loading

artemlivshits Apr 7, 2022

ddrid Apr 8, 2022

showuon Apr 8, 2022

ijuma Apr 14, 2022

artemlivshits Apr 14, 2022

ijuma Apr 14, 2022

artemlivshits Apr 22, 2022

ijuma Apr 28, 2022

artemlivshits commented Apr 8, 2022

showuon left a comment

showuon commented Apr 9, 2022

showuon commented Apr 9, 2022

dajac commented Apr 9, 2022

ijuma Apr 14, 2022

showuon Apr 14, 2022

ijuma Apr 14, 2022 •

edited

Loading

artemlivshits Apr 14, 2022

ijuma Apr 14, 2022

tombentley commented Apr 20, 2022

ijuma commented Apr 21, 2022

ijuma commented Apr 28, 2022

sql888 commented Apr 10, 2023 •

edited

Loading

KAFKA-13794: Follow up to fix comparator #12006

KAFKA-13794: Follow up to fix comparator #12006

Conversation

ddrid commented Apr 7, 2022

Committer Checklist (excluded from commit message)

ddrid commented Apr 7, 2022

ddrid commented Apr 7, 2022

showuon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddrid Apr 7, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

artemlivshits commented Apr 8, 2022

showuon left a comment

Choose a reason for hiding this comment

showuon commented Apr 9, 2022

showuon commented Apr 9, 2022

dajac commented Apr 9, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ijuma Apr 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tombentley commented Apr 20, 2022

ijuma commented Apr 21, 2022

ijuma commented Apr 28, 2022

sql888 commented Apr 10, 2023 • edited Loading

ddrid Apr 7, 2022 •

edited

Loading

ijuma Apr 14, 2022 •

edited

Loading

sql888 commented Apr 10, 2023 •

edited

Loading