KAFKA-5366: Add concurrent reads to transactions system test #3217

apurvam · 2017-06-02T22:31:29Z

This currently fails in multiple ways. One of which is most likely KAFKA-5355, where the concurrent consumer reads duplicates.

During broker bounces, the concurrent consumer misses messages completely. This is another bug.

asfbot · 2017-06-03T01:04:54Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/4838/
Test PASSed (JDK 7 and Scala 2.11).

asfbot · 2017-06-03T01:06:41Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/4822/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2017-06-06T01:46:26Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/4939/
Test PASSed (JDK 7 and Scala 2.11).

asfbot · 2017-06-06T02:17:29Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/4923/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2017-06-06T05:39:30Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/4949/
Test FAILed (JDK 7 and Scala 2.11).

asfbot · 2017-06-06T05:46:25Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/4933/
Test PASSed (JDK 8 and Scala 2.12).

… exercise the abort index

…esn't close prematurely when there are broker bounces

hachikuji

Thanks for the patch. Left a couple trivial comments.

hachikuji · 2017-06-06T18:23:07Z

clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java

@@ -242,6 +242,9 @@ public synchronized boolean ensurePartitionAdded(TopicPartition tp) {
            // We should enter this branch in an error state because if this partition is already in the transaction,
            // there is a chance that the corresponding batch is in retry. So we must let it completely flush.
            if (!(partitionsInTransaction.contains(tp) || isPartitionPending(tp))) {
+                log.debug("partitionsInTransaction: [{}], newPartitionsInTransaction: [{}], " +


Is this intentional or were you just using this for your own debugging? If it's intentional, we should probably add the log prefix and make the message a little clearer.

hachikuji · 2017-06-06T18:24:02Z

tests/kafkatest/tests/core/transactions_test.py

+        # ensure that the consumer is up.
+        wait_until(lambda: (len(consumer.messages_consumed[1]) > 0) == True,
+                   timeout_sec=60,
+                   err_msg="Consumer failed to consume any messages for for %ds" %\


nit: extra "for"

hachikuji · 2017-06-06T18:25:39Z

tests/kafkatest/tests/core/transactions_test.py

+        return consumer
+
+    def drain_consumer(self, consumer):
+        # wait until the consumer closes, which will be 15 seconds after


This comment doesn't seem to match what the wait_until is doing.

asfbot · 2017-06-06T19:47:32Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/4960/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2017-06-06T20:27:40Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/4976/
Test PASSed (JDK 7 and Scala 2.11).

apurvam · 2017-06-06T20:43:52Z

These are starting to stabilize:

================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.6.0
session_id:       2017-06-06--010
run time:         13 minutes 10.752 seconds
tests run:        4
passed:           4
failed:           0
ignored:          0
================================================================================
test_id:    kafkatest.tests.core.transactions_test.TransactionsTest.test_transactions.failure_mode=clean_bounce.bounce_target=brokers
status:     PASS
run time:   4 minutes 7.220 seconds
--------------------------------------------------------------------------------
test_id:    kafkatest.tests.core.transactions_test.TransactionsTest.test_transactions.failure_mode=clean_bounce.bounce_target=clients
status:     PASS
run time:   2 minutes 17.809 seconds
--------------------------------------------------------------------------------
test_id:    kafkatest.tests.core.transactions_test.TransactionsTest.test_transactions.failure_mode=hard_bounce.bounce_target=brokers
status:     PASS
run time:   3 minutes 2.829 seconds
--------------------------------------------------------------------------------
test_id:    kafkatest.tests.core.transactions_test.TransactionsTest.test_transactions.failure_mode=hard_bounce.bounce_target=clients
status:     PASS
run time:   3 minutes 42.069 seconds
--------------------------------------------------------------------------------

apurvam · 2017-06-06T20:57:34Z

Kicked off a branch builder for this single test. Not sure if jenkins is ready though:

http://jenkins.confluent.io/job/system-test-kafka-branch-builder/902/

apurvam · 2017-06-06T21:12:21Z

The test passed on jenkins: http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/2017-06-06--001.1496783202--apurvam--KAFKA-5366-add-concurrent-reads-to-transactions-system-test--61e2ffc/report.html

asfbot · 2017-06-06T21:54:33Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/4980/
Test PASSed (JDK 7 and Scala 2.11).

hachikuji

LGTM. The coordinator disconnect handling is a good find.

asfbot · 2017-06-06T23:00:34Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/4976/
Test FAILed (JDK 8 and Scala 2.12).

apurvam · 2017-06-06T23:04:53Z

Second successful run: http://jenkins.confluent.io/job/system-test-kafka-branch-builder-2/301/

asfbot · 2017-06-06T23:47:06Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/4992/
Test PASSed (JDK 7 and Scala 2.11).

apurvam · 2017-06-07T00:19:33Z

Third successful run: http://jenkins.confluent.io/job/system-test-kafka-branch-builder-2/302

hachikuji · 2017-06-07T00:19:45Z

Merging to trunk and 0.11.0.

This currently fails in multiple ways. One of which is most likely KAFKA-5355, where the concurrent consumer reads duplicates. During broker bounces, the concurrent consumer misses messages completely. This is another bug. Author: Apurva Mehta <apurva@confluent.io> Reviewers: Jason Gustafson <jason@confluent.io> Closes #3217 from apurvam/KAFKA-5366-add-concurrent-reads-to-transactions-system-test (cherry picked from commit 202cb8e) Signed-off-by: Jason Gustafson <jason@confluent.io>

apurvam force-pushed the KAFKA-5366-add-concurrent-reads-to-transactions-system-test branch from 71fcad1 to 051e77a Compare June 2, 2017 22:38

apurvam force-pushed the KAFKA-5366-add-concurrent-reads-to-transactions-system-test branch from 051e77a to f6f15cb Compare June 6, 2017 00:51

apurvam force-pushed the KAFKA-5366-add-concurrent-reads-to-transactions-system-test branch from f6f15cb to 52028b8 Compare June 6, 2017 04:49

Apurva Mehta added 6 commits June 6, 2017 11:23

Add a concurrent consumer in the transactions system tests. This will…

311aa67

… exercise the abort index

Bounce brokers as well

36e917e

Collect debug and data logs for transactions tests

39299f9

Add some debug logging

2927a68

Tweak timeouts and wait conditions so that the concurrent consumer do…

60ec2c3

…esn't close prematurely when there are broker bounces

Fix timing issues in test, broker hard bounce now passes

07b378f

apurvam force-pushed the KAFKA-5366-add-concurrent-reads-to-transactions-system-test branch from 36603a1 to 07b378f Compare June 6, 2017 18:24

hachikuji reviewed Jun 6, 2017

View reviewed changes

Look up the coordinator again whenever there is a disconnect

61e2ffc

Address PR Comments

2ea7503

Add tests for disconnect cases

abe5169

hachikuji approved these changes Jun 6, 2017

View reviewed changes

Bump the drain timeout as it is too low on jenkins

5ae38ee

asfgit closed this in 202cb8e Jun 7, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-5366: Add concurrent reads to transactions system test #3217

KAFKA-5366: Add concurrent reads to transactions system test #3217

apurvam commented Jun 2, 2017

asfbot commented Jun 3, 2017

asfbot commented Jun 3, 2017

asfbot commented Jun 6, 2017

asfbot commented Jun 6, 2017

asfbot commented Jun 6, 2017

asfbot commented Jun 6, 2017

hachikuji left a comment

hachikuji Jun 6, 2017

hachikuji Jun 6, 2017

hachikuji Jun 6, 2017

asfbot commented Jun 6, 2017

asfbot commented Jun 6, 2017

apurvam commented Jun 6, 2017 •

edited

Loading

apurvam commented Jun 6, 2017

apurvam commented Jun 6, 2017

asfbot commented Jun 6, 2017

hachikuji left a comment •

edited

Loading

asfbot commented Jun 6, 2017

apurvam commented Jun 6, 2017

asfbot commented Jun 6, 2017

apurvam commented Jun 7, 2017

hachikuji commented Jun 7, 2017

KAFKA-5366: Add concurrent reads to transactions system test #3217

KAFKA-5366: Add concurrent reads to transactions system test #3217

Conversation

apurvam commented Jun 2, 2017

asfbot commented Jun 3, 2017

asfbot commented Jun 3, 2017

asfbot commented Jun 6, 2017

asfbot commented Jun 6, 2017

asfbot commented Jun 6, 2017

asfbot commented Jun 6, 2017

hachikuji left a comment

Choose a reason for hiding this comment

hachikuji Jun 6, 2017

Choose a reason for hiding this comment

hachikuji Jun 6, 2017

Choose a reason for hiding this comment

hachikuji Jun 6, 2017

Choose a reason for hiding this comment

asfbot commented Jun 6, 2017

asfbot commented Jun 6, 2017

apurvam commented Jun 6, 2017 • edited Loading

apurvam commented Jun 6, 2017

apurvam commented Jun 6, 2017

asfbot commented Jun 6, 2017

hachikuji left a comment • edited Loading

Choose a reason for hiding this comment

asfbot commented Jun 6, 2017

apurvam commented Jun 6, 2017

asfbot commented Jun 6, 2017

apurvam commented Jun 7, 2017

hachikuji commented Jun 7, 2017

apurvam commented Jun 6, 2017 •

edited

Loading

hachikuji left a comment •

edited

Loading