KAFKA-5281: System tests for transactions #3149

apurvam · 2017-05-26T06:11:49Z

No description provided.

asfbot · 2017-05-26T06:15:01Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/4406/
Test FAILed (JDK 8 and Scala 2.12).

asfbot · 2017-05-26T06:15:09Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/4420/
Test FAILed (JDK 7 and Scala 2.11).

asfbot · 2017-05-26T07:09:36Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/4412/
Test FAILed (JDK 8 and Scala 2.12).

asfbot · 2017-05-26T07:10:39Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/4426/
Test FAILed (JDK 7 and Scala 2.11).

ijuma · 2017-05-26T09:27:19Z

tests/kafkatest/tests/core/transactions_test.py

+        self.num_brokers = 3
+
+        # Test parameters
+        self.num_input_partitions = 1


What is the reason for using a single input partition?

It's temporary. The idea is to have 2 parallel copy jobs so that we have 2 inflight transactions at any given time. So each needs to process a single partition to ensure no overlap in the copied data.

Right now, we have only a single copy job, so there is only one input partition.

asfbot · 2017-05-26T23:25:12Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/4457/
Test PASSed (JDK 7 and Scala 2.11).

asfbot · 2017-05-26T23:53:02Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/4443/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2017-05-27T01:11:10Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/4455/
Test FAILed (JDK 8 and Scala 2.12).

asfbot · 2017-05-27T01:20:49Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/4469/
Test FAILed (JDK 7 and Scala 2.11).

asfbot · 2017-05-27T08:26:17Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/4478/
Test PASSed (JDK 7 and Scala 2.11).

asfbot · 2017-05-27T08:36:51Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/4464/
Test PASSed (JDK 8 and Scala 2.12).

apurvam · 2017-05-27T20:06:31Z

System test job: http://jenkins.confluent.io/job/system-test-kafka-branch-builder/901/

asfbot · 2017-05-27T20:40:48Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/4496/
Test PASSed (JDK 7 and Scala 2.11).

asfbot · 2017-05-27T21:02:40Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/4481/
Test FAILed (JDK 8 and Scala 2.12).

apurvam · 2017-05-28T00:23:51Z

The hard bounce test seems to have revealed at least one bug: https://issues.apache.org/jira/browse/KAFKA-5339

asfbot · 2017-05-28T01:42:18Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/4500/
Test PASSed (JDK 7 and Scala 2.11).

asfbot · 2017-05-28T02:06:18Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/4485/
Test PASSed (JDK 8 and Scala 2.12).

…failures

… because we can't find the coordinator

…e copy is actually finished even when the copier is bounced.

…sary error messages to print on jenkins

asfbot · 2017-05-30T21:37:13Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/4554/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2017-05-30T21:44:57Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/4569/
Test PASSed (JDK 7 and Scala 2.11).

…tderr and stdout from transactional message copier

apurvam · 2017-05-30T22:48:37Z

A quick update:

The tests have revealed a bug: The TxnOffsetCommitResponse could have a REQUEST_TIMED_OUT error, which is retriable. The client did not handle this as a retriable error but instead considered it fatal and put the transaction manager in error state.

The TransactionalMessageCopier would then get a KafkaException on the next transactional method it invoked. It would catch this exception, and log it to stderr, and exit. So the copy process would never finish.

Unfortunately, due to an oversight in the python test code, we never collected the stderr stream from the message copier, so there was nothing suspicious in the logs collected. I only found this out because very occasionally the background sender thread would run before the process was shut down, detect that the transaction manager is in error state, and log it. Since we collected the transactional_message_copier logs (ie. the actual producer logs), I could get alerted to this issue .

I have now updated the code to both handle the REQUEST_TIMED_OUT error on the TxnOffsetCommitRequest (just like the ConsumerCoordinator does for the regular OffsetCommitRequest), and also collect the stderr log from the TransactionalMessageCopier. My testing so far shows that the test is stable after these changes.

asfbot · 2017-05-30T23:32:14Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/4580/
Test PASSed (JDK 7 and Scala 2.11).

asfbot · 2017-05-30T23:38:44Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/4565/
Test PASSed (JDK 8 and Scala 2.12).

hachikuji · 2017-05-31T03:36:56Z

tests/kafkatest/services/transactional_message_copier.py

+        return self.message_copy_finished
+
+    def progress_percent(self):
+        if self.remaining < 0:


Might need the lock here since you access to mutable variables.

hachikuji · 2017-05-31T03:41:36Z

tools/src/main/java/org/apache/kafka/tools/TransactionalMessageCopier.java

+                remainingMessages.set(maxMessages - numMessagesProcessed.addAndGet(messagesInCurrentTransaction));
+            }
+        } catch (Exception e) {
+            e.printStackTrace();


In VerifiableProducer and VerifiableConsumer, we pass uncaught exceptions to the argument parser handleError. It would be nice to be consistent.

Isn't that just for parsing arguments though?

Yeah, that might be right. Looks like we just let application errors bubble up. Guess we could do the same here since the exception would get printed anyway. Up to you.

hachikuji · 2017-05-31T03:55:18Z

tests/kafkatest/tests/core/transactions_test.py

+
+    @cluster(num_nodes=8)
+    @matrix(failure_mode=["clean_bounce", "hard_bounce"],
+            bounce_target=["brokers", "clients"])


Are we going to disable the brokers bounce tests?

I don't think so: with the bug fix for KAFKA-5351, they seem to be quite stable.

apurvam · 2017-05-31T18:19:28Z

The latest full system test suite passed, except for the log appender test, which I think is unrelated:

http://testing.confluent.io/confluent-kafka-branch-builder-system-test-results/?prefix=2017-05-31--001.1496235437--apurvam--KAFKA-5281-transactions-system-tests--d7551e9/

apurvam · 2017-05-31T22:10:13Z

The bounce test failures are being fixed in : #3184

apurvam · 2017-06-01T01:04:12Z

The bounce tests seem to be stable after incorporating #3184

…copy as gets very close to the limit on hard bounces

…ting it

asfbot · 2017-06-01T05:51:25Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/4687/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2017-06-01T05:54:17Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/4703/
Test PASSed (JDK 7 and Scala 2.11).

hachikuji

Thanks for the tests and fixes. LGTM.

Author: Apurva Mehta <apurva@confluent.io> Reviewers: Jason Gustafson <jason@confluent.io> Closes #3149 from apurvam/KAFKA-5281-transactions-system-tests (cherry picked from commit 1959835) Signed-off-by: Jason Gustafson <jason@confluent.io>

ijuma reviewed May 26, 2017

View reviewed changes

Apurva Mehta added 13 commits May 30, 2017 13:13

Initial commit of TransactionalMessageCopier program

23a924f

Initial commit of transactions system test

51bef2d

WIP

5f5dc0c

WIP - Got the basic copy / validate to work. Now to add bouncing and …

da604e2

…failures

Implemented clean and hard bounces of brokers. Hard bounce test fails…

bf4b4c0

… because we can't find the coordinator

Implemented code for bouncing clients as well

92e977e

Fix checkstyle issues and the hard bounce test. All 4 tests pass now

f4f5882

Kill clients properly

80d1ade

Fix checkstyle

22f0402

WIP commit: python test code is still flaky

df25aa1

fix some flakiness in the test code.. mainly around detecting that th…

c8a5abb

…e copy is actually finished even when the copier is bounced.

Add file I forgot to save

edab93d

Don't collect logs the message copier doesn't produce. Causes unneces…

304b872

…sary error messages to print on jenkins

Apurva Mehta added 2 commits May 30, 2017 13:15

Miscellaneous logging and logic improvements

361439e

Pass the isolation level config only to 0.11 and newer consumers

e26e872

apurvam force-pushed the KAFKA-5281-transactions-system-tests branch from 4df0f6b to e26e872 Compare May 30, 2017 20:16

Apurva Mehta added 2 commits May 30, 2017 14:46

Handle a RequestTimedOut error from TxnOffsetCommitRequest. Capture s…

3afa292

…tderr and stdout from transactional message copier

Always collect stderr and stdout for transactional message copier

d7551e9

hachikuji reviewed May 31, 2017

View reviewed changes

Apurva Mehta added 3 commits May 31, 2017 21:03

Address some review comments, bump the wait time out for the message …

2511ac8

…copy as gets very close to the limit on hard bounces

Let exception bubble up to the top level instead of catching and prin…

eea745b

…ting it

Enable client bounces as well

4c23ba4

hachikuji approved these changes Jun 1, 2017

View reviewed changes

asfgit closed this in 1959835 Jun 1, 2017

KAFKA-5281: System tests for transactions #3149

KAFKA-5281: System tests for transactions #3149

Conversation

apurvam commented May 26, 2017

asfbot commented May 26, 2017

asfbot commented May 26, 2017

asfbot commented May 26, 2017

asfbot commented May 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asfbot commented May 26, 2017

asfbot commented May 26, 2017

asfbot commented May 27, 2017

asfbot commented May 27, 2017

asfbot commented May 27, 2017

asfbot commented May 27, 2017

apurvam commented May 27, 2017

asfbot commented May 27, 2017

asfbot commented May 27, 2017

apurvam commented May 28, 2017

asfbot commented May 28, 2017

asfbot commented May 28, 2017

asfbot commented May 30, 2017

asfbot commented May 30, 2017

apurvam commented May 30, 2017

asfbot commented May 30, 2017

asfbot commented May 30, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

apurvam commented May 31, 2017

apurvam commented May 31, 2017

apurvam commented Jun 1, 2017

asfbot commented Jun 1, 2017

asfbot commented Jun 1, 2017

hachikuji left a comment

Choose a reason for hiding this comment