KAFKA-5362: Follow up to Streams EOS system test #3542

mjsax · 2017-07-18T04:29:47Z

improve tests to get rid of calls to sleep in Python
fixed some flaky test conditions
improve debugging

mjsax · 2017-07-18T04:30:21Z

Call for review @enothereska @dguy @bbejeck @guozhangwang

mjsax · 2017-07-18T04:31:02Z

streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java

@@ -259,7 +259,7 @@ private void setState(final State newState) {
                log.info("{} State transition from {} to {}.", logPrefix, oldState, newState);
            }
            state = newState;
-            if (stateListener != null) {
+            if (stateListener != null && state != oldState) {


\cc @enothereska Just want to bring this to your attention as you recently worked on this.

mjsax · 2017-07-18T04:34:46Z

We got from 2 thread to one and increase number of Streams instances from 2 to 3 to compensate. This is required to make rebalances predicable to allow replacing sleeps with wait-conditions.

Also triggered 25 rounds of system tests: https://jenkins.confluent.io/job/system-test-kafka-branch-builder/987/

asfgit · 2017-07-18T04:39:19Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6137/
Test FAILed (JDK 7 and Scala 2.11).

asfgit · 2017-07-18T04:45:27Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6122/
Test FAILed (JDK 8 and Scala 2.12).

mjsax · 2017-07-18T06:08:50Z

This should got to 0.11.0, too.

Test failures are unrelated. Seems to be a more general Jenkins issues with those test -- I saw multiple reports that this SSL tests failed (guess, re-testing does not help atm). But his PR only does system test stuff anyway.

enothereska · 2017-07-18T08:21:11Z

Seems like there was a system test failure in the above link?

bbejeck · 2017-07-18T19:36:19Z

streams/src/test/java/org/apache/kafka/streams/tests/EosTestClient.java

@@ -90,7 +100,7 @@ private KafkaStreams createKafkaStreams(final File stateDir,
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, APP_ID);
        props.put(StreamsConfig.STATE_DIR_CONFIG, stateDir.toString());
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, kafka);
-        props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 2);
+        props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 1);


bbejeck · 2017-07-18T19:36:55Z

streams/src/test/java/org/apache/kafka/streams/tests/EosTestDriver.java

@@ -54,7 +54,7 @@
 public class EosTestDriver extends SmokeTestUtil {

    private static final int MAX_NUMBER_OF_KEYS = 100;
-    private static final long MAX_IDLE_TIME_MS = 300000L;
+    private static final long MAX_IDLE_TIME_MS = 600000L;


bbejeck · 2017-07-18T20:13:42Z

Some of the changes are exactly what I had suspected during my on-call.

One general question - do we want to consider checking for the actual messages consumed (assuming sent with an incrementing number in the value) as well?

asfgit · 2017-07-24T08:45:09Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6296/
Test FAILed (JDK 7 and Scala 2.11).

asfgit · 2017-07-24T08:48:29Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6280/
Test FAILed (JDK 8 and Scala 2.12).

asfgit · 2017-07-24T11:29:03Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6304/
Test FAILed (JDK 7 and Scala 2.11).

asfgit · 2017-07-24T11:32:13Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6288/
Test FAILed (JDK 8 and Scala 2.12).

asfgit · 2017-07-24T13:22:03Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6292/
Test FAILed (JDK 8 and Scala 2.12).

asfgit · 2017-07-24T13:22:13Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6308/
Test FAILed (JDK 7 and Scala 2.11).

dguy · 2017-07-24T14:21:18Z

retest this please

mjsax · 2017-07-24T14:24:20Z

@dguy No need to trigger retests -- I am playing with this branch to figure out why system tests are still unstable...

asfgit · 2017-07-24T17:57:05Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.11/6311/
Test FAILed (JDK 7 and Scala 2.11).

asfgit · 2017-07-24T18:08:28Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6295/
Test FAILed (JDK 8 and Scala 2.12).

- reduce test runtime by removing sleep calls - improved debugging - minor fix for KafkaStreams state listener callback

- need to check topic-partition position even if no data is returned - should not kill producer on failure - improved debugging

mjsax · 2017-10-02T21:45:05Z

Rebased and extended closing timeout. Triggered system test: https://jenkins.confluent.io/job/system-test-kafka-branch-builder/1090/

mjsax · 2017-10-05T16:29:23Z

Retriggered system test: https://jenkins.confluent.io/job/system-test-kafka-branch-builder/1092/

…utput

dguy · 2017-10-05T18:26:09Z

I've just triggered another build as the last one didn't work for some aws issues: https://jenkins.confluent.io/job/system-test-kafka-branch-builder/1093/

guozhangwang · 2017-10-05T20:13:13Z

1093 still fails.

mjsax · 2017-10-05T20:57:48Z

There was a 5 minute gap in the logs for one Streams instance where nothing happened... Might have been environmental.

Retriggered: https://jenkins.confluent.io/job/system-test-kafka-branch-builder/1094/

dguy · 2017-10-05T21:17:46Z

I had a quick look and it seemed like there may have been some connectivity issues with the brokers and between the brokers

mjsax · 2017-10-06T20:23:20Z

Retest this please

mjsax · 2017-10-06T20:25:08Z

The last run 1094 got more than 100 successful runs -- we never got this many before. I guess we should merge this now. The latest changes increasing the timeouts and adding some flush statements for stdout seems to do the trick.

guozhangwang · 2017-10-06T22:40:22Z

retest this please

- improve tests to get rid of calls to `sleep` in Python - fixed some flaky test conditions - improve debugging Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Damian Guy <damian.guy@gmail.com>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <wangguoz@gmail.com> Closes #3542 from mjsax/failing-eos-system-tests (cherry picked from commit 5106344) Signed-off-by: Guozhang Wang <wangguoz@gmail.com>

guozhangwang · 2017-10-07T00:49:02Z

Merged to trunk and 1.0.

mjsax commented Jul 18, 2017

View reviewed changes

bbejeck reviewed Jul 18, 2017

View reviewed changes

mjsax force-pushed the failing-eos-system-tests branch from 26db0ef to 8b45153 Compare July 24, 2017 11:25

mjsax force-pushed the failing-eos-system-tests branch from 8b45153 to cf2e286 Compare July 24, 2017 13:18

mjsax force-pushed the failing-eos-system-tests branch from cf2e286 to 9b624b6 Compare July 24, 2017 14:22

mjsax force-pushed the failing-eos-system-tests branch from 9b624b6 to c9ed7e5 Compare August 30, 2017 00:37

mjsax added 4 commits October 2, 2017 14:26

KAFKA-5362: Follow up EOS system test

e03b1d9

- reduce test runtime by removing sleep calls - improved debugging - minor fix for KafkaStreams state listener callback

MINOR: fix flaky Stream EOS system test

5637301

- need to check topic-partition position even if no data is returned - should not kill producer on failure - improved debugging

Improve debugging

e6d55fb

Increased close timeout

c6a6e32

mjsax force-pushed the failing-eos-system-tests branch from c9ed7e5 to c6a6e32 Compare October 2, 2017 21:38

Updated timeout to same value as in streams_eos_test.py

382f447

Adding some flushs for stdout to make sure Python code will receive o…

133fb4c

…utput

asfgit closed this in 5106344 Oct 7, 2017

mjsax deleted the failing-eos-system-tests branch October 9, 2017 18:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-5362: Follow up to Streams EOS system test #3542

KAFKA-5362: Follow up to Streams EOS system test #3542

mjsax commented Jul 18, 2017

mjsax commented Jul 18, 2017

mjsax Jul 18, 2017

mjsax commented Jul 18, 2017

asfgit commented Jul 18, 2017

asfgit commented Jul 18, 2017

mjsax commented Jul 18, 2017

enothereska commented Jul 18, 2017

bbejeck Jul 18, 2017

bbejeck Jul 18, 2017

bbejeck commented Jul 18, 2017

asfgit commented Jul 24, 2017

asfgit commented Jul 24, 2017

asfgit commented Jul 24, 2017

asfgit commented Jul 24, 2017

asfgit commented Jul 24, 2017

asfgit commented Jul 24, 2017

dguy commented Jul 24, 2017

mjsax commented Jul 24, 2017

asfgit commented Jul 24, 2017

asfgit commented Jul 24, 2017

mjsax commented Oct 2, 2017

mjsax commented Oct 5, 2017

dguy commented Oct 5, 2017 •

edited

Loading

guozhangwang commented Oct 5, 2017

mjsax commented Oct 5, 2017

dguy commented Oct 5, 2017

mjsax commented Oct 6, 2017

mjsax commented Oct 6, 2017

guozhangwang commented Oct 6, 2017

guozhangwang commented Oct 7, 2017

KAFKA-5362: Follow up to Streams EOS system test #3542

KAFKA-5362: Follow up to Streams EOS system test #3542

Conversation

mjsax commented Jul 18, 2017

mjsax commented Jul 18, 2017

mjsax Jul 18, 2017

Choose a reason for hiding this comment

mjsax commented Jul 18, 2017

asfgit commented Jul 18, 2017

asfgit commented Jul 18, 2017

mjsax commented Jul 18, 2017

enothereska commented Jul 18, 2017

bbejeck Jul 18, 2017

Choose a reason for hiding this comment

bbejeck Jul 18, 2017

Choose a reason for hiding this comment

bbejeck commented Jul 18, 2017

asfgit commented Jul 24, 2017

asfgit commented Jul 24, 2017

asfgit commented Jul 24, 2017

asfgit commented Jul 24, 2017

asfgit commented Jul 24, 2017

asfgit commented Jul 24, 2017

dguy commented Jul 24, 2017

mjsax commented Jul 24, 2017

asfgit commented Jul 24, 2017

asfgit commented Jul 24, 2017

mjsax commented Oct 2, 2017

mjsax commented Oct 5, 2017

dguy commented Oct 5, 2017 • edited Loading

guozhangwang commented Oct 5, 2017

mjsax commented Oct 5, 2017

dguy commented Oct 5, 2017

mjsax commented Oct 6, 2017

mjsax commented Oct 6, 2017

guozhangwang commented Oct 6, 2017

guozhangwang commented Oct 7, 2017

dguy commented Oct 5, 2017 •

edited

Loading