KAFKA-4916: test streams with brokers failing (for 0.10.2) #2793

enothereska · 2017-04-03T10:53:44Z

No description provided.

enothereska · 2017-04-03T10:53:58Z

0.10.2 version of #2719

asfbot · 2017-04-03T11:17:23Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/2628/
Test FAILed (JDK 8 and Scala 2.12).

asfbot · 2017-04-03T11:23:03Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/2628/
Test FAILed (JDK 7 and Scala 2.10).

asfbot · 2017-04-03T11:36:45Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/2632/
Test PASSed (JDK 8 and Scala 2.11).

enothereska · 2017-04-03T12:23:52Z

Known Kafka failures.

enothereska · 2017-04-03T17:52:42Z

retest this please

asfbot · 2017-04-03T17:52:58Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/2651/
Test FAILed (JDK 8 and Scala 2.12).

asfbot · 2017-04-03T17:53:25Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/2651/
Test FAILed (JDK 7 and Scala 2.10).

asfbot · 2017-04-03T17:53:43Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/2655/
Test FAILed (JDK 8 and Scala 2.11).

enothereska · 2017-04-03T17:56:51Z

Jenkins seems down

enothereska · 2017-04-03T17:57:26Z

retest this please

asfbot · 2017-04-03T17:57:39Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/2654/
Test FAILed (JDK 8 and Scala 2.12).

asfbot · 2017-04-03T17:57:46Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/2654/
Test FAILed (JDK 7 and Scala 2.10).

asfbot · 2017-04-03T18:27:28Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/2658/
Test FAILed (JDK 8 and Scala 2.11).

enothereska · 2017-04-03T21:18:36Z

retest this please

asfbot · 2017-04-03T21:47:03Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/2662/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2017-04-03T21:50:59Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/2662/
Test PASSed (JDK 7 and Scala 2.10).

asfbot · 2017-04-04T00:19:17Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/2666/
Test FAILed (JDK 8 and Scala 2.11).

asfbot · 2017-04-04T21:37:24Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/2716/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2017-04-04T21:39:53Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/2716/
Test FAILed (JDK 7 and Scala 2.10).

asfbot · 2017-04-04T21:39:54Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/2720/
Test PASSed (JDK 8 and Scala 2.11).

ewencp

@enothereska There's a bunch of small stuff from other PRs that I can't easily judge whether they are safe in a backport. (The significant different diffstat was a dead giveaway, and differences in the licenses are almost certainly best resolved to ours when cherry-picking unless you were making changes to licenses).

We should either revert those changes or have someone conclude that they are safe to combine. And since some of them involve system tests, it might warrant running the tests before merging.

ewencp · 2017-04-04T22:51:39Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/InternalTopicManager.java

@@ -95,15 +114,17 @@ public void close() {
    private Map<InternalTopicConfig, Integer> validateTopicPartitions(final Map<InternalTopicConfig, Integer> topicsPartitionsMap,
                                                                      final Map<String, Integer> existingTopicNamesPartitions) {
        final Map<InternalTopicConfig, Integer> topicsToBeCreated = new HashMap<>();
-        for (InternalTopicConfig topic: topicsPartitionsMap.keySet()) {
+        for (Map.Entry<InternalTopicConfig, Integer> entry : topicsPartitionsMap.entrySet()) {


This appears to be unrelated to the original patch, it seems to be from findbugs cleanup

Good catch, thanks.

ewencp · 2017-04-04T22:52:05Z

streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java

        // flush state
        firstException.compareAndSet(null, flushAllState());
        // only commit after all state has been flushed and there hasn't been an exception
        if (firstException.get() == null) {
-            firstException.set(commitOffsets());
+            // TODO: currently commit failures will not be thrown to users


This also seems unrelated. It's in another patch that's being backported anyway, but probably shouldn't have made it into a cherry-pick.

ewencp · 2017-04-04T22:52:51Z

streams/src/test/java/org/apache/kafka/streams/tests/SmokeTestClient.java

@@ -211,7 +227,17 @@ public Double apply(Long value1, Long value2) {
                    "cntByCnt"
        ).to(stringSerde, longSerde, "tagg");

-        return new KafkaStreams(builder, props);
+        final KafkaStreams streamsClient = new KafkaStreams(builder, props);


Also seems unrelated, this is from a patch for KAFKA-4885

You have eagle eyes :) Good catch again.

In this case I actually want the new change since I rely on exception handlers to catch problems.

ewencp · 2017-04-04T22:54:22Z

streams/src/test/java/org/apache/kafka/streams/tests/SmokeTestDriver.java

@@ -169,7 +172,7 @@ public void run() {
                    public void onCompletion(final RecordMetadata metadata, final Exception exception) {
                        if (exception != null) {
                            exception.printStackTrace();
-                            System.exit(-1);
+                            System.exit(1);


This looks like half the change from KAFKA-4039?

Probably harmless but reverting back.

ewencp · 2017-04-04T22:55:18Z

tests/kafkatest/services/streams.py

@@ -50,6 +50,7 @@ def __init__(self, test_context, kafka, streams_class_name, user_test_args):
        self.kafka = kafka
        self.args = {'streams_class_name': streams_class_name,
                     'user_test_args': user_test_args}
+        self.log_level = "DEBUG"


From KAFKA-4702?

Yeah, it's a good time to turn on logging in 0.10.2 to help debug. So this is good.

ewencp · 2017-04-04T22:55:19Z

tests/kafkatest/services/streams.py

@@ -94,7 +95,7 @@ def abortThenRestart(self):
            self.logger.info("Restarting Kafka Streams on " + str(node.account))
            self.start_node(node)

-    def wait(self, timeout_sec=360):
+    def wait(self, timeout_sec=1440):


From KAFKA-4843?

Yeah I consciously upped this to match trunk since the new test also requires it. So this is good.

ewencp · 2017-04-04T22:56:11Z

tests/kafkatest/tests/streams/streams_smoke_test.py

@@ -27,7 +27,7 @@ class StreamsSmokeTest(KafkaTest):
    """

    def __init__(self, test_context):
-        super(StreamsSmokeTest, self).__init__(test_context, num_zk=1, num_brokers=2, topics={
+        super(StreamsSmokeTest, self).__init__(test_context, num_zk=1, num_brokers=3, topics={


From KAFKA-4716?

It's actually needed for this PR since this PR changes the replication factor to 3 in SmokeTestClient.java. It just so happened that in trunk we had already done that change, so it appears as if it is part of KAFKA_4716, but it's part of this PR.

ewencp · 2017-04-05T01:45:16Z

@enothereska Might be easier to do the cherry-pick cleanly now that it's been squashed to a single commit on trunk. It looks like it's just 3 relatively small conflicts -- only reason I didn't resolve and commit myself is that I wasn't sure if we also wanted/needed to run system tests against the cherry-picked version given the size of the change.

enothereska · 2017-04-05T05:42:17Z

Thanks @ewencp I'll double check. Ran streams test and passed: https://jenkins.confluent.io/job/system-test-kafka-branch-builder-2/245/

…10.2

enothereska · 2017-04-05T12:48:08Z

@ewencp I'm getting on plane, don't commit until I ack since I haven't had a chance to look yet. Thanks.

asfbot · 2017-04-05T13:33:32Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/2745/
Test FAILed (JDK 8 and Scala 2.11).

asfbot · 2017-04-05T13:40:01Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/2741/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2017-04-05T13:46:19Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/2741/
Test PASSed (JDK 7 and Scala 2.10).

asfbot · 2017-04-06T10:35:11Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/2785/
Test PASSed (JDK 8 and Scala 2.11).

asfbot · 2017-04-06T10:35:19Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/2781/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2017-04-06T10:40:11Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/2781/
Test PASSed (JDK 7 and Scala 2.10).

asfbot · 2017-04-06T12:53:56Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/2787/
Test PASSed (JDK 8 and Scala 2.11).

asfbot · 2017-04-06T13:11:55Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/2783/
Test PASSed (JDK 7 and Scala 2.10).

asfbot · 2017-04-06T13:22:23Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/2783/
Test PASSed (JDK 8 and Scala 2.12).

enothereska · 2017-04-06T16:53:36Z

@ewencp this is good to go now. I did a test last night https://jenkins.confluent.io/job/system-test-kafka-branch-builder-2/246/. I have another scheduled waiting on branch builder, but I think it's good to go.

ewencp

LGTM

Author: Eno Thereska <eno@confluent.io> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #2793 from enothereska/KAFKA-4916-0.10.2

0.10.2 cherrypick

8bc7aa9

enothereska mentioned this pull request Apr 4, 2017

KAFKA-4916: test streams with brokers failing #2719

Closed

Remove unnecessary call

35f5be5

ewencp reviewed Apr 4, 2017

View reviewed changes

Merge remote-tracking branch 'apache-kafka/0.10.2' into KAFKA-4916-0.…

b674d7a

…10.2

Eno Thereska added 2 commits April 6, 2017 02:37

Revert some unwanted minor changes

c9c2010

Spacing

934d892

Rebase

6f24af5

ewencp approved these changes Apr 6, 2017

View reviewed changes

asfgit pushed a commit that referenced this pull request Apr 6, 2017

KAFKA-4916: test streams with brokers failing

51ae9c2

Author: Eno Thereska <eno@confluent.io> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #2793 from enothereska/KAFKA-4916-0.10.2

enothereska closed this Apr 6, 2017

enothereska deleted the KAFKA-4916-0.10.2 branch April 6, 2017 22:06

KAFKA-4916: test streams with brokers failing (for 0.10.2) #2793

KAFKA-4916: test streams with brokers failing (for 0.10.2) #2793

Conversation

enothereska commented Apr 3, 2017

enothereska commented Apr 3, 2017

asfbot commented Apr 3, 2017

asfbot commented Apr 3, 2017

asfbot commented Apr 3, 2017

enothereska commented Apr 3, 2017

enothereska commented Apr 3, 2017

asfbot commented Apr 3, 2017

asfbot commented Apr 3, 2017

asfbot commented Apr 3, 2017

enothereska commented Apr 3, 2017 • edited Loading

enothereska commented Apr 3, 2017

asfbot commented Apr 3, 2017

asfbot commented Apr 3, 2017

asfbot commented Apr 3, 2017

enothereska commented Apr 3, 2017

asfbot commented Apr 3, 2017

asfbot commented Apr 3, 2017

asfbot commented Apr 4, 2017

asfbot commented Apr 4, 2017

asfbot commented Apr 4, 2017

asfbot commented Apr 4, 2017

ewencp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ewencp commented Apr 5, 2017

enothereska commented Apr 5, 2017 • edited Loading

enothereska commented Apr 5, 2017

asfbot commented Apr 5, 2017

asfbot commented Apr 5, 2017

asfbot commented Apr 5, 2017

asfbot commented Apr 6, 2017

asfbot commented Apr 6, 2017

asfbot commented Apr 6, 2017

asfbot commented Apr 6, 2017

asfbot commented Apr 6, 2017

asfbot commented Apr 6, 2017

enothereska commented Apr 6, 2017

ewencp left a comment

Choose a reason for hiding this comment

enothereska commented Apr 3, 2017 •

edited

Loading

enothereska commented Apr 5, 2017 •

edited

Loading