KAFKA-4916: test streams with brokers failing #2719

enothereska · 2017-03-21T18:48:08Z

Several fixes for handling broker failures:

default replication value for internal topics is now 3 in test itself (not in streams code, that will require a KIP.
streams producer waits for acks from all replicas in test itself (not in streams code, that will require a KIP.
backoff time for streams client to try again after a failure to contact controller.
fix bug related to state store locks (this helps in multi-threaded scenarios)
fix related to catching exceptions property for network errors.
system test for all the above

enothereska · 2017-03-21T18:48:55Z

Already getting an error from this basic test, that needs to be fixed before this PR goes in.
org.apache.kafka.streams.errors.StreamsException: Could not create internal topics.
at org.apache.kafka.streams.processor.internals.InternalTopicManager.makeReady(InternalTopicManager.java:69). Looks like this might not be a real error though, since the replication factor of the internal topics is not maintained. Adding another broker and re-testing.

asfbot · 2017-03-21T19:38:57Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/2303/
Test PASSed (JDK 7 and Scala 2.10).

asfbot · 2017-03-21T20:00:36Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/2307/
Test PASSed (JDK 8 and Scala 2.11).

asfbot · 2017-03-21T21:54:29Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/2303/
Test FAILed (JDK 8 and Scala 2.12).

asfbot · 2017-03-22T10:46:43Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/2320/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2017-03-22T10:52:45Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/2320/
Test PASSed (JDK 7 and Scala 2.10).

asfbot · 2017-03-22T11:18:38Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/2324/
Test PASSed (JDK 8 and Scala 2.11).

enothereska · 2017-03-22T12:30:36Z

@dguy want to have a look? Thanks.

dguy

Thanks @enothereska just left a couple of questions

dguy · 2017-03-22T12:36:06Z

streams/src/test/java/org/apache/kafka/streams/tests/SmokeTestDriver.java

@@ -136,6 +136,9 @@ public void run() {
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafka);
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class);
+        props.put(ProducerConfig.RETRIES_CONFIG, 5);


I wonder if we should set this higher? is there any harm in setting it to Integer.MAX_VALUE?

dguy · 2017-03-22T12:39:55Z

tests/kafkatest/tests/streams/streams_broker_bounce_test.py

+        # Pick a random topic and bounce it's leader
+        topic_index = randint(0, len(self.topics.keys()) - 1)
+        topic = self.topics.keys()[topic_index]
+        failures[failure_mode](self, topic, broker_type)


Do you think we should maybe kill some more brokers? Or is that going to be too non-deterministic in terms of test failures?

We probably can. The main problem I have right now is how to sidestep all the Kafka corner cases when it comes to failures while still showing that streams is resilient. Stay tuned.

asfbot · 2017-03-22T13:15:24Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/2321/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2017-03-22T13:17:05Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/2325/
Test PASSed (JDK 8 and Scala 2.11).

asfbot · 2017-03-22T15:31:32Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/2321/
Test FAILed (JDK 7 and Scala 2.10).

enothereska · 2017-03-22T16:41:54Z

@dguy Making as WiP again since 2 tests are failing and I need to investigate.

[2017-03-22 16:25:05,575] ERROR stream-thread [SmokeTest-b25363c8-5bce-4d72-8c51-8fe0b49715f6-StreamThread-1] Streams application error during processing: (org.apache.kafka.streams.processor.internals.StreamThread)
java.lang.IllegalStateException: Attempt to send a request to node 1 which is not ready.
at org.apache.kafka.clients.NetworkClient.doSend(NetworkClient.java:284)
at org.apache.kafka.clients.NetworkClient.send(NetworkClient.java:265)
at org.apache.kafka.streams.processor.internals.StreamsKafkaClient.sendRequest(StreamsKafkaClient.java:238)
at org.apache.kafka.streams.processor.internals.StreamsKafkaClient.createTopics(StreamsKafkaClient.java:170)
at org.apache.kafka.streams.processor.internals.InternalTopicManager.makeReady(InternalTopicManager.java:63)
at org.apache.kafka.streams.processor.internals.StreamPartitionAssignor.prepareTopic(StreamPartitionAssignor.java:615)
at org.apache.kafka.streams.processor.internals.StreamPartitionAssignor.assign(StreamPartitionAssignor.java:445)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.performAssignment(ConsumerCoordinator.java:347)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onJoinLeader(AbstractCoordinator.java:505)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.access$1100(AbstractCoordinator.java:93)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:455)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$JoinGroupResponseHandler.handle(AbstractCoordinator.java:437)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:788)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:769)
at org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:190)
at org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:153)
at org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:120)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.fireCompletion(ConsumerNetworkClient.java:498)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.firePendingCompletedRequests(ConsumerNetworkClient.java:342)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:251)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:167)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:351)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:307)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:294)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1033)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:999)
at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:542)
at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:326)

…unce-test

asfbot · 2017-03-31T09:53:26Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/2572/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2017-03-31T09:53:40Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/2572/
Test PASSed (JDK 7 and Scala 2.10).

asfbot · 2017-03-31T10:26:26Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/2576/
Test PASSed (JDK 8 and Scala 2.11).

enothereska · 2017-04-01T15:55:28Z

https://jenkins.confluent.io/job/system-test-kafka-branch-builder/820/ is green

enothereska · 2017-04-03T10:54:47Z

@mjsax @dguy can I get a final approve so that @ijuma or others can check in? Also just opened 0.10.2 cherry picked version of this PR. Thanks.

dguy

Thanks @enothereska, LGTM

asfbot · 2017-04-03T12:54:18Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/2634/
Test FAILed (JDK 8 and Scala 2.11).

asfbot · 2017-04-03T12:54:54Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/2630/
Test FAILed (JDK 8 and Scala 2.12).

asfbot · 2017-04-03T12:55:36Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/2630/
Test FAILed (JDK 7 and Scala 2.10).

enothereska · 2017-04-03T13:06:56Z

retest this please

asfbot · 2017-04-03T13:37:00Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/2631/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2017-04-03T13:37:26Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/2631/
Test PASSed (JDK 7 and Scala 2.10).

asfbot · 2017-04-03T13:49:37Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/2635/
Test FAILed (JDK 8 and Scala 2.11).

enothereska · 2017-04-03T16:55:06Z

Env failure, unrelated to PR

mjsax

LGTM

enothereska · 2017-04-03T21:21:08Z

@gwenshap @ewencp @ijuma this can go in, thanks. Also the 0.10.2 version of this: #2793. Thanks.

asfbot · 2017-04-04T21:39:10Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/2717/
Test PASSed (JDK 8 and Scala 2.12).

asfbot · 2017-04-04T21:41:51Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/2721/
Test PASSed (JDK 8 and Scala 2.11).

asfbot · 2017-04-04T22:10:17Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-jdk7-scala2.10/2717/
Test FAILed (JDK 7 and Scala 2.10).

Eno Thereska and others added 12 commits March 5, 2017 14:57

Temporarily use only 2 relevant tests for branch builder

187f9b6

Undo previous

87f06a8

Merge remote-tracking branch 'apache-kafka/trunk' into trunk

978f925

Merge remote-tracking branch 'apache-kafka/trunk' into trunk

57856b0

Merge remote-tracking branch 'apache-kafka/trunk' into trunk

5086dd7

Merge remote-tracking branch 'apache-kafka/trunk' into trunk

40394f3

Merge remote-tracking branch 'apache-kafka/trunk' into trunk

1ee01c1

Merge remote-tracking branch 'apache-kafka/trunk' into trunk

0d30ba4

Merge remote-tracking branch 'apache-kafka/trunk' into trunk

7dae290

Merge remote-tracking branch 'apache-kafka/trunk' into trunk

7252e62

Merge branch 'trunk' of github.com:enothereska/kafka into trunk

4eb3670

Initial test

9df1010

Added more tests

829f57a

Tighten test options to minimize data loss or duplicates

f702616

enothereska changed the title ~~KAFKA-4916: test streams with brokers failing [WiP]~~ KAFKA-4916: test streams with brokers failing Mar 22, 2017

dguy reviewed Mar 22, 2017

View reviewed changes

Added tests with failing more than one broker

7f68745

enothereska changed the title ~~KAFKA-4916: test streams with brokers failing~~ KAFKA-4916: test streams with brokers failing [WiP] Mar 22, 2017

Eno Thereska added 2 commits March 31, 2017 09:23

Merge remote-tracking branch 'origin/trunk' into KAFKA-4916-broker-bo…

0b5743c

…unce-test

Addressed comments

ad1848f

enothereska added 2 commits April 3, 2017 11:21

Merge branch 'trunk' of github.com:enothereska/kafka into trunk

f66bcd3

Merge remote-tracking branch 'apache-kafka/trunk' into trunk

186d7d0

enothereska mentioned this pull request Apr 3, 2017

KAFKA-4916: test streams with brokers failing (for 0.10.2) #2793

Closed

dguy approved these changes Apr 3, 2017

View reviewed changes

enothereska added 2 commits April 3, 2017 13:15

Merge remote-tracking branch 'apache-kafka/trunk' into trunk

bc68fe8

Merge with trunk

bca3a69

mjsax approved these changes Apr 3, 2017

View reviewed changes

Remove unnecessary call

d690183

ewencp approved these changes Apr 5, 2017

View reviewed changes

asfgit closed this in 49f80b2 Apr 5, 2017

enothereska deleted the KAFKA-4916-broker-bounce-test branch April 5, 2017 05:40

KAFKA-4916: test streams with brokers failing #2719

KAFKA-4916: test streams with brokers failing #2719

Conversation

enothereska commented Mar 21, 2017 • edited

enothereska commented Mar 21, 2017 • edited

asfbot commented Mar 21, 2017

asfbot commented Mar 21, 2017

asfbot commented Mar 21, 2017

asfbot commented Mar 22, 2017

asfbot commented Mar 22, 2017

asfbot commented Mar 22, 2017

enothereska commented Mar 22, 2017

dguy left a comment

Choose a reason for hiding this comment

dguy Mar 22, 2017

Choose a reason for hiding this comment

enothereska Mar 22, 2017

Choose a reason for hiding this comment

dguy Mar 22, 2017

Choose a reason for hiding this comment

enothereska Mar 22, 2017

Choose a reason for hiding this comment

asfbot commented Mar 22, 2017

asfbot commented Mar 22, 2017

asfbot commented Mar 22, 2017

enothereska commented Mar 22, 2017 • edited

asfbot commented Mar 31, 2017

asfbot commented Mar 31, 2017

asfbot commented Mar 31, 2017

enothereska commented Apr 1, 2017

enothereska commented Apr 3, 2017

dguy left a comment

Choose a reason for hiding this comment

asfbot commented Apr 3, 2017

asfbot commented Apr 3, 2017

asfbot commented Apr 3, 2017

enothereska commented Apr 3, 2017

asfbot commented Apr 3, 2017

asfbot commented Apr 3, 2017

asfbot commented Apr 3, 2017

enothereska commented Apr 3, 2017

mjsax left a comment

Choose a reason for hiding this comment

enothereska commented Apr 3, 2017 • edited

asfbot commented Apr 4, 2017

asfbot commented Apr 4, 2017

asfbot commented Apr 4, 2017

enothereska commented Mar 21, 2017 •

edited

enothereska commented Mar 21, 2017 •

edited

enothereska commented Mar 22, 2017 •

edited

enothereska commented Apr 3, 2017 •

edited