[SPARK-4964][Streaming][Kafka] More updates to Exactly-once Kafka stream #4384

tdas · 2015-02-05T04:07:54Z

Changes

Added example
Added a critical unit test that verifies that offset ranges can be recovered through checkpoints

Might add more changes.

SparkQA · 2015-02-05T04:12:30Z

Test build #26822 has started for PR 4384 at commit 4986784.

This patch merges cleanly.

koeninger · 2015-02-05T04:16:36Z

...les/scala-2.10/src/main/scala/org/apache/spark/examples/streaming/DirectKafkaWordCount.scala

+/**
+ * Consumes messages from one or more topics in Kafka and does wordcount.
+ * Usage: DirectKafkaWordCount <brokers> <topics>
+ *   <brokers> is a list of one or more zookeeper servers that make quorum


These are kafka servers, not zookeeper servers

Right. Thanks!

tdas · 2015-02-05T04:27:32Z

@koeninger Can you please take a look.
@pwendell (Optional) Take a look if you are interested.

tdas · 2015-02-05T04:29:21Z

external/kafka/src/test/scala/org/apache/spark/streaming/kafka/DirectKafkaStreamSuite.scala

+import org.apache.spark.streaming.dstream.{DStream, InputDStream}
+import org.apache.spark.util.Utils
+
+class DirectKafkaStreamSuite extends KafkaStreamSuiteBase


Just renamed the file and testsuite class from KafkaDirectStreamSuite to DirectKafkaStreamSuite, but Git/Github considered it to be a move. The first unit test is unmodified.

SparkQA · 2015-02-05T05:03:17Z

Test build #26822 has finished for PR 4384 at commit 4986784.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-05T05:03:21Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26822/
Test FAILed.

SparkQA · 2015-02-05T05:48:49Z

Test build #586 has started for PR 4384 at commit 4986784.

This patch merges cleanly.

SparkQA · 2015-02-05T07:00:29Z

Test build #586 has finished for PR 4384 at commit 4986784.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

… docs.

SparkQA · 2015-02-06T01:22:57Z

Test build #26878 has started for PR 4384 at commit 50f2b56.

This patch merges cleanly.

SparkQA · 2015-02-06T02:14:00Z

Test build #26878 has finished for PR 4384 at commit 50f2b56.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-06T02:14:04Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26878/
Test FAILed.

SparkQA · 2015-02-06T03:02:42Z

Test build #26888 has started for PR 4384 at commit bb65232.

This patch merges cleanly.

SparkQA · 2015-02-06T03:37:58Z

Test build #26891 has started for PR 4384 at commit e4abf69.

This patch merges cleanly.

SparkQA · 2015-02-06T04:14:26Z

Test build #26888 has finished for PR 4384 at commit bb65232.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-06T04:14:30Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26888/
Test PASSed.

SparkQA · 2015-02-06T04:46:25Z

Test build #26891 has finished for PR 4384 at commit e4abf69.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-06T04:46:29Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26891/
Test PASSed.

SparkQA · 2015-02-06T07:19:53Z

Test build #589 has started for PR 4384 at commit e4abf69.

This patch merges cleanly.

SparkQA · 2015-02-06T09:09:27Z

Test build #589 has finished for PR 4384 at commit e4abf69.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

koeninger · 2015-02-06T16:27:21Z

external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaRDD.scala

- * @param batch Each KafkaRDDPartition in the batch corresponds to a
- *   range of offsets for a given Kafka topic/partition
+ * configuration parameters</a>. Requires "metadata.broker.list" or "bootstrap.servers" to be set
+ * with Kafka broker(s) specified in host1:port1,host2:port2 form.


You removed the "not zookeeper servers" note here, but left it in for the other methods that take kafkaParams

The only reason we were writing "not zookeepers" is to make the difference with the earlier stream clear, for people who want to switch from the old one or the new. That applies to the public API. This is internal private API. I can add it back, no issues.

koeninger · 2015-02-06T16:50:48Z

Thanks for adding the java friendly kafka uitls methods.

Your original reason for wanting Array[Leader] rather than Map[TopicAndPartition, Broker] was for java compatibility.

But since there are separate scala and java-specific KafkaUtils.createRDD methods now, couldn't the scala one take a Map and the java one take a JMap?

As it stands currently, what is the meaning if someone passes in an array with multiple Leader objects that have the same topic and partition but different broker?

The first thing we do with the array of leaders is convert it to a map...seems better to just take a map and avoid both the possibility of confusion and the extra conversion.

koeninger · 2015-02-06T16:52:33Z

external/kafka/src/main/scala/org/apache/spark/streaming/kafka/OffsetRange.scala

+ * :: Experimental ::
+ * Represents any object that has a collection of [[OffsetRange]]s. This can be used access the
+ * offset ranges in RDDs generated by the direct Kafka DStream (see
+ * [[KafkaUtils.createDirectStream()]]).


It's probably good for the doc for createDirectStream to link to here, in addition to the other way around

Good call. Let me add the references.

SparkQA · 2015-02-06T22:42:36Z

Test build #26966 has started for PR 4384 at commit 26df23c.

This patch merges cleanly.

SparkQA · 2015-02-06T23:54:44Z

Test build #26966 has finished for PR 4384 at commit 26df23c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-06T23:54:48Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26966/
Test PASSed.

tdas · 2015-02-09T02:23:20Z

@pwendell Could you take a look at the scala docs?

SparkQA · 2015-02-09T04:12:56Z

Test build #27078 has started for PR 4384 at commit 83d0402.

This patch merges cleanly.

SparkQA · 2015-02-09T05:26:26Z

Test build #27078 has finished for PR 4384 at commit 83d0402.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public final class JavaDirectKafkaWordCount

AmplabJenkins · 2015-02-09T05:26:30Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27078/
Test PASSed.

pwendell · 2015-02-10T03:09:27Z

external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala

+
+  /**
+   * :: Experimental ::
+   * Create an input stream that pulls messages from a Kafka Broker. This stream can guarantee


Does the other one pull messages from a Kafka Broker as well? It might be good to focus on something that is distinctive.

what about saying "without using receivers" at the end?

SparkQA · 2015-02-10T05:42:28Z

Test build #27178 has started for PR 4384 at commit 7c931c3.

This patch merges cleanly.

Changes - Added example - Added a critical unit test that verifies that offset ranges can be recovered through checkpoints Might add more changes. Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #4384 from tdas/new-kafka-fixes and squashes the following commits: 7c931c3 [Tathagata Das] Small update 3ed9284 [Tathagata Das] updated scala doc 83d0402 [Tathagata Das] Added JavaDirectKafkaWordCount example. 26df23c [Tathagata Das] Updates based on PR comments from Cody e4abf69 [Tathagata Das] Scala doc improvements and stuff. bb65232 [Tathagata Das] Fixed test bug and refactored KafkaStreamSuite 50f2b56 [Tathagata Das] Added Java API and added more Scala and Java unit tests. Also updated docs. e73589c [Tathagata Das] Minor changes. 4986784 [Tathagata Das] Added unit test to kafka offset recovery 6a91cab [Tathagata Das] Added example (cherry picked from commit c151346) Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>

SparkQA · 2015-02-10T06:57:30Z

Test build #27178 has finished for PR 4384 at commit 7c931c3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-10T06:57:34Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27178/
Test PASSed.

…Suite The test was incorrect. Instead of counting the number of records, it counted the number of partitions of RDD generated by DStream. Which is not its intention. I will be testing this patch multiple times to understand its flakiness. PS: This was caused by my refactoring in #4384 koeninger check it out. Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #4597 from tdas/kafka-flaky-test and squashes the following commits: d236235 [Tathagata Das] Unignored last test. e9a1820 [Tathagata Das] fix test (cherry picked from commit 3912d33) Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>

…Suite The test was incorrect. Instead of counting the number of records, it counted the number of partitions of RDD generated by DStream. Which is not its intention. I will be testing this patch multiple times to understand its flakiness. PS: This was caused by my refactoring in #4384 koeninger check it out. Author: Tathagata Das <tathagata.das1565@gmail.com> Closes #4597 from tdas/kafka-flaky-test and squashes the following commits: d236235 [Tathagata Das] Unignored last test. e9a1820 [Tathagata Das] fix test

tdas added 2 commits February 4, 2015 19:56

Added example

6a91cab

Added unit test to kafka offset recovery

4986784

koeninger reviewed Feb 5, 2015
View reviewed changes

tdas reviewed Feb 5, 2015
View reviewed changes

Minor changes.

e73589c

Added Java API and added more Scala and Java unit tests. Also updated…

50f2b56

… docs.

Fixed test bug and refactored KafkaStreamSuite

bb65232

Scala doc improvements and stuff.

e4abf69

koeninger reviewed Feb 6, 2015
View reviewed changes

Updates based on PR comments from Cody

26df23c

Added JavaDirectKafkaWordCount example.

83d0402

pwendell reviewed Feb 10, 2015
View reviewed changes

tdas added 2 commits February 9, 2015 21:37

updated scala doc

3ed9284

Small update

7c931c3

asfgit closed this in c151346 Feb 10, 2015

tdas mentioned this pull request Feb 13, 2015

[SPARK-5731][Streaming][Test] Fix incorrect test in DirectKafkaStreamSuite #4597

Closed

[SPARK-4964][Streaming][Kafka] More updates to Exactly-once Kafka stream #4384

[SPARK-4964][Streaming][Kafka] More updates to Exactly-once Kafka stream #4384

Conversation

tdas commented Feb 5, 2015

SparkQA commented Feb 5, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tdas commented Feb 5, 2015

Choose a reason for hiding this comment

SparkQA commented Feb 5, 2015

AmplabJenkins commented Feb 5, 2015

SparkQA commented Feb 5, 2015

SparkQA commented Feb 5, 2015

SparkQA commented Feb 6, 2015

SparkQA commented Feb 6, 2015

AmplabJenkins commented Feb 6, 2015

SparkQA commented Feb 6, 2015

SparkQA commented Feb 6, 2015

SparkQA commented Feb 6, 2015

AmplabJenkins commented Feb 6, 2015

SparkQA commented Feb 6, 2015

AmplabJenkins commented Feb 6, 2015

SparkQA commented Feb 6, 2015

SparkQA commented Feb 6, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

koeninger commented Feb 6, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Feb 6, 2015

SparkQA commented Feb 6, 2015

AmplabJenkins commented Feb 6, 2015

tdas commented Feb 9, 2015

SparkQA commented Feb 9, 2015

SparkQA commented Feb 9, 2015

AmplabJenkins commented Feb 9, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Feb 10, 2015

SparkQA commented Feb 10, 2015

AmplabJenkins commented Feb 10, 2015