[SPARK-3952] [Streaming] [PySpark] add Python examples in Streaming Programming Guide #2808

davies · 2014-10-15T00:17:20Z

Having Python examples in Streaming Programming Guide.

Also add RecoverableNetworkWordCount example.

SparkQA · 2014-10-15T00:24:58Z

QA tests have started for PR 2808 at commit 7e4bb8a.

This patch merges cleanly.

SparkQA · 2014-10-15T01:34:49Z

QA tests have finished for PR 2808 at commit 7e4bb8a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-15T01:34:53Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21756/
Test PASSed.

JoshRosen · 2014-10-17T02:45:24Z

docs/streaming-programming-guide.md

+
+# Create a local StreamingContext with two working thread and batch interval of 1 second
+sc = SparkContext("local[2]", "NetworkWordCount")
+ssc = new StreamingContext(sc, 1)


We don't need the new in Python.

JoshRosen · 2014-10-17T03:04:08Z

This looks good to me. I tried running the examples and they worked as expected. I left a couple of minor comments, but besides those I think this is ready for merging.

add code tab for design patterns sections.

davies · 2014-10-17T05:46:32Z

@JoshRosen I had addressed your comments, also added code tabs for design patterns section.

SparkQA · 2014-10-17T05:49:43Z

QA tests have started for PR 2808 at commit 3821c4d.

This patch merges cleanly.

SparkQA · 2014-10-17T07:04:06Z

QA tests have finished for PR 2808 at commit 3821c4d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-17T07:04:11Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21847/
Test PASSed.

JoshRosen · 2014-10-17T17:56:56Z

It looks like there's a missing tag or space or something, because now the markup after the connection pool section is messed up:

AmplabJenkins · 2014-10-17T19:02:24Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21861/
Test FAILed.

davies · 2014-10-17T21:09:35Z

@JoshRosen fixed.

JoshRosen · 2014-10-17T21:48:35Z

Found one more issue (sorry, hopefully this is the last one):

SparkQA · 2014-10-18T06:06:17Z

QA tests have started for PR 2808 at commit 26a7e37.

This patch merges cleanly.

SparkQA · 2014-10-18T07:15:42Z

QA tests have finished for PR 2808 at commit 26a7e37.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

davies · 2014-10-19T00:33:41Z

@JoshRosen updated the readme.

SparkQA · 2014-10-19T00:39:41Z

QA tests have started for PR 2808 at commit 8d4bec4.

This patch merges cleanly.

SparkQA · 2014-10-19T01:48:54Z

QA tests have finished for PR 2808 at commit 8d4bec4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-10-19T01:48:58Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21879/
Test PASSed.

JoshRosen · 2014-10-19T02:13:55Z

LGTM; it seems fine to me to bump the Jekyll version, since the prior version was really old and didn't seem to properly handle some of the cases that we use here. We can always revisit if we notice problems.

i don't think this should be merged until after 1.3.0 is final Author: cody koeninger <cody@koeninger.org> Author: Helena Edelson <helena.edelson@datastax.com> Closes #4537 from koeninger/wip-2808-kafka-0.8.2-upgrade and squashes the following commits: 803aa2c [cody koeninger] [SPARK-2808][Streaming][Kafka] code cleanup per TD e6dfaf6 [cody koeninger] [SPARK-2808][Streaming][Kafka] pointless whitespace change to trigger jenkins again 1770abc [cody koeninger] [SPARK-2808][Streaming][Kafka] make waitUntilLeaderOffset easier to call, call it from python tests as well d4267e9 [cody koeninger] [SPARK-2808][Streaming][Kafka] fix stderr redirect in python test script 30d991d [cody koeninger] [SPARK-2808][Streaming][Kafka] remove stderr prints since it breaks python 3 syntax 1d896e2 [cody koeninger] [SPARK-2808][Streaming][Kafka] add even even more logging to python test 4c4557f [cody koeninger] [SPARK-2808][Streaming][Kafka] add even more logging to python test 115aeee [cody koeninger] Merge branch 'master' into wip-2808-kafka-0.8.2-upgrade 2712649 [cody koeninger] [SPARK-2808][Streaming][Kafka] add more logging to python test, see why its timing out in jenkins 2b92d3f [cody koeninger] [SPARK-2808][Streaming][Kafka] wait for leader offsets in the java test as well 3824ce3 [cody koeninger] [SPARK-2808][Streaming][Kafka] naming / comments per tdas 61b3464 [cody koeninger] [SPARK-2808][Streaming][Kafka] delay for second send in boundary condition test af6f3ec [cody koeninger] [SPARK-2808][Streaming][Kafka] delay test until latest leader offset matches expected value 9edab4c [cody koeninger] [SPARK-2808][Streaming][Kafka] more shots in the dark on jenkins failing test c70ee43 [cody koeninger] [SPARK-2808][Streaming][Kafka] add more asserts to test, try to figure out why it fails on jenkins but not locally 1d10751 [cody koeninger] Merge branch 'master' into wip-2808-kafka-0.8.2-upgrade ed02d2c [cody koeninger] [SPARK-2808][Streaming][Kafka] move default argument for api version to overloaded method, for binary compat 407382e [cody koeninger] [SPARK-2808][Streaming][Kafka] update kafka to 0.8.2.1 77de6c2 [cody koeninger] Merge branch 'master' into wip-2808-kafka-0.8.2-upgrade 6953429 [cody koeninger] [SPARK-2808][Streaming][Kafka] update kafka to 0.8.2 2e67c66 [Helena Edelson] #SPARK-2808 Update to Kafka 0.8.2.0 GA from beta. d9dc2bc [Helena Edelson] Merge remote-tracking branch 'upstream/master' into wip-2808-kafka-0.8.2-upgrade e768164 [Helena Edelson] #2808 update kafka to version 0.8.2

i don't think this should be merged until after 1.3.0 is final Author: cody koeninger <cody@koeninger.org> Author: Helena Edelson <helena.edelson@datastax.com> Closes apache#4537 from koeninger/wip-2808-kafka-0.8.2-upgrade and squashes the following commits: 803aa2c [cody koeninger] [SPARK-2808][Streaming][Kafka] code cleanup per TD e6dfaf6 [cody koeninger] [SPARK-2808][Streaming][Kafka] pointless whitespace change to trigger jenkins again 1770abc [cody koeninger] [SPARK-2808][Streaming][Kafka] make waitUntilLeaderOffset easier to call, call it from python tests as well d4267e9 [cody koeninger] [SPARK-2808][Streaming][Kafka] fix stderr redirect in python test script 30d991d [cody koeninger] [SPARK-2808][Streaming][Kafka] remove stderr prints since it breaks python 3 syntax 1d896e2 [cody koeninger] [SPARK-2808][Streaming][Kafka] add even even more logging to python test 4c4557f [cody koeninger] [SPARK-2808][Streaming][Kafka] add even more logging to python test 115aeee [cody koeninger] Merge branch 'master' into wip-2808-kafka-0.8.2-upgrade 2712649 [cody koeninger] [SPARK-2808][Streaming][Kafka] add more logging to python test, see why its timing out in jenkins 2b92d3f [cody koeninger] [SPARK-2808][Streaming][Kafka] wait for leader offsets in the java test as well 3824ce3 [cody koeninger] [SPARK-2808][Streaming][Kafka] naming / comments per tdas 61b3464 [cody koeninger] [SPARK-2808][Streaming][Kafka] delay for second send in boundary condition test af6f3ec [cody koeninger] [SPARK-2808][Streaming][Kafka] delay test until latest leader offset matches expected value 9edab4c [cody koeninger] [SPARK-2808][Streaming][Kafka] more shots in the dark on jenkins failing test c70ee43 [cody koeninger] [SPARK-2808][Streaming][Kafka] add more asserts to test, try to figure out why it fails on jenkins but not locally 1d10751 [cody koeninger] Merge branch 'master' into wip-2808-kafka-0.8.2-upgrade ed02d2c [cody koeninger] [SPARK-2808][Streaming][Kafka] move default argument for api version to overloaded method, for binary compat 407382e [cody koeninger] [SPARK-2808][Streaming][Kafka] update kafka to 0.8.2.1 77de6c2 [cody koeninger] Merge branch 'master' into wip-2808-kafka-0.8.2-upgrade 6953429 [cody koeninger] [SPARK-2808][Streaming][Kafka] update kafka to 0.8.2 2e67c66 [Helena Edelson] #SPARK-2808 Update to Kafka 0.8.2.0 GA from beta. d9dc2bc [Helena Edelson] Merge remote-tracking branch 'upstream/master' into wip-2808-kafka-0.8.2-upgrade e768164 [Helena Edelson] apache#2808 update kafka to version 0.8.2

add Python examples in Streaming Programming Guide

7e4bb8a

davies changed the title ~~[SPARK-3952] add Python examples in Streaming Programming Guide~~ [SPARK-3952] [Streaming] [PySpark] add Python examples in Streaming Programming Guide Oct 15, 2014

JoshRosen reviewed Oct 17, 2014
View reviewed changes

address comments, add missing file

3821c4d

add code tab for design patterns sections.

fix format

26a7e37

update readme

8d4bec4

asfgit closed this in 05db2da Oct 19, 2014

helena pushed a commit to helena/spark that referenced this pull request Dec 7, 2014

apache#2808 update kafka to version 0.8.2

e768164

helena mentioned this pull request Dec 7, 2014

#SPARK-2808 update kafka to version 0.8.2 #3631

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-3952] [Streaming] [PySpark] add Python examples in Streaming Programming Guide #2808

[SPARK-3952] [Streaming] [PySpark] add Python examples in Streaming Programming Guide #2808

davies commented Oct 15, 2014

SparkQA commented Oct 15, 2014

SparkQA commented Oct 15, 2014

AmplabJenkins commented Oct 15, 2014

JoshRosen Oct 17, 2014

JoshRosen commented Oct 17, 2014

davies commented Oct 17, 2014

SparkQA commented Oct 17, 2014

SparkQA commented Oct 17, 2014

AmplabJenkins commented Oct 17, 2014

JoshRosen commented Oct 17, 2014

AmplabJenkins commented Oct 17, 2014

davies commented Oct 17, 2014

JoshRosen commented Oct 17, 2014

SparkQA commented Oct 18, 2014

SparkQA commented Oct 18, 2014

davies commented Oct 19, 2014

SparkQA commented Oct 19, 2014

SparkQA commented Oct 19, 2014

AmplabJenkins commented Oct 19, 2014

JoshRosen commented Oct 19, 2014

[SPARK-3952] [Streaming] [PySpark] add Python examples in Streaming Programming Guide #2808

[SPARK-3952] [Streaming] [PySpark] add Python examples in Streaming Programming Guide #2808

Conversation

davies commented Oct 15, 2014

SparkQA commented Oct 15, 2014

SparkQA commented Oct 15, 2014

AmplabJenkins commented Oct 15, 2014

JoshRosen Oct 17, 2014

Choose a reason for hiding this comment

JoshRosen commented Oct 17, 2014

davies commented Oct 17, 2014

SparkQA commented Oct 17, 2014

SparkQA commented Oct 17, 2014

AmplabJenkins commented Oct 17, 2014

JoshRosen commented Oct 17, 2014

AmplabJenkins commented Oct 17, 2014

davies commented Oct 17, 2014

JoshRosen commented Oct 17, 2014

SparkQA commented Oct 18, 2014

SparkQA commented Oct 18, 2014

davies commented Oct 19, 2014

SparkQA commented Oct 19, 2014

SparkQA commented Oct 19, 2014

AmplabJenkins commented Oct 19, 2014

JoshRosen commented Oct 19, 2014