[SPARK-5155] [PySpark] [Streaming] Mqtt streaming support in Python #4229

prabeesh · 2015-01-27T19:16:16Z

Python API for MQTT, with a word count example and Python unit test.

SparkQA · 2015-01-27T19:18:52Z

Test build #26174 has finished for PR 4229 at commit 58aa907.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class KafkaUtils(object):
- class MQTTUtils(object):

davies · 2015-01-27T20:02:47Z

python/pyspark/streaming/mqtt.py

+        jlevel = ssc._sc._getJavaStorageLevel(storageLevel)
+
+        def getClassByName(name):
+            return ssc._jvm.org.apache.spark.util.Utils.classForName(name)


getClassByName is not used anywhere, please remove it.

SparkQA · 2015-01-27T20:13:41Z

Test build #26182 has finished for PR 4229 at commit 3b45aca.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class KafkaUtils(object):
- class MQTTUtils(object):

SparkQA · 2015-02-03T19:13:50Z

Test build #26663 has finished for PR 4229 at commit 53678ad.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MQTTUtils(object):

SparkQA · 2015-02-03T19:28:53Z

Test build #26665 has finished for PR 4229 at commit b0f30db.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MQTTUtils(object):

SparkQA · 2015-02-03T20:40:56Z

Test build #26666 has finished for PR 4229 at commit baeba95.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MQTTUtils(object):

SparkQA · 2015-02-03T21:02:30Z

Test build #26668 has finished for PR 4229 at commit 3810c7d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MQTTUtils(object):

davies · 2015-02-04T18:19:55Z

examples/src/main/python/streaming/mqtt_wordcount.py

+ Mosquitto (http://mosquitto.org/) is an open source Mqtt Broker
+ In ubuntu mosquitto can be installed using the command  `$ sudo apt-get install mosquitto`
+ Run Mqtt publisher as
+  `$ bin/run-example \


This should be ran inside of mosquitto, could you mention that?

Can we modify like this ?
Publisher publishes data to mosquitto broker set in local host. Run Mqtt publisher as

bin/run-example is also inside spark, I misunderstood that, sorry.

never-mind.

Ya it is actually running the Scala publisher inside the Scala streaming word count example

SparkQA · 2015-02-05T21:26:34Z

Test build #26857 has finished for PR 4229 at commit 713ec66.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MQTTUtils(object):

prabeesh · 2015-02-20T18:01:36Z

@tdas please review this

tdas · 2015-02-23T05:13:18Z

Will do as soon as I get some Spark 1.3 release stuff out of the way.

On Fri, Feb 20, 2015 at 10:02 AM, Prabeesh K notifications@github.com
wrote:

@tdas https://github.com/tdas please review this

—
Reply to this email directly or view it on GitHub
#4229 (comment).

SparkQA · 2015-03-27T12:39:03Z

Test build #29304 has finished for PR 4229 at commit 713ec66.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MQTTUtils(object):

SparkQA · 2015-03-29T18:44:28Z

Test build #29366 has finished for PR 4229 at commit 3982069.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MQTTUtils(object):
This patch does not change any dependencies.

SparkQA · 2015-04-21T13:33:36Z

Test build #30666 has finished for PR 4229 at commit 3982069.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MQTTUtils(object):
This patch does not change any dependencies.

SparkQA · 2015-04-22T12:43:14Z

Test build #30752 has finished for PR 4229 at commit 0cdba97.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MQTTUtils(object):
This patch does not change any dependencies.

prabeesh · 2015-06-02T07:10:40Z

@tdas please take a look

tdas · 2015-06-20T00:54:32Z

@prabeesh Lets revive this PR.
@zsxwing Since you have already done the Flume Python API in #6830 , could shepherd this PR?

zsxwing · 2015-06-22T10:00:39Z

examples/src/main/python/streaming/mqtt_wordcount.py

+  `$ bin/run-example \
+       org.apache.spark.examples.streaming.MQTTPublisher tcp://localhost:1883 foo`
+ and then run the example as
+    `$ bin/spark-submit --driver-class-path external/mqtt-assembly/target/scala-*/\


Could you change --driver-class-path to --jars? It may mislead the user. MQTT libraries should be put both in the driver and executors. Using --driver-class-path only works in the local mode.

prabeesh · 2015-07-26T08:39:17Z

examples/src/main/python/streaming/mqtt_wordcount.py

+ In ubuntu mosquitto can be installed using the command  `$ sudo apt-get install mosquitto`
+ Run Mqtt publisher as
+  `$ bin/run-example \
+       org.apache.spark.examples.streaming.MQTTPublisher tcp://localhost:1883 foo`


@zsxwing / @tdas, the publisher here is the Scala publisher in the MQTT Scala word count example. Should we keep this here or we should update it to general instruction to run MQTT publisher.

I think that's fine. We can also add something to highlight this MQTTPublisher is only for demo / example.

SparkQA · 2015-07-26T10:43:24Z

Test build #38462 has finished for PR 4229 at commit b90b709.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MQTTUtils(object):

zsxwing · 2015-07-26T14:25:35Z

retest this please

SparkQA · 2015-07-26T16:31:48Z

Test build #109 has finished for PR 4229 at commit b90b709.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-07-26T16:34:01Z

Test build #38474 has finished for PR 4229 at commit b90b709.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MQTTUtils(object):

zsxwing · 2015-07-27T15:10:38Z

retest this please

SparkQA · 2015-07-27T17:02:15Z

Test build #116 has finished for PR 4229 at commit b90b709.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MQTTUtils(object):

SparkQA · 2015-07-27T17:28:18Z

Test build #38554 has finished for PR 4229 at commit b90b709.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MQTTUtils(object):

SparkQA · 2015-07-30T12:29:22Z

Test build #39036 has finished for PR 4229 at commit 126608a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MQTTUtils(object):

zsxwing · 2015-07-31T16:28:49Z

@prabeesh I opened #7833 to fix the maven build issue in this PR. When my PR is merged, it will still show that you are the author. Thanks a lot for your contribution.

Pr4229

SparkQA · 2015-08-02T20:17:52Z

Test build #39437 has finished for PR 4229 at commit 03f3e88.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MQTTUtils(object):

prabeesh · 2015-08-03T19:01:33Z

@tdas please have a look.

prabeesh · 2015-08-04T13:22:09Z

welcome @zsxwing thanks for your great help

zsxwing · 2015-08-04T13:44:21Z

@prabeesh thanks for your contribution again. Actually, you don't need to update this PR. When #7833 is merged by @tdas , he will set the author to you and you will see it in the master and 1.5 branch. (The merge script will automatically choose the one with the most commits of a PR as the major author. )

prabeesh · 2015-08-04T13:50:50Z

I am not going to update this PR again. Can I close this PR or it automatically closes after #7833 merge ?

zsxwing · 2015-08-04T14:16:53Z

You can close this PR :)

This PR is based on #4229, thanks prabeesh. Closes #4229 Author: Prabeesh K <prabsmails@gmail.com> Author: zsxwing <zsxwing@gmail.com> Author: prabs <prabsmails@gmail.com> Author: Prabeesh K <prabeesh.k@namshi.com> Closes #7833 from zsxwing/pr4229 and squashes the following commits: 9570bec [zsxwing] Fix the variable name and check null in finally 4a9c79e [zsxwing] Fix pom.xml indentation abf5f18 [zsxwing] Merge branch 'master' into pr4229 935615c [zsxwing] Fix the flaky MQTT tests 47278c5 [zsxwing] Include the project class files 478f844 [zsxwing] Add unpack 5f8a1d4 [zsxwing] Make the maven build generate the test jar for Python MQTT tests 734db99 [zsxwing] Merge branch 'master' into pr4229 126608a [Prabeesh K] address the comments b90b709 [Prabeesh K] Merge pull request #1 from zsxwing/pr4229 d07f454 [zsxwing] Register StreamingListerner before starting StreamingContext; Revert unncessary changes; fix the python unit test a6747cb [Prabeesh K] wait for starting the receiver before publishing data 87fc677 [Prabeesh K] address the comments: 97244ec [zsxwing] Make sbt build the assembly test jar for streaming mqtt 80474d1 [Prabeesh K] fix 1f0cfe9 [Prabeesh K] python style fix e1ee016 [Prabeesh K] scala style fix a5a8f9f [Prabeesh K] added Python test 9767d82 [Prabeesh K] implemented Python-friendly class a11968b [Prabeesh K] fixed python style 795ec27 [Prabeesh K] address comments ee387ae [Prabeesh K] Fix assembly jar location of mqtt-assembly 3f4df12 [Prabeesh K] updated version b34c3c1 [prabs] adress comments 3aa7fff [prabs] Added Python streaming mqtt word count example b7d42ff [prabs] Mqtt streaming support in Python (cherry picked from commit 853809e) Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>

This PR is based on #4229, thanks prabeesh. Closes #4229 Author: Prabeesh K <prabsmails@gmail.com> Author: zsxwing <zsxwing@gmail.com> Author: prabs <prabsmails@gmail.com> Author: Prabeesh K <prabeesh.k@namshi.com> Closes #7833 from zsxwing/pr4229 and squashes the following commits: 9570bec [zsxwing] Fix the variable name and check null in finally 4a9c79e [zsxwing] Fix pom.xml indentation abf5f18 [zsxwing] Merge branch 'master' into pr4229 935615c [zsxwing] Fix the flaky MQTT tests 47278c5 [zsxwing] Include the project class files 478f844 [zsxwing] Add unpack 5f8a1d4 [zsxwing] Make the maven build generate the test jar for Python MQTT tests 734db99 [zsxwing] Merge branch 'master' into pr4229 126608a [Prabeesh K] address the comments b90b709 [Prabeesh K] Merge pull request #1 from zsxwing/pr4229 d07f454 [zsxwing] Register StreamingListerner before starting StreamingContext; Revert unncessary changes; fix the python unit test a6747cb [Prabeesh K] wait for starting the receiver before publishing data 87fc677 [Prabeesh K] address the comments: 97244ec [zsxwing] Make sbt build the assembly test jar for streaming mqtt 80474d1 [Prabeesh K] fix 1f0cfe9 [Prabeesh K] python style fix e1ee016 [Prabeesh K] scala style fix a5a8f9f [Prabeesh K] added Python test 9767d82 [Prabeesh K] implemented Python-friendly class a11968b [Prabeesh K] fixed python style 795ec27 [Prabeesh K] address comments ee387ae [Prabeesh K] Fix assembly jar location of mqtt-assembly 3f4df12 [Prabeesh K] updated version b34c3c1 [prabs] adress comments 3aa7fff [prabs] Added Python streaming mqtt word count example b7d42ff [prabs] Mqtt streaming support in Python

This PR is based on apache#4229, thanks prabeesh. Closes apache#4229 Author: Prabeesh K <prabsmails@gmail.com> Author: zsxwing <zsxwing@gmail.com> Author: prabs <prabsmails@gmail.com> Author: Prabeesh K <prabeesh.k@namshi.com> Closes apache#7833 from zsxwing/pr4229 and squashes the following commits: 9570bec [zsxwing] Fix the variable name and check null in finally 4a9c79e [zsxwing] Fix pom.xml indentation abf5f18 [zsxwing] Merge branch 'master' into pr4229 935615c [zsxwing] Fix the flaky MQTT tests 47278c5 [zsxwing] Include the project class files 478f844 [zsxwing] Add unpack 5f8a1d4 [zsxwing] Make the maven build generate the test jar for Python MQTT tests 734db99 [zsxwing] Merge branch 'master' into pr4229 126608a [Prabeesh K] address the comments b90b709 [Prabeesh K] Merge pull request #1 from zsxwing/pr4229 d07f454 [zsxwing] Register StreamingListerner before starting StreamingContext; Revert unncessary changes; fix the python unit test a6747cb [Prabeesh K] wait for starting the receiver before publishing data 87fc677 [Prabeesh K] address the comments: 97244ec [zsxwing] Make sbt build the assembly test jar for streaming mqtt 80474d1 [Prabeesh K] fix 1f0cfe9 [Prabeesh K] python style fix e1ee016 [Prabeesh K] scala style fix a5a8f9f [Prabeesh K] added Python test 9767d82 [Prabeesh K] implemented Python-friendly class a11968b [Prabeesh K] fixed python style 795ec27 [Prabeesh K] address comments ee387ae [Prabeesh K] Fix assembly jar location of mqtt-assembly 3f4df12 [Prabeesh K] updated version b34c3c1 [prabs] adress comments 3aa7fff [prabs] Added Python streaming mqtt word count example b7d42ff [prabs] Mqtt streaming support in Python

davies reviewed Jan 27, 2015
View reviewed changes

prabeesh force-pushed the mqtt_python branch from 58aa907 to 3b45aca Compare January 27, 2015 20:09

prabeesh force-pushed the mqtt_python branch from 3b45aca to 53678ad Compare February 3, 2015 19:10

prabeesh force-pushed the mqtt_python branch from 53678ad to b0f30db Compare February 3, 2015 19:22

prabeesh force-pushed the mqtt_python branch 2 times, most recently from baeba95 to 3810c7d Compare February 3, 2015 19:50

davies reviewed Feb 4, 2015
View reviewed changes

zsxwing reviewed Jun 22, 2015
View reviewed changes

prabeesh force-pushed the mqtt_python branch from db48d80 to 9d8162a Compare June 23, 2015 19:21

prabeesh reviewed Jul 26, 2015
View reviewed changes

address the comments

126608a

zsxwing added 2 commits July 31, 2015 09:31

Merge branch 'master' into pr4229

734db99

Make the maven build generate the test jar for Python MQTT tests

5f8a1d4

zsxwing mentioned this pull request Jul 31, 2015

[SPARK-5155] [PySpark] [Streaming] Mqtt streaming support in Python #7833

Closed

zsxwing and others added 5 commits August 1, 2015 00:44

Add unpack

478f844

Include the project class files

47278c5

Fix the flaky MQTT tests

935615c

Merge branch 'master' into pr4229

abf5f18

Merge pull request #2 from zsxwing/pr4229

03f3e88

Pr4229

prabeesh closed this Aug 4, 2015

prabeesh deleted the mqtt_python branch August 11, 2015 13:24

[SPARK-5155] [PySpark] [Streaming] Mqtt streaming support in Python #4229

[SPARK-5155] [PySpark] [Streaming] Mqtt streaming support in Python #4229

Conversation

prabeesh commented Jan 27, 2015

SparkQA commented Jan 27, 2015

Choose a reason for hiding this comment

SparkQA commented Jan 27, 2015

SparkQA commented Feb 3, 2015

SparkQA commented Feb 3, 2015

SparkQA commented Feb 3, 2015

SparkQA commented Feb 3, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Feb 5, 2015

prabeesh commented Feb 20, 2015

tdas commented Feb 23, 2015

SparkQA commented Mar 27, 2015

SparkQA commented Mar 29, 2015

SparkQA commented Apr 21, 2015

SparkQA commented Apr 22, 2015

prabeesh commented Jun 2, 2015

tdas commented Jun 20, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jul 26, 2015

zsxwing commented Jul 26, 2015

SparkQA commented Jul 26, 2015

SparkQA commented Jul 26, 2015

zsxwing commented Jul 27, 2015

SparkQA commented Jul 27, 2015

SparkQA commented Jul 27, 2015

SparkQA commented Jul 30, 2015

zsxwing commented Jul 31, 2015

SparkQA commented Aug 2, 2015

prabeesh commented Aug 3, 2015

prabeesh commented Aug 4, 2015

zsxwing commented Aug 4, 2015

prabeesh commented Aug 4, 2015

zsxwing commented Aug 4, 2015