Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-5155] [PySpark] [Streaming] Mqtt streaming support in Python #7833

Closed
wants to merge 26 commits into from
Closed

[SPARK-5155] [PySpark] [Streaming] Mqtt streaming support in Python #7833

wants to merge 26 commits into from

Conversation

zsxwing
Copy link
Member

@zsxwing zsxwing commented Jul 31, 2015

This PR is based on #4229, thanks @prabeesh.

Closes #4229

prabeesh and others added 20 commits July 4, 2015 21:39
@prabeesh
Mqtt streaming support in Python
b7d42ff
@prabeesh
Added Python streaming mqtt word count example
3aa7fff
@prabeesh
adress comments
b34c3c1
@prabeesh
updated version
3f4df12
@prabeesh
Fix assembly jar location of mqtt-assembly
ee387ae
@prabeesh
address comments
795ec27
@prabeesh
fixed python style
a11968b
@prabeesh
implemented Python-friendly class
9767d82
@prabeesh
added Python test
a5a8f9f
@prabeesh
scala style fix
e1ee016
@prabeesh
python style fix
1f0cfe9
@prabeesh
fix
80474d1
@zsxwing
Make sbt build the assembly test jar for streaming mqtt
97244ec
address the comments:
87fc677 
	keep the whole MQTTTestUtils in test and then link to test jar from python
	fix issue under Maven build
	return JavaDStream[String] directly.
@prabeesh
wait for starting the receiver before publishing data
a6747cb
@zsxwing
Register StreamingListerner before starting StreamingContext; Revert …
d07f454 
…unncessary changes; fix the python unit test
@prabeesh
Merge pull request #1 from zsxwing/pr4229
b90b709 
Register StreamingListerner before starting StreamingContext; Revert …
address the comments
126608a
@zsxwing
Merge branch 'master' into pr4229
734db99
@zsxwing
Make the maven build generate the test jar for Python MQTT tests
5f8a1d4
@zsxwing zsxwing mentioned this pull request Jul 31, 2015
Closed
@zsxwing
Copy link
Member Author

zsxwing commented Jul 31, 2015

cc @tdas

@zsxwing
Copy link
Member Author

zsxwing commented Jul 31, 2015

Ran the following commands locally and verified the Python tests could find the MQTT test jar.

mvn -DskipTests clean package
python/run-tests --python-executables=python --modules=pyspark-streaming

@zsxwing
Copy link
Member Author

zsxwing commented Jul 31, 2015

@tdas found the MQTT test is flaky. It's not guaranteed that Receiver.onStart must happen before the StreamingListenerReceiverStarted event. So the MQTT test should not wait for this event. I fixed this by retrying to send the messages.

@zsxwing
Copy link
Member Author

zsxwing commented Jul 31, 2015

@srowen could you help review the changes of pom.xml, since you are more knowledgeable about maven?

The tricky thing here is MQTT unit tests use org.apache.activemq:activemq-core to set up a server for testing, but the user doesn't need it so it's in the test scope. However, Python tests also need to use org.apache.activemq:activemq-core to set up a sever for testing. So I want to build a jar including all test dependencies but it should not be published.

@SparkQA
Copy link

SparkQA commented Jul 31, 2015

Test build #39236 has finished for PR 7833 at commit 478f844.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class MQTTUtils(object):

@SparkQA
Copy link

SparkQA commented Jul 31, 2015

Test build #39234 has finished for PR 7833 at commit 5f8a1d4.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class MQTTUtils(object):

@zsxwing
Copy link
Member Author

zsxwing commented Aug 5, 2015

retest this please

@SparkQA
Copy link

SparkQA commented Aug 5, 2015

Test build #1355 has finished for PR 7833 at commit 9570bec.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class MQTTUtils(object):

@zsxwing
Copy link
Member Author

zsxwing commented Aug 5, 2015

retest this please

@SparkQA
Copy link

SparkQA commented Aug 5, 2015

Test build #39859 has finished for PR 7833 at commit 9570bec.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class MQTTUtils(object):

@SparkQA
Copy link

SparkQA commented Aug 5, 2015

Test build #230 timed out for PR 7833 at commit 9570bec after a configured wait of 175m.

@tdas
Copy link
Contributor

tdas commented Aug 5, 2015

Jenkins, test this please.

@SparkQA
Copy link

SparkQA commented Aug 5, 2015

Test build #39924 has finished for PR 7833 at commit 9570bec.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class MQTTUtils(object):

@zsxwing
Copy link
Member Author

zsxwing commented Aug 6, 2015

retest this please

@SparkQA
Copy link

SparkQA commented Aug 6, 2015

Test build #246 has finished for PR 7833 at commit 9570bec.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class MQTTUtils(object):

@zsxwing
Copy link
Member Author

zsxwing commented Aug 6, 2015

retest this please

@SparkQA
Copy link

SparkQA commented Aug 6, 2015

Test build #40010 has finished for PR 7833 at commit 9570bec.

  • This patch fails SparkR unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class MQTTUtils(object):

@SparkQA
Copy link

SparkQA commented Aug 6, 2015

Test build #250 has finished for PR 7833 at commit 9570bec.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class MQTTUtils(object):

@SparkQA
Copy link

SparkQA commented Aug 6, 2015

Test build #40030 has finished for PR 7833 at commit 9570bec.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class MQTTUtils(object):

@zsxwing
Copy link
Member Author

zsxwing commented Aug 6, 2015

retest this please

@SparkQA
Copy link

SparkQA commented Aug 6, 2015

Test build #253 has finished for PR 7833 at commit 9570bec.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class MQTTUtils(object):

@tdas
Copy link
Contributor

tdas commented Aug 6, 2015

retest this please.

@SparkQA
Copy link

SparkQA commented Aug 7, 2015

Test build #40078 has finished for PR 7833 at commit 9570bec.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class MQTTUtils(object):

@SparkQA
Copy link

SparkQA commented Aug 7, 2015

Test build #1392 has finished for PR 7833 at commit 9570bec.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class MQTTUtils(object):

@zsxwing
Copy link
Member Author

zsxwing commented Aug 7, 2015

retest this please.

@SparkQA
Copy link

SparkQA commented Aug 7, 2015

Test build #40121 has finished for PR 7833 at commit 9570bec.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class MQTTUtils(object):

@zsxwing
Copy link
Member Author

zsxwing commented Aug 7, 2015

retest this please.

@SparkQA
Copy link

SparkQA commented Aug 7, 2015

Test build #40141 has finished for PR 7833 at commit 9570bec.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class MQTTUtils(object):

@tdas
Copy link
Contributor

tdas commented Aug 10, 2015

retest this please.

@SparkQA
Copy link

SparkQA commented Aug 10, 2015

Test build #40315 has finished for PR 7833 at commit 9570bec.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • class MQTTUtils(object):

@tdas
Copy link
Contributor

tdas commented Aug 10, 2015

This has passed tests couple of times. I am merging this to master and 1.5 Thanks @prabeesh for doing the majority of the work, and @zsxwing for getting the last mile done.

asfgit pushed a commit that referenced this pull request Aug 10, 2015
@prabeesh @tdas
[SPARK-5155] [PYSPARK] [STREAMING] Mqtt streaming support in Python
8f4014f 
This PR is based on #4229, thanks prabeesh.

Closes #4229

Author: Prabeesh K <prabsmails@gmail.com>
Author: zsxwing <zsxwing@gmail.com>
Author: prabs <prabsmails@gmail.com>
Author: Prabeesh K <prabeesh.k@namshi.com>

Closes #7833 from zsxwing/pr4229 and squashes the following commits:

9570bec [zsxwing] Fix the variable name and check null in finally
4a9c79e [zsxwing] Fix pom.xml indentation
abf5f18 [zsxwing] Merge branch 'master' into pr4229
935615c [zsxwing] Fix the flaky MQTT tests
47278c5 [zsxwing] Include the project class files
478f844 [zsxwing] Add unpack
5f8a1d4 [zsxwing] Make the maven build generate the test jar for Python MQTT tests
734db99 [zsxwing] Merge branch 'master' into pr4229
126608a [Prabeesh K] address the comments
b90b709 [Prabeesh K] Merge pull request #1 from zsxwing/pr4229
d07f454 [zsxwing] Register StreamingListerner before starting StreamingContext; Revert unncessary changes; fix the python unit test
a6747cb [Prabeesh K] wait for starting the receiver before publishing data
87fc677 [Prabeesh K] address the comments:
97244ec [zsxwing] Make sbt build the assembly test jar for streaming mqtt
80474d1 [Prabeesh K] fix
1f0cfe9 [Prabeesh K] python style fix
e1ee016 [Prabeesh K] scala style fix
a5a8f9f [Prabeesh K] added Python test
9767d82 [Prabeesh K] implemented Python-friendly class
a11968b [Prabeesh K] fixed python style
795ec27 [Prabeesh K] address comments
ee387ae [Prabeesh K] Fix assembly jar location of mqtt-assembly
3f4df12 [Prabeesh K] updated version
b34c3c1 [prabs] adress comments
3aa7fff [prabs] Added Python streaming mqtt word count example
b7d42ff [prabs] Mqtt streaming support in Python

(cherry picked from commit 853809e)
Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>
@asfgit asfgit closed this in 853809e Aug 10, 2015
@zsxwing zsxwing deleted the pr4229 branch August 11, 2015 01:17
CodingCat pushed a commit to CodingCat/spark that referenced this pull request Aug 17, 2015
@prabeesh @CodingCat
[SPARK-5155] [PYSPARK] [STREAMING] Mqtt streaming support in Python
b751a29 
This PR is based on apache#4229, thanks prabeesh.

Closes apache#4229

Author: Prabeesh K <prabsmails@gmail.com>
Author: zsxwing <zsxwing@gmail.com>
Author: prabs <prabsmails@gmail.com>
Author: Prabeesh K <prabeesh.k@namshi.com>

Closes apache#7833 from zsxwing/pr4229 and squashes the following commits:

9570bec [zsxwing] Fix the variable name and check null in finally
4a9c79e [zsxwing] Fix pom.xml indentation
abf5f18 [zsxwing] Merge branch 'master' into pr4229
935615c [zsxwing] Fix the flaky MQTT tests
47278c5 [zsxwing] Include the project class files
478f844 [zsxwing] Add unpack
5f8a1d4 [zsxwing] Make the maven build generate the test jar for Python MQTT tests
734db99 [zsxwing] Merge branch 'master' into pr4229
126608a [Prabeesh K] address the comments
b90b709 [Prabeesh K] Merge pull request #1 from zsxwing/pr4229
d07f454 [zsxwing] Register StreamingListerner before starting StreamingContext; Revert unncessary changes; fix the python unit test
a6747cb [Prabeesh K] wait for starting the receiver before publishing data
87fc677 [Prabeesh K] address the comments:
97244ec [zsxwing] Make sbt build the assembly test jar for streaming mqtt
80474d1 [Prabeesh K] fix
1f0cfe9 [Prabeesh K] python style fix
e1ee016 [Prabeesh K] scala style fix
a5a8f9f [Prabeesh K] added Python test
9767d82 [Prabeesh K] implemented Python-friendly class
a11968b [Prabeesh K] fixed python style
795ec27 [Prabeesh K] address comments
ee387ae [Prabeesh K] Fix assembly jar location of mqtt-assembly
3f4df12 [Prabeesh K] updated version
b34c3c1 [prabs] adress comments
3aa7fff [prabs] Added Python streaming mqtt word count example
b7d42ff [prabs] Mqtt streaming support in Python
@shivaram shivaram mentioned this pull request Aug 19, 2015
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@zsxwing @SparkQA @tdas @prabeesh