[SPARK-3982] [Streaming] [PySpark] Python API: receiverStream()

[davies]

This patch brings receiverStream() for Python API, it could be used to create an input stream with any arbitrary user implemented receiver.

The Python receiver depends on py4j, so can only be run in driver (locally).

cc @tdas

[SparkQA]

QA tests have started for PR 2833 at commit 3b7b407.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2833 at commit 3b7b407.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class Receiver(object):
  • >>> class MyReceiver(Receiver):

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21830/
Test FAILed.

[SparkQA]

QA tests have started for PR 2833 at commit 3b7b407.

• This patch merges cleanly.

[SparkQA]

QA tests have started for PR 2833 at commit 118279b.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2833 at commit 118279b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class Receiver(object):
  • >>> class MyReceiver(Receiver):

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21849/
Test PASSed.

[SparkQA]

Tests timed out for PR 2833 at commit 3b7b407 after a configured wait of 120m.

[SparkQA]

QA tests have started for PR 2833 at commit 118279b.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2833 at commit 118279b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class Receiver(object):
  • >>> class MyReceiver(Receiver):

[theoryno3]

Just wondering when this pull request will be accepted?

[tdas]

Taking a detailed look right now.

[SparkQA]

QA tests have started for PR 2833 at commit d523fbf.

• This patch merges cleanly.

[SparkQA]

Tests timed out for PR 2833 at commit d523fbf after a configured wait of 120m.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22017/
Test FAILed.

[giwa]

@tdas CC @davies
If this feature is ready for 1.2 release, it would be great! Could you take a look why this test fails in Jenkins since code freeze for 1.2 is November 1st?

[SparkQA]

Test build #480 has started for PR 2833 at commit d523fbf.

• This patch merges cleanly.

[davies]

@giwa In this PR, the Python receiver can only run in driver, so it's not as scalable as scala one (which run in executor). In order to let it run in executor, there a bunch things needed to do, so this maybe can not catch the 1.2 release.

@tdas Could you update a little bit about your plan?

[SparkQA]

Test build #480 has finished for PR 2833 at commit d523fbf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class Receiver(object):
  • >>> class MyReceiver(Receiver):

[tdas]

@davies Let's focus on this after Kafka Python API gets in. Since doing this the right way (that is running the receiver in the executor) is complicated, lets do a design doc and discussion on this.

In the meantime, I leave it up to you to decide whether you want to leave this PR open.

[davies]

@tdas I think we could leave this PR open as a proposed approach, discuss the pro/con in docs.

[tdas]

Alright, then could you come write up a design doc, post it on the JIRA and then we can discuss on it?

[SparkQA]

Test build #24871 has started for PR 2833 at commit 3eb0d9f.

• This patch merges cleanly.

[SparkQA]

Test build #24872 has started for PR 2833 at commit 38c6342.

• This patch merges cleanly.

[SparkQA]

Test build #24871 has finished for PR 2833 at commit 3eb0d9f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class GaussianMixtureModel(
  • class Receiver(object):
  • >>> class MyReceiver(Receiver):

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24871/
Test PASSed.

[SparkQA]

Test build #24872 has finished for PR 2833 at commit 38c6342.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class GaussianMixtureModel(
  • class Receiver(object):
  • >>> class MyReceiver(Receiver):

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24872/
Test PASSed.

[tdas]

@davies Since we are not working on this feature any more mind closing this PR? We can open it once when we decide to focus on it.