[SPARK-7799][SPARK-12786][Streaming]Add "streaming-akka" project #10744

zsxwing · 2016-01-13T22:48:22Z

Include the following changes:

Add "streaming-akka" project and org.apache.spark.streaming.akka.AkkaUtils for creating an actorStream
Remove "StreamingContext.actorStream" and "JavaStreamingContext.actorStream"
Update the ActorWordCount example and add the JavaActorWordCount example
Make "streaming-zeromq" depend on "streaming-akka" and update the codes accordingly

Include the following changes: 1. Add "streaming-akka" project and org.apache.spark.streaming.akka.AkkaUtils for creating an actorStream 2. Remove "StreamingContext.actorStream" and "JavaStreamingContext.actorStream" 3. Update the ActorWordCount example and add the JavaActorWordCount example 4. Make "streaming-zeromq" depend on "streaming-akka" and update the codes accordingly

SparkQA · 2016-01-13T23:11:41Z

Test build #49340 has finished for PR 10744 at commit a5ec644.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-01-14T01:14:42Z

Test build #49345 has finished for PR 10744 at commit c20b51a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2016-01-14T10:29:07Z

I'm probably late to the party here, but I thought Akka was simply going away? does this need to stay around in 2.x?

zsxwing · 2016-01-14T17:54:50Z

I'm probably late to the party here, but I thought Akka was simply going away? does this need to stay around in 2.x?

Yes. This PR removed actorStream so that core and streaming won't depend on Akka. However, we still want to support actorStream in an external project like Kafka and Flume.

BrianLondon · 2016-01-15T15:25:44Z

examples/src/main/scala/org/apache/spark/examples/streaming/ActorWordCount.scala

-    val actorSystem = AkkaUtils.createActorSystem("test", host, port.toInt, conf = conf,
-      securityManager = new SecurityManager(conf))._1
+    val akkaConf = ConfigFactory.parseString(
+      s"""akka.actor.provider = "akka.remote.RemoteActorRefProvider"


I believe this change will resolve SPARK-12786.

@zsxwing Could you also tag that JIRA in this PR and resolve it when this is closed.

Updated the title

zsxwing · 2016-01-18T22:09:05Z

@tdas addressed your comments and also updated docs/streaming-custom-receivers.md

zsxwing · 2016-01-18T22:40:44Z

retest this please

SparkQA · 2016-01-18T22:48:55Z

Test build #49631 has finished for PR 10744 at commit 4e25229.

This patch fails Java style tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2016-01-18T23:06:46Z

lint-java doesn't support to add a new project :(

zsxwing · 2016-01-19T00:29:11Z

retest this please. I just disabled the java style checker.

tdas · 2016-01-19T01:05:12Z

examples/src/main/java/org/apache/spark/examples/streaming/JavaActorWordCount.java

+        jssc,
+        new WordcountActorSystemCreator(),
+        Props.create(JavaSampleActorReceiver.class, feederActorURI),
+        "SampleReceiver");


~~indent.~~ nvm. my mistake.

SparkQA · 2016-01-19T02:46:47Z

Test build #49645 has finished for PR 10744 at commit 4e25229.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing · 2016-01-19T19:49:18Z

retest this please

tdas · 2016-01-19T21:33:26Z

external/akka/src/main/scala/org/apache/spark/streaming/akka/AkkaUtils.scala

+      jssc: JavaStreamingContext,
+      propsForActor: Props,
+      actorName: String,
+      actorSystemCreator: JFunction0[ActorSystem]


Too many variations for the Java one. Let's just have 3.
1> props, name
2> 1 + storage level // for Advanced Spark users
3> 2 + actorSystemCreator + supervisorStrategy // for advanced Spark and Akka users

SparkQA · 2016-01-19T21:44:45Z

Test build #49700 has finished for PR 10744 at commit 0c8bd04.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SampleActorReceiver[T](urlOfPublisher: String) extends ActorReceiver

tdas · 2016-01-19T21:57:55Z

external/zeromq/src/main/scala/org/apache/spark/streaming/zeromq/ZeroMQUtils.scala

+  }
+
+  /**
+   * Create an input stream that receives messages pushed by a zeromq publisher.


Here too please reduce the number of createStream versions.

tdas · 2016-01-19T22:08:26Z

Add unit tests for scala and java signatures for AkkaUtils.createStream

zsxwing · 2016-01-20T01:34:39Z

retest this please

SparkQA · 2016-01-20T01:50:03Z

Test build #49742 has finished for PR 10744 at commit 9e36df1.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-01-20T04:05:24Z

Test build #49748 has finished for PR 10744 at commit 5e2ac50.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2016-01-20T20:07:33Z

LGTM. Will merge when tests pass. Thank @zsxwing !

SparkQA · 2016-01-20T21:51:09Z

Test build #49804 has finished for PR 10744 at commit c31be35.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…uctions for streaming-akka project Since `actorStream` is an external project, we should add the linking and deploying instructions for it. A follow up PR of #10744 Author: Shixiong Zhu <shixiong@databricks.com> Closes #10856 from zsxwing/akka-link-instruction.

Fix Mima tests

c20b51a

BrianLondon reviewed Jan 15, 2016
View reviewed changes

zsxwing changed the title ~~[SPARK-7799][Streaming]Add "streaming-akka" project~~ [SPARK-7799][SPARK-12786][Streaming]Add "streaming-akka" project Jan 18, 2016

zsxwing added 4 commits January 18, 2016 12:47

actorStream -> createStream

f267157

Use standard Akka confs

f650ff5

Use Function0 rather than ActorSystemFactory and update a lot of docs

f2a07d4

Update zeromq

463569c

Minor doc fix

4e25229

zsxwing mentioned this pull request Jan 19, 2016

[SPARK-12829] Turn Java style checker on #10763

Closed

tdas reviewed Jan 19, 2016
View reviewed changes

zsxwing added 2 commits January 18, 2016 17:15

doc for Python API not supported

9a1b275

Fix docs

d495ecc

Add a default ActorSystem

0c8bd04

tdas reviewed Jan 19, 2016
View reviewed changes

Remove some Java overloads

5ac46f9

Add unit tests

9e36df1

Fix the compiler error

5e2ac50

Merge branch 'master' into streaming-akka-2

c31be35

asfgit closed this in b7d74a6 Jan 20, 2016

zsxwing deleted the streaming-akka-2 branch January 20, 2016 21:59

This was referenced Jan 21, 2016

[SPARK-7799][Streaming][Document]Add the linking and deploying instructions for streaming-akka project #10856

Closed

[SPARK-7997][Core]Remove Akka from Spark Core and Streaming #10854

Closed

zsxwing mentioned this pull request Jun 10, 2019

[SPARK-27300][GRAPH] Add Spark Graph modules and dependencies #24490

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-7799][SPARK-12786][Streaming]Add "streaming-akka" project #10744

[SPARK-7799][SPARK-12786][Streaming]Add "streaming-akka" project #10744

zsxwing commented Jan 13, 2016

SparkQA commented Jan 13, 2016

SparkQA commented Jan 14, 2016

srowen commented Jan 14, 2016

zsxwing commented Jan 14, 2016

BrianLondon Jan 15, 2016

tdas Jan 18, 2016

zsxwing Jan 18, 2016

zsxwing commented Jan 18, 2016

zsxwing commented Jan 18, 2016

SparkQA commented Jan 18, 2016

zsxwing commented Jan 18, 2016

zsxwing commented Jan 19, 2016

tdas Jan 19, 2016

SparkQA commented Jan 19, 2016

zsxwing commented Jan 19, 2016

tdas Jan 19, 2016

SparkQA commented Jan 19, 2016

tdas Jan 19, 2016

tdas commented Jan 19, 2016

zsxwing commented Jan 20, 2016

SparkQA commented Jan 20, 2016

SparkQA commented Jan 20, 2016

tdas commented Jan 20, 2016

SparkQA commented Jan 20, 2016

[SPARK-7799][SPARK-12786][Streaming]Add "streaming-akka" project #10744

[SPARK-7799][SPARK-12786][Streaming]Add "streaming-akka" project #10744

Conversation

zsxwing commented Jan 13, 2016

SparkQA commented Jan 13, 2016

SparkQA commented Jan 14, 2016

srowen commented Jan 14, 2016

zsxwing commented Jan 14, 2016

BrianLondon Jan 15, 2016

Choose a reason for hiding this comment

tdas Jan 18, 2016

Choose a reason for hiding this comment

zsxwing Jan 18, 2016

Choose a reason for hiding this comment

zsxwing commented Jan 18, 2016

zsxwing commented Jan 18, 2016

SparkQA commented Jan 18, 2016

zsxwing commented Jan 18, 2016

zsxwing commented Jan 19, 2016

tdas Jan 19, 2016

Choose a reason for hiding this comment

SparkQA commented Jan 19, 2016

zsxwing commented Jan 19, 2016

tdas Jan 19, 2016

Choose a reason for hiding this comment

SparkQA commented Jan 19, 2016

tdas Jan 19, 2016

Choose a reason for hiding this comment

tdas commented Jan 19, 2016

zsxwing commented Jan 20, 2016

SparkQA commented Jan 20, 2016

SparkQA commented Jan 20, 2016

tdas commented Jan 20, 2016

SparkQA commented Jan 20, 2016