[SPARK-4608][Streaming] Reorganize StreamingContext implicit to improve API convenience

[zsxwing]

There is only one implicit function toPairDStreamFunctions in StreamingContext. This PR did similar reorganization like SPARK-4397.

Compiled the following codes with Spark Streaming 1.1.0 and ran it with this PR. Everything is fine.

import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._

object StreamingApp {

  def main(args: Array[String]) {
    val conf = new SparkConf().setMaster("local[2]").setAppName("FileWordCount")
    val ssc = new StreamingContext(conf, Seconds(10))
    val lines = ssc.textFileStream("/some/path")
    val words = lines.flatMap(_.split(" "))
    val pairs = words.map(word => (word, 1))
    val wordCounts = pairs.reduceByKey(_ + _)
    wordCounts.print()

    ssc.start()
    ssc.awaitTermination()
  }
}

[zsxwing]

/cc @tdas

[SparkQA]

Test build #23864 has started for PR 3464 at commit c15162c.

• This patch merges cleanly.

[tdas]

Could you add a test to test that this actually works without importing StreamignContext._ ?
Also it would be great if all the StreamingContext._ in different files can be removed, which would make this more reliable. I am not planning to backport this to 1.2- so this change should be fine.

[zsxwing]

Could you add a test to test that this actually works without importing StreamignContext._ ?

Do you mean streaming/src/test/scala/org/apache/spark/streamingtest/ImplicitSuite.scala in this PR?

[zsxwing]

Also it would be great if all the StreamingContext._ in different files can be removed, which would make this more reliable.

Done.

[tdas]

Actually, on second thought, lets not remove it from the programming guide. People might see the programmign guide of one version and try it on an older version. I would say, key this, but add a comment not necessary in Spark 1.3+

[zsxwing]

How about now?

[tdas]

Looks good.

[SparkQA]

Test build #23871 has started for PR 3464 at commit 27833bb.

• This patch merges cleanly.

[SparkQA]

Test build #23872 has started for PR 3464 at commit e6f9cc9.

• This patch merges cleanly.

[SparkQA]

Test build #23864 has finished for PR 3464 at commit c15162c.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class CompressedSerializer(FramedSerializer):

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23864/
Test FAILed.

[zsxwing]

This patch adds the following public classes (experimental):
class CompressedSerializer(FramedSerializer):

Some bug in the detecting codes? This PR doesn't add any class.

[SparkQA]

Test build #23871 has finished for PR 3464 at commit 27833bb.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23871/
Test PASSed.

[SparkQA]

Test build #23872 has finished for PR 3464 at commit e6f9cc9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23872/
Test PASSed.

[SparkQA]

Test build #23887 has started for PR 3464 at commit f74c190.

• This patch merges cleanly.

[SparkQA]

Test build #23887 has finished for PR 3464 at commit f74c190.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class MatrixFactorizationModel(

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23887/
Test PASSed.

[zsxwing]

ping @tdas

[zsxwing]

ping @tdas

[tdas]

I think this a copy-paste error ;) should be named dstream instead of rdd.

[zsxwing]

Good catch. Done.

[tdas]

LGTM, except one comment.

[SparkQA]

Test build #24826 has started for PR 3464 at commit aa6d44a.

• This patch merges cleanly.

[SparkQA]

Test build #24826 has finished for PR 3464 at commit aa6d44a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24826/
Test PASSed.

[tdas]

Merging this. Thanks @zsxwing!