[SPARK-5616] [PySpark] Add examples for PySpark API

[lazyman500]

add examples for PySpark

[srowen]

The title of the PR should include the title of the JIRA. There is a typo throughout this PR: it's "broadcast" and not "boardcast". What are the other new files besides the main example file?

[srowen]

I'm not sure it's clear what this is an example of. Yes it uses broadcast variables, but the als.py example already does too. Why does it need to be done several times and how does this show the difference versus non-broadcast variables? that plus comments might make this more useful.

[lazyman500]

I had added some comment to explain why we use broadcast variables . and print the performance report.

Using broadcast: Iteration 0 cost time 0.829586982727
Using broadcast: Iteration 1 cost time 0.0809919834137
Using broadcast: Iteration 2 cost time 0.0794229507446
Don't use broadcast: Iteration 0 cost time 2.80766296387
Don't use broadcast: Iteration 1 cost time 2.83087706566
Don't use broadcast: Iteration 2 cost time 3.16146707535

[srowen]

ok to test

[SparkQA]

Test build #27037 has finished for PR 4417 at commit f7f7f22.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #27043 has finished for PR 4417 at commit 1cf3e59.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #27044 has finished for PR 4417 at commit d2ec368.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[lazyman500]

Do I need to fix the python style ? I have use lint-python to check my python style.
There are only some E501 errors like that:
[examples/src/main/python/broadcast.py:46:80: E501 line too long (105 > 79 characters)]
But I saw it at other scripts too. If I fix it I will make my comment ugly.
[examples/src/main/python/streaming/recoverable_network_wordcount.py:47:80: E501 line too long (85 > 79 characters)]

[srowen]

@lazyman500 yes, this needs to pass python style checks. You can use ./dev/lint-python to check your code. But I think it's also important for the Python maintainers to confirm whether this is an additive example.

[SparkQA]

Test build #27045 has finished for PR 4417 at commit 28b8a55.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

We change the default Broadcast factory to TorrentBroadcastFactory, is there any reason we use Http as default here?

[lazyman500]

@davies Thanks for your review
I imitate the scala example (https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/BroadcastTest.scala)
I guess that author want to tell us how to change boardCast Type :)
Do I need change the default Broadcast factory to TorrentBroadcastFactory ?

[davies]

It's good to have these examples, thanks for working on it. I had took a round on it, left few comments.

[AmplabJenkins]

Can one of the admins verify this patch?

[rxin]

I'm going to close this pull request. If this is still relevant and you are interested in pushing it forward, please open a new pull request. Thanks!