[SPARK-546] Add full outer join to RDD and DStream.

[staple]

leftOuterJoin and rightOuterJoin are already implemented. This patch adds fullOuterJoin.

[AmplabJenkins]

Can one of the admins verify this patch?

[tdas]

Jenkins, this is ok to test.

[tdas]

Jenkins, test this please.

[tdas]

@pwendell This patch involves the core RDD API as well. Can you take a look as well?

[SparkQA]

QA tests have started for PR 1395. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17469/consoleFull

[tdas]

This does not need to be on the next line. It fits in the previous line I think.

[staple]

It's slightly over the limit at 103 characters

[tdas]

Aaargh. My visual estimation failed. :(

[staple]

No worries :)

[tdas]

This looks pretty good to me. However, I dont see a python unit test. Can you add that?

[SparkQA]

QA results for PR 1395:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17469/consoleFull

[staple]

Thanks for taking a look. The example provided in the descriptive comment for rdd.py's fullOuterJoin implementation will be executed and checked as part of the python test suite. Any other concerns?

[tdas]

Oh yeah, right about the python tests. My bad. I will let @pwendell chime in about the Spark RDD api.

[staple]

Cool, thanks!

[tdas]

Also adding @rxin since he reported the JIRA

[davies]

+1, LGTM

[JoshRosen]

Bump @tdas @pwendell. Does this look okay to you?

[tdas]

I am good to go with this. Need someone to sign off from the RDD and core
point of view.

On Wed, Sep 3, 2014 at 7:44 PM, Josh Rosen notifications@github.com wrote:

Bump @tdas https://github.com/tdas @pwendell
https://github.com/pwendell. Does this look okay to you?

—
Reply to this email directly or view it on GitHub
#1395 (comment).

[JoshRosen]

It looks like this was last tested a really long time ago, so I'm concerned that it might fail some of the style checks. Therefore:

Jenkins, retest this please.

[JoshRosen]

From a cursory look, the Java API part of this looks fine to me, since the type signatures of the new methods are the same as existing Java API methods.

[SparkQA]

Can one of the admins verify this patch?

[pwendell]

Jenkins, test this please.

[SparkQA]

QA tests have started for PR 1395 at commit 084b2d5.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 1395 at commit 084b2d5.

• This patch fails unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

@staple Could you fix the PEP8 style?

./python/pyspark/join.py:98:1: E302 expected 2 blank lines, found 1

[staple]

Sure, I fixed the new python style issue.

[davies]

Jenkins, retest this please.

[SparkQA]

QA tests have started for PR 1395 at commit 1f5595c.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 1395 at commit 1f5595c.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

QA tests have started for PR 1395 at commit 1f5595c.

• This patch merges cleanly.

[SparkQA]

QA tests have started for PR 1395 at commit 1f5595c.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 1395 at commit 1f5595c.

• This patch passes unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[tdas]

@JoshRosen @pwendell If you dont have any objections on this addition to Spark core, I will merge this PR.

[pwendell]

Go for it @tdas!