[SPARK-7984][Graphx]Complex Operators between Graphs: Union

[shijinkui]

The union of two graphs G(VG, EG) and H(VH, EH) is the union of their vertex sets and their edge families. Which means G ∪ H = (VG ∪ VH, EG ∪ EH).

I implement by fullOuterJoin the current Graph's VertexRDD with the other Graph's VertexRDD merging the same Attribute. Then convert the two Graph EdgeRDD to key value pair like srcId+DstId -> Edge, fullOuterJoin the two and merge the same Edge attribute with the parameter function mergeSameEdgeAttr.

[rxin]

cc @ankurdave

[andrewor14]

ok to test

[SparkQA]

Test build #35186 has finished for PR 6685 at commit 5ecb8f1.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35264 has finished for PR 6685 at commit 9953308.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[shijinkui]

what does this building error mean?
is there more detail error log for Binary compatibility check failed?

[info]  [SUCCESSFUL ] org.apache.spark#spark-mllib_2.10;1.4.0-rc4!spark-mllib_2.10.jar (1085ms)
[info] spark-mllib: found 0 potential binary incompatibilities (filtered 5)
[info] downloading https://repository.apache.org/content/repositories/orgapachespark-1112/org/apache/spark/spark-streaming-zeromq_2.10/1.4.0-rc4/spark-streaming-zeromq_2.10-1.4.0-rc4.jar ...
[info]  [SUCCESSFUL ] org.apache.spark#spark-streaming-zeromq_2.10;1.4.0-rc4!spark-streaming-zeromq_2.10.jar (635ms)
[info] spark-streaming-zeromq: found 0 potential binary incompatibilities (filtered 1)
java.lang.RuntimeException: spark-graphx: Binary compatibility check failed!
    at scala.sys.package$.error(package.scala:27)
    at com.typesafe.tools.mima.plugin.SbtMima$.reportErrors(SbtMima.scala:64)
    at com.typesafe.tools.mima.plugin.MimaPlugin$$anonfun$mimaReportSettings$3.apply(MimaPlugin.scala:23)
    at com.typesafe.tools.mima.plugin.MimaPlugin$$anonfun$mimaReportSettings$3.apply(MimaPlugin.scala:23)
    at scala.Function5$$anonfun$tupled$1.apply(Function5.scala:35)
    at scala.Function5$$anonfun$tupled$1.apply(Function5.scala:34)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
    at sbt.std.Transform$$anon$4.work(System.scala:63)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.Execute.work(Execute.scala:235)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
[error] (graphx/*:mimaReportBinaryIssues) spark-graphx: Binary compatibility check failed!
[error] Total time: 181 s, completed Jun 19, 2015 5:22:38 AM
[error] running ['/home/jenkins/workspace/SparkPullRequestBuilder/dev/../dev/mima'] ; received return code 1
Archiving unit tests logs...
> No log files found.
Attempting to post to Github...
 > Post successful.
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
ERROR: Publisher 'Publish JUnit test result report' failed: No test report files were found. Configuration error?
Finished: FAILURE

[SparkQA]

Test build #35428 has finished for PR 6685 at commit 21912dc.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35543 has finished for PR 6685 at commit 1654932.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35546 has finished for PR 6685 at commit 9c2f425.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[shijinkui]

hi @andrewor14 , what can i do for this error

[error] running /home/jenkins/workspace/SparkPullRequestBuilder/dev/../dev/mima ; received return code 255
Archiving unit tests logs...
> No log files found.
Attempting to post to Github...
 > Post successful.
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
ERROR: Publisher 'Publish JUnit test result report' failed: No test report files were found. Configuration error?
Finished: FAILURE

[andrewor14]

probably not your fault... retest this please

[SparkQA]

Test build #35583 has finished for PR 6685 at commit 9c2f425.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[andrewor14]

@ankurdave could you have a look?

[AmplabJenkins]

Can one of the admins verify this patch?

[krcz]

I believe you should add some delimiter when concatenating. Otherwise two different edges can be glued together (i.e. 1 -> 12 and 11 -> 2).

[srowen]

I think this has gone stale. Could you close this PR?