[SPARK-4795][Core] Redesign the "primitive type => Writable" implicit APIs to make them be activated automatically #3642

zsxwing · 2014-12-09T08:58:40Z

Try to redesign the "primitive type => Writable" implicit APIs to make them be activated automatically and without breaking binary compatibility.

However, this PR will breaking the source compatibility if people use xxxToXxxWritable occasionally. See the unit test in graphx.

SparkQA · 2014-12-09T09:02:55Z

Test build #24243 has started for PR 3642 at commit ac1e903.

This patch merges cleanly.

SparkQA · 2014-12-09T09:10:44Z

Test build #24244 has started for PR 3642 at commit 3009328.

This patch merges cleanly.

SparkQA · 2014-12-09T10:01:42Z

Test build #24243 has finished for PR 3642 at commit ac1e903.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- m => m.getReturnType().toString != "class java.lang.Object" &&

AmplabJenkins · 2014-12-09T10:01:45Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24243/
Test FAILed.

SparkQA · 2014-12-09T10:10:08Z

Test build #24244 has finished for PR 3642 at commit 3009328.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-09T10:10:12Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24244/
Test FAILed.

SparkQA · 2014-12-09T11:20:14Z

Test build #24245 has started for PR 3642 at commit 39343de.

This patch merges cleanly.

zsxwing · 2014-12-09T11:20:16Z

graphx/src/test/scala/org/apache/spark/graphx/lib/ShortestPathsSuite.scala

@@ -40,7 +40,7 @@ class ShortestPathsSuite extends FunSuite with LocalSparkContext {
      val graph = Graph.fromEdgeTuples(edges, 1)
      val landmarks = Seq(1, 4).map(_.toLong)
      val results = ShortestPaths.run(graph, landmarks).vertices.collect.map {
-        case (v, spMap) => (v, spMap.mapValues(_.get))


This is an example of breaking source compatibility, although it's used the implicit intToIntWritable occasionally. _.get => intToIntWritable(_).get.

Or we can keep xxxToXxxWritable still implicit for the source compatibility?

i don't think you can keep that and have the new one because the compiler will complain

Do you have any example? I added implicit to them and compiled codes successfully.

If they are not ambiguous, I'd add the implicits back to make sure we never break.

@ankurdave - why is there a get in this test case? Is the get just redundant?

It does look like the get is redundant, since Int should be sufficient for this purpose.

If they are not ambiguous, I'd add the implicits back to make sure we never break.

I added them back.

SparkQA · 2014-12-09T12:46:09Z

Test build #24245 has finished for PR 3642 at commit 39343de.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-12-09T12:46:13Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24245/
Test PASSed.

rxin · 2015-02-03T05:46:46Z

@zsxwing if we merge this one, are there any other usecases for importing SparkContext._ ?

Conflicts: core/src/main/scala/org/apache/spark/SparkContext.scala

SparkQA · 2015-02-03T08:12:50Z

Test build #26636 has started for PR 3642 at commit 0b9017f.

This patch merges cleanly.

zsxwing · 2015-02-03T08:49:17Z

@zsxwing if we merge this one, are there any other usecases for importing SparkContext._ ?

There will be no implicit methods/objects in SparkContext object. So people don't need to import SparkContext._ for implicit methods/objects.

Since the compatibility is very important, it's better to get more pairs of eyes to look at it.

SparkQA · 2015-02-03T09:03:12Z

Test build #26636 has finished for PR 3642 at commit 0b9017f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-03T09:03:16Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26636/
Test FAILed.

zsxwing · 2015-02-03T09:14:21Z

Jenkins, retest this please.

rxin · 2015-02-03T09:15:45Z

Jenkins, retest this please.

SparkQA · 2015-02-03T09:17:42Z

Test build #26647 has started for PR 3642 at commit 0b9017f.

This patch merges cleanly.

SparkQA · 2015-02-03T10:06:34Z

Test build #26647 has finished for PR 3642 at commit 0b9017f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-03T10:06:38Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26647/
Test FAILed.

zsxwing · 2015-02-03T10:57:31Z

Jenkins, retest this please.

SparkQA · 2015-02-03T11:02:49Z

Test build #26651 has started for PR 3642 at commit 0b9017f.

This patch merges cleanly.

SparkQA · 2015-02-03T12:14:49Z

Test build #26651 has finished for PR 3642 at commit 0b9017f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-03T12:14:53Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26651/
Test PASSed.

SparkQA · 2015-02-04T02:07:38Z

Test build #26704 has started for PR 3642 at commit 914b2d6.

This patch merges cleanly.

SparkQA · 2015-02-04T03:17:01Z

Test build #26704 has finished for PR 3642 at commit 914b2d6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-04T03:17:04Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26704/
Test PASSed.

rxin · 2015-02-04T04:16:46Z

Ok I'm going to merge this. Thanks for working on it.

… APIs to make them be activated automatically Try to redesign the "primitive type => Writable" implicit APIs to make them be activated automatically and without breaking binary compatibility. However, this PR will breaking the source compatibility if people use `xxxToXxxWritable` occasionally. See the unit test in `graphx`. Author: zsxwing <zsxwing@gmail.com> Closes #3642 from zsxwing/SPARK-4795 and squashes the following commits: 914b2d6 [zsxwing] Add implicit back to the Writables methods 0b9017f [zsxwing] Add some docs a0e8509 [zsxwing] Merge branch 'master' into SPARK-4795 39343de [zsxwing] Fix the unit test 64853af [zsxwing] Reorganize the rest 'implicit' methods in SparkContext (cherry picked from commit d37978d) Signed-off-by: Reynold Xin <rxin@databricks.com>

zsxwing added 2 commits December 9, 2014 19:07

Reorganize the rest 'implicit' methods in SparkContext

64853af

Fix the unit test

39343de

zsxwing reviewed Dec 9, 2014
View reviewed changes

zsxwing added 2 commits February 3, 2015 14:38

Merge branch 'master' into SPARK-4795

a0e8509

Conflicts: core/src/main/scala/org/apache/spark/SparkContext.scala

Add some docs

0b9017f

Add implicit back to the Writables methods

914b2d6

asfgit closed this in d37978d Feb 4, 2015

zsxwing deleted the SPARK-4795 branch February 4, 2015 09:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-4795][Core] Redesign the "primitive type => Writable" implicit APIs to make them be activated automatically #3642

[SPARK-4795][Core] Redesign the "primitive type => Writable" implicit APIs to make them be activated automatically #3642

zsxwing commented Dec 9, 2014

SparkQA commented Dec 9, 2014

SparkQA commented Dec 9, 2014

SparkQA commented Dec 9, 2014

AmplabJenkins commented Dec 9, 2014

SparkQA commented Dec 9, 2014

AmplabJenkins commented Dec 9, 2014

SparkQA commented Dec 9, 2014

zsxwing Dec 9, 2014

zsxwing Dec 10, 2014

rxin Feb 3, 2015

zsxwing Feb 3, 2015

rxin Feb 3, 2015

ankurdave Feb 3, 2015

zsxwing Feb 4, 2015

SparkQA commented Dec 9, 2014

AmplabJenkins commented Dec 9, 2014

rxin commented Feb 3, 2015

SparkQA commented Feb 3, 2015

zsxwing commented Feb 3, 2015

SparkQA commented Feb 3, 2015

AmplabJenkins commented Feb 3, 2015

zsxwing commented Feb 3, 2015

rxin commented Feb 3, 2015

SparkQA commented Feb 3, 2015

SparkQA commented Feb 3, 2015

AmplabJenkins commented Feb 3, 2015

zsxwing commented Feb 3, 2015

SparkQA commented Feb 3, 2015

SparkQA commented Feb 3, 2015

AmplabJenkins commented Feb 3, 2015

SparkQA commented Feb 4, 2015

SparkQA commented Feb 4, 2015

AmplabJenkins commented Feb 4, 2015

rxin commented Feb 4, 2015

[SPARK-4795][Core] Redesign the "primitive type => Writable" implicit APIs to make them be activated automatically #3642

[SPARK-4795][Core] Redesign the "primitive type => Writable" implicit APIs to make them be activated automatically #3642

Conversation

zsxwing commented Dec 9, 2014

SparkQA commented Dec 9, 2014

SparkQA commented Dec 9, 2014

SparkQA commented Dec 9, 2014

AmplabJenkins commented Dec 9, 2014

SparkQA commented Dec 9, 2014

AmplabJenkins commented Dec 9, 2014

SparkQA commented Dec 9, 2014

zsxwing Dec 9, 2014

Choose a reason for hiding this comment

zsxwing Dec 10, 2014

Choose a reason for hiding this comment

rxin Feb 3, 2015

Choose a reason for hiding this comment

zsxwing Feb 3, 2015

Choose a reason for hiding this comment

rxin Feb 3, 2015

Choose a reason for hiding this comment

ankurdave Feb 3, 2015

Choose a reason for hiding this comment

zsxwing Feb 4, 2015

Choose a reason for hiding this comment

SparkQA commented Dec 9, 2014

AmplabJenkins commented Dec 9, 2014

rxin commented Feb 3, 2015

SparkQA commented Feb 3, 2015

zsxwing commented Feb 3, 2015

SparkQA commented Feb 3, 2015

AmplabJenkins commented Feb 3, 2015

zsxwing commented Feb 3, 2015

rxin commented Feb 3, 2015

SparkQA commented Feb 3, 2015

SparkQA commented Feb 3, 2015

AmplabJenkins commented Feb 3, 2015

zsxwing commented Feb 3, 2015

SparkQA commented Feb 3, 2015

SparkQA commented Feb 3, 2015

AmplabJenkins commented Feb 3, 2015

SparkQA commented Feb 4, 2015

SparkQA commented Feb 4, 2015

AmplabJenkins commented Feb 4, 2015

rxin commented Feb 4, 2015