[MLlib] SPARK-5954: Top by key #5075

coderxiang · 2015-03-17T21:02:45Z

This PR implements two functions

topByKey(num: Int): RDD[(K, Array[V])] finds the top-k values for each key in a pair RDD. This can be used, e.g., in computing top recommendations.
takeOrderedByKey(num: Int): RDD[(K, Array[V])] does the opposite of topByKey

The sorted is used here as the toArray method of the PriorityQueue does not return a necessarily sorted array.

SparkQA · 2015-03-17T21:08:13Z

Test build #28735 has started for PR 5075 at commit 0895c17.

This patch merges cleanly.

mengxr · 2015-03-17T21:11:04Z

core/src/test/scala/org/apache/spark/rdd/PairRDDFunctionsSuite.scala

+  test("topByKey") {
+    val pairs = sc.parallelize(Array((1, 1), (1, 2), (3, 2), (3, 7), (3, 5), (5, 1), (5, 3)), 2)
+
+    val sets = pairs.topByKey(2).collect()


Use colelctAsMap.

SparkQA · 2015-03-17T22:06:20Z

Test build #28735 has finished for PR 5075 at commit 0895c17.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-03-17T22:06:23Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28735/
Test FAILed.

SparkQA · 2015-03-17T22:22:49Z

Test build #28746 has started for PR 5075 at commit 70c6e35.

This patch merges cleanly.

SparkQA · 2015-03-17T23:44:00Z

Test build #28746 has finished for PR 5075 at commit 70c6e35.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-03-17T23:44:05Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28746/
Test PASSed.

srowen · 2015-03-18T13:36:24Z

core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala

+        queue1 ++= queue2
+        queue1
+      }
+    ).mapValues(_.toArray.sorted(ord.reverse))


Does the toArray not already give you the top k in order? this seems to be the behavior already as it returns the array formed from the iterator in the underlying PriorityQueue. Worth testing I think. (That said, I suppose sorting an already-sorted array is pretty fast.)

Nit: a few lines above, an extra space in front of =>

Good catch. Removing sorted will fail the test as the toArray does not generate a sorted sequence.

Hm OK that surprises me but if you verified it is required, leave it of course

srowen · 2015-03-18T13:36:49Z

Looking good, but what about the Java / Python APIs too?

SparkQA · 2015-03-18T16:43:06Z

Test build #28815 has started for PR 5075 at commit 901b0af.

This patch merges cleanly.

rxin · 2015-03-18T17:12:48Z

Thanks for the contribution.

I think we should first put this in MLlib, and then consider moving it to RDD in the future. The RDD API is fairly crowded, and adding more functions to it will make it harder to learn/read/understand.

SparkQA · 2015-03-18T18:04:00Z

Test build #28815 has finished for PR 5075 at commit 901b0af.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-03-18T18:04:03Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28815/
Test PASSed.

SparkQA · 2015-03-19T22:17:39Z

Test build #28891 has started for PR 5075 at commit 82dded9.

This patch merges cleanly.

coderxiang · 2015-03-19T22:18:00Z

@rxin @mengxr per the comments, I created MLPairRDDFunctions.scala and moved the function there in the update.

SparkQA · 2015-03-19T23:02:37Z

Test build #28897 has started for PR 5075 at commit a80e0ec.

This patch merges cleanly.

SparkQA · 2015-03-19T23:08:07Z

Test build #28898 has started for PR 5075 at commit 6f565c0.

This patch merges cleanly.

SparkQA · 2015-03-19T23:41:44Z

Test build #28891 has finished for PR 5075 at commit 82dded9.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MLPairRDDFunctions[K: ClassTag, V: ClassTag](self: RDD[(K, V)]) extends Serializable
- class CheckpointWriteHandler(

AmplabJenkins · 2015-03-19T23:41:49Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28891/
Test PASSed.

SparkQA · 2015-03-20T00:23:14Z

Test build #28897 has finished for PR 5075 at commit a80e0ec.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MLPairRDDFunctions[K: ClassTag, V: ClassTag](self: RDD[(K, V)]) extends Serializable
- class CheckpointWriteHandler(

AmplabJenkins · 2015-03-20T00:23:18Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28897/
Test PASSed.

SparkQA · 2015-03-20T00:29:43Z

Test build #28898 has finished for PR 5075 at commit 6f565c0.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MLPairRDDFunctions[K: ClassTag, V: ClassTag](self: RDD[(K, V)]) extends Serializable

AmplabJenkins · 2015-03-20T00:29:47Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28898/
Test PASSed.

mengxr · 2015-03-20T01:59:17Z

mllib/src/test/scala/org/apache/spark/mllib/rdd/MLPairRDDFunctionsSuite.scala

+    val topMap = pairs.topByKey(2).collectAsMap()
+
+    assert(topMap.size === 3)
+    val valuesFor1 = topMap(1)


We don't need this variable. Just use topMap(1) in assert.

SparkQA · 2015-03-20T04:43:14Z

Test build #28906 has started for PR 5075 at commit 1611c37.

This patch merges cleanly.

SparkQA · 2015-03-20T06:03:47Z

Test build #28906 has finished for PR 5075 at commit 1611c37.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class MLPairRDDFunctions[K: ClassTag, V: ClassTag](self: RDD[(K, V)]) extends Serializable

AmplabJenkins · 2015-03-20T06:03:52Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28906/
Test PASSed.

mengxr · 2015-03-20T18:46:08Z

LGTM. Merged into master. Thanks!

coderxiang · 2015-03-20T18:48:39Z

@mengxr Thanks!

coderxiang added 3 commits March 11, 2015 13:46

topByKey

debccad

Merge remote-tracking branch 'upstream/master' into topByKey

b10e325

Merge remote-tracking branch 'upstream/master' into topByKey

0895c17

mengxr reviewed Mar 17, 2015
View reviewed changes

remove takeOrderedByKey, update doc and test

70c6e35

srowen reviewed Mar 18, 2015
View reviewed changes

style check

901b0af

coderxiang added 2 commits March 19, 2015 15:15

move to MLPairRDDFunctions

d202745

Merge remote-tracking branch 'upstream/master' into topByKey

82dded9

coderxiang changed the title ~~[Core] SPARK-5954: Top by key~~ [MLlib] SPARK-5954: Top by key Mar 19, 2015

typo and warning

a80e0ec

naming

6f565c0

mengxr reviewed Mar 20, 2015
View reviewed changes

code clean up

1611c37

asfgit closed this in 5e6ad24 Mar 20, 2015

coderxiang deleted the topByKey branch March 20, 2015 18:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MLlib] SPARK-5954: Top by key #5075

[MLlib] SPARK-5954: Top by key #5075

coderxiang commented Mar 17, 2015

SparkQA commented Mar 17, 2015

mengxr Mar 17, 2015

SparkQA commented Mar 17, 2015

AmplabJenkins commented Mar 17, 2015

SparkQA commented Mar 17, 2015

SparkQA commented Mar 17, 2015

AmplabJenkins commented Mar 17, 2015

srowen Mar 18, 2015

coderxiang Mar 18, 2015

srowen Mar 19, 2015

srowen commented Mar 18, 2015

SparkQA commented Mar 18, 2015

rxin commented Mar 18, 2015

SparkQA commented Mar 18, 2015

AmplabJenkins commented Mar 18, 2015

SparkQA commented Mar 19, 2015

coderxiang commented Mar 19, 2015

SparkQA commented Mar 19, 2015

SparkQA commented Mar 19, 2015

SparkQA commented Mar 19, 2015

AmplabJenkins commented Mar 19, 2015

SparkQA commented Mar 20, 2015

AmplabJenkins commented Mar 20, 2015

SparkQA commented Mar 20, 2015

AmplabJenkins commented Mar 20, 2015

mengxr Mar 20, 2015

SparkQA commented Mar 20, 2015

SparkQA commented Mar 20, 2015

AmplabJenkins commented Mar 20, 2015

mengxr commented Mar 20, 2015

coderxiang commented Mar 20, 2015

[MLlib] SPARK-5954: Top by key #5075

[MLlib] SPARK-5954: Top by key #5075

Conversation

coderxiang commented Mar 17, 2015

SparkQA commented Mar 17, 2015

mengxr Mar 17, 2015

Choose a reason for hiding this comment

SparkQA commented Mar 17, 2015

AmplabJenkins commented Mar 17, 2015

SparkQA commented Mar 17, 2015

SparkQA commented Mar 17, 2015

AmplabJenkins commented Mar 17, 2015

srowen Mar 18, 2015

Choose a reason for hiding this comment

coderxiang Mar 18, 2015

Choose a reason for hiding this comment

srowen Mar 19, 2015

Choose a reason for hiding this comment

srowen commented Mar 18, 2015

SparkQA commented Mar 18, 2015

rxin commented Mar 18, 2015

SparkQA commented Mar 18, 2015

AmplabJenkins commented Mar 18, 2015

SparkQA commented Mar 19, 2015

coderxiang commented Mar 19, 2015

SparkQA commented Mar 19, 2015

SparkQA commented Mar 19, 2015

SparkQA commented Mar 19, 2015

AmplabJenkins commented Mar 19, 2015

SparkQA commented Mar 20, 2015

AmplabJenkins commented Mar 20, 2015

SparkQA commented Mar 20, 2015

AmplabJenkins commented Mar 20, 2015

mengxr Mar 20, 2015

Choose a reason for hiding this comment

SparkQA commented Mar 20, 2015

SparkQA commented Mar 20, 2015

AmplabJenkins commented Mar 20, 2015

mengxr commented Mar 20, 2015

coderxiang commented Mar 20, 2015