[SPARK-1672][MLLIB] Separate user and product partitioning in ALS by mengxr · Pull Request #1014 · apache/spark

mengxr · 2014-06-09T03:55:36Z

Some clean up work following #593.

Allow to set different number user blocks and number product blocks in ALS.
Update MovieLensALS to reflect the change.

…directly.

…r users and products.

…verywhere it needs to go.

Conflicts: mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala

add numUserBlocks and numProductBlocks to MovieLensALS

AmplabJenkins · 2014-06-09T03:57:49Z

Merged build triggered.

AmplabJenkins · 2014-06-09T04:18:50Z

Merged build started.

AmplabJenkins · 2014-06-09T04:59:31Z

Merged build finished.

AmplabJenkins · 2014-06-09T04:59:32Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15554/

AmplabJenkins · 2014-06-10T07:47:51Z

Merged build triggered.

mengxr · 2014-06-10T16:58:16Z

Jenkins, retest this please.

AmplabJenkins · 2014-06-10T17:02:09Z

Merged build triggered.

AmplabJenkins · 2014-06-10T17:05:01Z

Merged build started.

AmplabJenkins · 2014-06-10T18:37:01Z

Merged build finished.

AmplabJenkins · 2014-06-10T18:37:01Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15627/

mengxr · 2014-06-11T07:59:46Z

Jenkins, retest this please.

AmplabJenkins · 2014-06-11T08:02:06Z

Merged build triggered.

AmplabJenkins · 2014-06-11T08:02:15Z

Merged build started.

AmplabJenkins · 2014-06-11T08:44:18Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-11T08:44:19Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15675/

mateiz · 2014-06-12T00:46:17Z

mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala

Does private[this] make sense at the top level? Shouldn't it be private[recommendation] or something like that?

mateiz · 2014-06-12T00:48:02Z

Looks good to me modulo one small comment (what private[this] means). If you make that private[recommendation], or private[this] at the top-level turns out to mean the same thing, then we should go for it.

AmplabJenkins · 2014-06-12T01:12:06Z

Merged build triggered.

AmplabJenkins · 2014-06-12T01:12:15Z

Merged build started.

mengxr · 2014-06-12T01:16:06Z

@mateiz I changed it to private[recommendation] to be consistent with existing code. I'm going to merge it. Thanks for reviewing it!

AmplabJenkins · 2014-06-12T01:54:46Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-06-12T01:54:47Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15697/

Some clean up work following apache#593. 1. Allow to set different number user blocks and number product blocks in `ALS`. 2. Update `MovieLensALS` to reflect the change. Author: Tor Myklebust <tmyklebu@gmail.com> Author: Xiangrui Meng <meng@databricks.com> Closes apache#1014 from mengxr/SPARK-1672 and squashes the following commits: 0e910dd [Xiangrui Meng] change private[this] to private[recommendation] 36420c7 [Xiangrui Meng] set exclusion rules for ALS 9128b77 [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into SPARK-1672 294efe9 [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into SPARK-1672 9bab77b [Xiangrui Meng] clean up add numUserBlocks and numProductBlocks to MovieLensALS 84c8e8c [Xiangrui Meng] Merge branch 'master' into SPARK-1672 d17a8bf [Xiangrui Meng] merge master a4925fd [Tor Myklebust] Style. bd8a75c [Tor Myklebust] Merge branch 'master' of github.com:apache/spark into alsseppar 021f54b [Tor Myklebust] Separate user and product blocks. dcf583a [Tor Myklebust] Remove the partitioner member variable; instead, thread that needle everywhere it needs to go. 23d6f91 [Tor Myklebust] Stop making the partitioner configurable. 495784f [Tor Myklebust] Merge branch 'master' of https://github.com/apache/spark 674933a [Tor Myklebust] Fix style. 40edc23 [Tor Myklebust] Fix missing space. f841345 [Tor Myklebust] Fix daft bug creating 'pairs', also for -> foreach. 5ec9e6c [Tor Myklebust] Clean a couple of things up using 'map'. 36a0f43 [Tor Myklebust] Make the partitioner private. d872b09 [Tor Myklebust] Add negative id ALS test. df27697 [Tor Myklebust] Support custom partitioners. Currently we use the same partitioner for users and products. c90b6d8 [Tor Myklebust] Scramble user and product ids before bucketing. c774d7d [Tor Myklebust] Make the partitioner a member variable and use it instead of modding directly.

tmyklebu and others added 18 commits April 14, 2014 18:01

Make the partitioner a member variable and use it instead of modding …

c774d7d

…directly.

Scramble user and product ids before bucketing.

c90b6d8

Support custom partitioners. Currently we use the same partitioner fo…

df27697

…r users and products.

Add negative id ALS test.

d872b09

Make the partitioner private.

36a0f43

Clean a couple of things up using 'map'.

5ec9e6c

Fix daft bug creating 'pairs', also for -> foreach.

f841345

Fix missing space.

40edc23

Fix style.

674933a

Merge branch 'master' of https://github.com/apache/spark

495784f

Stop making the partitioner configurable.

23d6f91

Remove the partitioner member variable; instead, thread that needle e…

dcf583a

…verywhere it needs to go.

Separate user and product blocks.

021f54b

Merge branch 'master' of github.com:apache/spark into alsseppar

bd8a75c

Conflicts: mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala

Style.

a4925fd

merge master

d17a8bf

Merge branch 'master' into SPARK-1672

84c8e8c

clean up

9bab77b

add numUserBlocks and numProductBlocks to MovieLensALS

mengxr changed the title ~~[SPARK-1672] Separate user and product partitioning in ALS~~ [SPARK-1672][MLLIB] Separate user and product partitioning in ALS Jun 9, 2014

mengxr added 3 commits June 9, 2014 22:22

Merge remote-tracking branch 'apache/master' into SPARK-1672

294efe9

Merge remote-tracking branch 'apache/master' into SPARK-1672

9128b77

set exclusion rules for ALS

36420c7

mateiz reviewed Jun 12, 2014
View reviewed changes

change private[this] to private[recommendation]

0e910dd

asfgit closed this in d920335 Jun 12, 2014

Conversation

mengxr commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 9, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

mengxr commented Jun 10, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

AmplabJenkins commented Jun 10, 2014

Uh oh!

mengxr commented Jun 11, 2014

Uh oh!

AmplabJenkins commented Jun 11, 2014

Uh oh!

AmplabJenkins commented Jun 11, 2014

Uh oh!

AmplabJenkins commented Jun 11, 2014

Uh oh!

AmplabJenkins commented Jun 11, 2014

Uh oh!

mateiz Jun 12, 2014

Choose a reason for hiding this comment

Uh oh!

mateiz commented Jun 12, 2014

Uh oh!

AmplabJenkins commented Jun 12, 2014

Uh oh!

AmplabJenkins commented Jun 12, 2014

Uh oh!

mengxr commented Jun 12, 2014

Uh oh!

AmplabJenkins commented Jun 12, 2014

Uh oh!

AmplabJenkins commented Jun 12, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants