[SPARK-12247] [ML] [DOC] Documentation for spark.ml's ALS and collaborative filtering in general #10411

BenFradet · 2015-12-21T10:25:24Z

This documents the implementation of ALS in spark.ml with example code in scala, java and python.

SparkQA · 2015-12-21T11:18:26Z

Test build #48109 has finished for PR 10411 at commit 0787362.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * public class JavaALSExample\n

BenFradet · 2015-12-21T11:31:39Z

cc @thunterdb

SparkQA · 2015-12-21T22:11:50Z

Test build #48130 has finished for PR 10411 at commit ab0f301.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * public class JavaALSExample\n * case class Rating(userId: Int, movieId: Int, rating: Float, timestamp: Long)\n

SparkQA · 2015-12-22T08:52:31Z

Test build #48174 has finished for PR 10411 at commit 3a860b1.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * public class JavaALSExample\n * public static class Rating implements Serializable\n * case class Rating(userId: Int, movieId: Int, rating: Float, timestamp: Long)\n

SparkQA · 2015-12-22T09:07:52Z

Test build #48176 has finished for PR 10411 at commit b086ffd.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * public class JavaALSExample\n * public static class Rating implements Serializable\n * case class Rating(userId: Int, movieId: Int, rating: Float, timestamp: Long)\n

SparkQA · 2015-12-23T09:35:22Z

Test build #48234 has finished for PR 10411 at commit e336ebd.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-12-24T15:28:01Z

Test build #48308 has finished for PR 10411 at commit 4176788.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

BenFradet · 2015-12-24T15:30:36Z

examples/src/main/java/org/apache/spark/examples/ml/JavaALSExample.java

+    DataFrame rawPredictions = model.transform(test);
+    DataFrame predictions = rawPredictions
+      .withColumn("rating", rawPredictions.col("rating").cast(DataTypes.DoubleType))
+      .withColumn("prediction", rawPredictions.col("prediction").cast(DataTypes.DoubleType));


There might be a better way to do this, input welcome.

BenFradet · 2016-01-22T18:26:32Z

pinging @thunterdb and @jkbradley

SparkQA · 2016-02-09T21:42:47Z

Test build #50995 has finished for PR 10411 at commit 2603e42.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

BenFradet · 2016-02-10T08:43:14Z

Jenkins, retest this please.

SparkQA · 2016-02-10T09:02:17Z

Test build #51032 has finished for PR 10411 at commit 2603e42.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

…aborative-filtering

SparkQA · 2016-02-10T22:07:39Z

Test build #51051 has finished for PR 10411 at commit 9021f36.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

mengxr · 2016-02-11T23:03:51Z

@srowen @coderxiang Do you have time to review this PR?

srowen · 2016-02-12T10:53:47Z

examples/src/main/scala/org/apache/spark/examples/ml/ALSExample.scala

+    import sqlContext.implicits._
+
+    // $example on$
+    val ratings = sc.textFile("data/mllib/als/sample_movielens_ratings.txt")


It looks like this file was removed though right? is it because we can't distribute even a sample of it?

Nope, the one removed is sample_movielens_movies.txt as it was only used in MovieLens.scala which has been removed, cf the discussion in the jira.

BenFradet · 2016-02-12T13:14:59Z

@srowen thanks for the review, will make the necessary changes.

BenFradet · 2016-02-13T17:18:38Z

docs/ml-collaborative-filtering.md

+It is common in many real-world use cases to only have access to *implicit feedback* (e.g. views, 
+clicks, purchases, likes, shares etc.). The approach used in `spark.mllib` to deal with such data is taken
+from [Collaborative Filtering for Implicit Feedback Datasets](http://dx.doi.org/10.1109/ICDM.2008.22).
+Essentially, instead of trying to model the matrix of ratings directly, this approach treats the data


@srowen tried to take your remarks into account, I don't know if it's clearer now though.

SparkQA · 2016-02-13T18:12:47Z

Test build #51242 has finished for PR 10411 at commit 7e72c60.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2016-02-14T11:56:24Z

I'm OK merging this

srowen · 2016-02-15T09:28:07Z

@BenFradet yeah I like your last edit. If you're willing to make that change and the sentence fragment change I'll merge

BenFradet · 2016-02-15T09:31:02Z

Great, I'll do that later today.

SparkQA · 2016-02-15T18:22:43Z

Test build #51318 has finished for PR 10411 at commit 9b351e9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2016-02-16T13:03:50Z

Merged to master

BenFradet reviewed Dec 24, 2015
View reviewed changes

BenFradet added 18 commits February 10, 2016 19:08

typo in ml's als implementation

220d639

started the doc for ml's als

5513cbf

typo in mllib-collaborative-filtering doc

8be8e5f

made [scala|java|python] doc references more consistent in mllib-coll…

ddecd67

…aborative-filtering

cleanup of the examples

1a8e9e2

ALS example in scala

9695e8a

added links to the collaborative filtering section

dc2eeed

added a few comments

813bd34

rmd case class

2a46ffc

rmd dep on file

d055681

fix typing issue in the scala example

5d3f009

java example

fb1fff5

python example

2aac2f8

explanation on implicit feedback

bf0e8a8

updated scala example with the movie lens dataset

dbde40e

renamed parseString in parseRating

05d74a1

reworked the java example to use the movielens dataset

ac94d52

reworked the python example to use the movielens dataset

c2aeeeb

BenFradet added 7 commits February 10, 2016 19:08

removed the original MovieLensALS example

a29d5f0

removed the sample_movielens_movies dataset as it is not used anymore

00b8055

reworked the scala example by using a regression evaluator

a086983

reworked the java example by using a regression evaluator

5fba2b6

reworked the python example by using a regression evaluator

18bc9e7

e.g. -> i.e.

c6f9003

fixed scalastyle

9021f36

srowen reviewed Feb 12, 2016
View reviewed changes

BenFradet added 2 commits February 13, 2016 17:39

simplified the java example by removing the setters

5af5577

addressed @srowen 's comments regarding implicit feedback

7e72c60

BenFradet reviewed Feb 13, 2016
View reviewed changes

rmd sentence fragment

33135ab

better explanation of the input data regarding implicit feedback

9b351e9

asfgit closed this in 00c72d2 Feb 16, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-12247] [ML] [DOC] Documentation for spark.ml's ALS and collaborative filtering in general #10411

[SPARK-12247] [ML] [DOC] Documentation for spark.ml's ALS and collaborative filtering in general #10411

BenFradet commented Dec 21, 2015

SparkQA commented Dec 21, 2015

BenFradet commented Dec 21, 2015

SparkQA commented Dec 21, 2015

SparkQA commented Dec 22, 2015

SparkQA commented Dec 22, 2015

SparkQA commented Dec 23, 2015

SparkQA commented Dec 24, 2015

BenFradet Dec 24, 2015

BenFradet commented Jan 22, 2016

SparkQA commented Feb 9, 2016

BenFradet commented Feb 10, 2016

SparkQA commented Feb 10, 2016

SparkQA commented Feb 10, 2016

mengxr commented Feb 11, 2016

srowen Feb 12, 2016

BenFradet Feb 12, 2016

BenFradet commented Feb 12, 2016

BenFradet Feb 13, 2016

SparkQA commented Feb 13, 2016

srowen commented Feb 14, 2016

srowen commented Feb 15, 2016

BenFradet commented Feb 15, 2016

SparkQA commented Feb 15, 2016

srowen commented Feb 16, 2016

[SPARK-12247] [ML] [DOC] Documentation for spark.ml's ALS and collaborative filtering in general #10411

[SPARK-12247] [ML] [DOC] Documentation for spark.ml's ALS and collaborative filtering in general #10411

Conversation

BenFradet commented Dec 21, 2015

SparkQA commented Dec 21, 2015

BenFradet commented Dec 21, 2015

SparkQA commented Dec 21, 2015

SparkQA commented Dec 22, 2015

SparkQA commented Dec 22, 2015

SparkQA commented Dec 23, 2015

SparkQA commented Dec 24, 2015

BenFradet Dec 24, 2015

Choose a reason for hiding this comment

BenFradet commented Jan 22, 2016

SparkQA commented Feb 9, 2016

BenFradet commented Feb 10, 2016

SparkQA commented Feb 10, 2016

SparkQA commented Feb 10, 2016

mengxr commented Feb 11, 2016

srowen Feb 12, 2016

Choose a reason for hiding this comment

BenFradet Feb 12, 2016

Choose a reason for hiding this comment

BenFradet commented Feb 12, 2016

BenFradet Feb 13, 2016

Choose a reason for hiding this comment

SparkQA commented Feb 13, 2016

srowen commented Feb 14, 2016

srowen commented Feb 15, 2016

BenFradet commented Feb 15, 2016

SparkQA commented Feb 15, 2016

srowen commented Feb 16, 2016