[SPARK-8481] [MLlib] GaussianMixtureModel.predict, GaussianMixtureModel.predictSoft variants for a single vector #6906

dkobylarz · 2015-06-19T15:04:02Z

This PR adds GaussianMixtureModel.predict & GaussianMixtureModel.predictSoft variants for a single vector which are useful when applying the trained model in environments where spark context is not required (or not desired) and predictions are made for single data points (vectors).
Test case included.

srowen · 2015-06-19T15:13:39Z

OK to test

mengxr · 2015-06-19T17:22:35Z

ok to test

SparkQA · 2015-06-19T17:32:53Z

Test build #35290 has finished for PR 6906 at commit dce2432.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

dkobylarz · 2015-06-19T18:42:58Z

A blank line with two space characters (added by IDE) caused the test to fail. Is there some source code validation tool I could apply to my code before contributing?

JoshRosen · 2015-06-19T18:46:42Z

Jenkins, retest this please.

JoshRosen · 2015-06-19T18:47:20Z

I think that this particular build failure is due to code that failed our style checker being committed to master; we've fixed this with a hotfix commit.

You can use the dev/lint-scala script to run these style checks yourself.

SparkQA · 2015-06-19T18:52:30Z

Test build #35306 has finished for PR 6906 at commit dce2432.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-06-19T22:30:02Z

Test build #35318 has finished for PR 6906 at commit cb87180.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jkbradley · 2015-06-24T21:52:21Z

@dkobylarz Thanks for the PR! This looks fine, but could you please add a simple test to the unit tests? It could be a new unit test in the GMM suite, or just an extra line testing this method in an existing unit test.

jkbradley · 2015-07-01T18:47:33Z

Ping

dkobylarz · 2015-07-02T23:58:52Z

it will be added on Monday

jkbradley · 2015-07-05T19:55:09Z

Great thanks!

SparkQA · 2015-07-06T11:45:52Z

Test build #36568 has finished for PR 6906 at commit 54af203.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dkobylarz · 2015-07-06T21:42:29Z

I quickly reviewed the Jenkins log. Mllib tests seem to be passed, it's the sql test that fails. Have there been any changes to 1.4 branch that are responsible for it?

hadfield · 2015-07-08T19:29:57Z

Jenkins, retest this please.

jkbradley · 2015-07-08T20:56:27Z

@dkobylarz Could you please update the PR description to say what was done in this PR? It will become part of the git commit message (so the current description does not quite fit).

Jenkins test this please

jkbradley · 2015-07-08T21:05:05Z

mllib/src/main/scala/org/apache/spark/mllib/clustering/GaussianMixtureModel.scala

@@ -66,6 +66,12 @@ class GaussianMixtureModel(
    responsibilityMatrix.map(r => r.indexOf(r.max))
  }

+  /** Maps given point to its cluster index. */
+  def predict(point: Vector) : Int = {


scala style: no space between parenthesis and colon

…ctor

SparkQA · 2015-07-17T00:48:41Z

Test build #37547 has finished for PR 6906 at commit cef1f0a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jkbradley · 2015-07-21T05:50:23Z

@dkobylarz Could you please update the PR description to say what was done in this PR? It will become part of the git commit message (so the current description does not quite fit).

Also, as you make updates, it is helpful if you only rebase in order to fix merge conflicts. Otherwise, it's hard to tell what changes you have made in each update.

I'll take another look.

dkobylarz · 2015-07-21T06:03:59Z

Description updated.
I treat this PR as an atomic request. The commits are always squashed.

jkbradley · 2015-07-21T06:06:51Z

What did you update the description to? I still see "This PR addresses the issue I have just reported. After taking a look at the code It seemed trivial to implement.", which says nothing about what this PR is doing. (Is github being slow to update or something?)

When this PR gets merged, the merge script will squash all commits in this PR anyways. Separating commits during review makes review easier.

jkbradley · 2015-07-21T06:08:03Z

This LGTM once the PR description is updated. Thanks!

dkobylarz · 2015-07-21T06:13:20Z

Could you please tell me how to edit the PR description. In this view I am only able to edit the PR title.
OK, I won't squash them next time.

dkobylarz · 2015-07-21T06:17:44Z

Nevermind, I didn't realize the first comment in this stream is also the PR description.

jkbradley · 2015-07-21T17:38:10Z

@dkobylarz No problem, this looks fine now. I'll merge it with master.
Thanks for the PR!

…el.predictSoft variants for a single vector This PR adds GaussianMixtureModel.predict & GaussianMixtureModel.predictSoft variants for a single vector which are useful when applying the trained model in environments where spark context is not required (or not desired) and predictions are made for single data points (vectors). Test case included. Author: Dariusz Kobylarz <darek.kobylarz@gmail.com> Closes #6906 from dkobylarz/branch-1.4 and squashes the following commits: cef1f0a [Dariusz Kobylarz] [SPARK-8481] [MLlib] GaussianMixtureModel predict accepting single vector

jkbradley · 2015-07-24T01:17:54Z

@dkobylarz The merge script failed to close this for some reason. Would you mind closing it manually? Thanks

dkobylarz · 2015-07-24T09:48:33Z

Done

yu-iskw · 2015-08-07T01:41:51Z

@jkbradley let me just check, was this PR merged into only branch-1.4? It seems that predict(point: Vector) doesn't exist master branch. Is this right?

Related to: #7662

jkbradley · 2015-08-07T18:42:15Z

Uh oh, I'm not sure how I ended up merging that with branch-1.4...that's very strange. I'll see about fixing that.

jkbradley · 2015-08-07T20:47:24Z

@dkobylarz I just reverted this change in branch-1.4. It was because this PR was targeted at branch-1.4, rather than the master. Please make sure PRs are targeted at the master branch in the future (and I will need to be more careful at merge time!). I'll create a new PR and send it for branch-1.5 and master.

dkobylarz · 2015-08-07T20:50:46Z

ok, I was thinking of getting this to 1.4 branch asap. I will contribute to master branch only in future. Sorry about that and thanks for the guidance!

jkbradley · 2015-08-07T20:51:52Z

No problem, I need to be careful too!

yu-iskw · 2015-08-07T21:43:04Z

@jkbradley @dkobylarz I appreciate your checking! I look forward to merging it into the master branch!

…ctor Resubmit of [#6906] for adding single-vec predict to GMMs CC: dkobylarz mengxr To be merged with master and branch-1.5 Primary author: dkobylarz Author: Dariusz Kobylarz <darek.kobylarz@gmail.com> Closes #8039 from jkbradley/gmm-predict-vec and squashes the following commits: bfbedc4 [Dariusz Kobylarz] [SPARK-8481] [MLlib] GaussianMixtureModel predict accepting single vector (cherry picked from commit e2fbbe7) Signed-off-by: Joseph K. Bradley <joseph@databricks.com>

…ctor Resubmit of [#6906] for adding single-vec predict to GMMs CC: dkobylarz mengxr To be merged with master and branch-1.5 Primary author: dkobylarz Author: Dariusz Kobylarz <darek.kobylarz@gmail.com> Closes #8039 from jkbradley/gmm-predict-vec and squashes the following commits: bfbedc4 [Dariusz Kobylarz] [SPARK-8481] [MLlib] GaussianMixtureModel predict accepting single vector

…ctor Resubmit of [apache#6906] for adding single-vec predict to GMMs CC: dkobylarz mengxr To be merged with master and branch-1.5 Primary author: dkobylarz Author: Dariusz Kobylarz <darek.kobylarz@gmail.com> Closes apache#8039 from jkbradley/gmm-predict-vec and squashes the following commits: bfbedc4 [Dariusz Kobylarz] [SPARK-8481] [MLlib] GaussianMixtureModel predict accepting single vector

…ctor Resubmit of [apache/spark#6906] for adding single-vec predict to GMMs CC: dkobylarz mengxr To be merged with master and branch-1.5 Primary author: dkobylarz Author: Dariusz Kobylarz <darek.kobylarz@gmail.com> Closes #8039 from jkbradley/gmm-predict-vec and squashes the following commits: bfbedc4 [Dariusz Kobylarz] [SPARK-8481] [MLlib] GaussianMixtureModel predict accepting single vector

dkobylarz force-pushed the branch-1.4 branch from dce2432 to cb87180 Compare June 19, 2015 20:24

dkobylarz force-pushed the branch-1.4 branch from cb87180 to 54af203 Compare July 6, 2015 10:08

jkbradley reviewed Jul 8, 2015
View reviewed changes

dkobylarz force-pushed the branch-1.4 branch from 54af203 to cef1f0a Compare July 16, 2015 22:28

[SPARK-8481] [MLlib] GaussianMixtureModel predict accepting single ve…

cef1f0a

…ctor

dkobylarz changed the title ~~[SPARK-8481] [MLlib] GaussianMixtureModel predict accepting single vector~~ [SPARK-8481] [MLlib] GaussianMixtureModel.predict, GaussianMixtureModel.predictSoft variants for single vector Jul 21, 2015

dkobylarz changed the title ~~[SPARK-8481] [MLlib] GaussianMixtureModel.predict, GaussianMixtureModel.predictSoft variants for single vector~~ [SPARK-8481] [MLlib] GaussianMixtureModel.predict, GaussianMixtureModel.predictSoft variants for a single vector Jul 21, 2015

dkobylarz closed this Jul 24, 2015

yu-iskw mentioned this pull request Aug 7, 2015

SPARK-8918 Add @since tags to mllib.clustering #7662

Closed

jkbradley mentioned this pull request Aug 7, 2015

[SPARK-8481] [MLlib] GaussianMixtureModel predict accepting single vector #8039

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-8481] [MLlib] GaussianMixtureModel.predict, GaussianMixtureModel.predictSoft variants for a single vector #6906

[SPARK-8481] [MLlib] GaussianMixtureModel.predict, GaussianMixtureModel.predictSoft variants for a single vector #6906

dkobylarz commented Jun 19, 2015

srowen commented Jun 19, 2015

mengxr commented Jun 19, 2015

SparkQA commented Jun 19, 2015

dkobylarz commented Jun 19, 2015

JoshRosen commented Jun 19, 2015

JoshRosen commented Jun 19, 2015

SparkQA commented Jun 19, 2015

SparkQA commented Jun 19, 2015

jkbradley commented Jun 24, 2015

jkbradley commented Jul 1, 2015

dkobylarz commented Jul 2, 2015

jkbradley commented Jul 5, 2015

SparkQA commented Jul 6, 2015

dkobylarz commented Jul 6, 2015

hadfield commented Jul 8, 2015

jkbradley commented Jul 8, 2015

jkbradley Jul 8, 2015

SparkQA commented Jul 17, 2015

jkbradley commented Jul 21, 2015

dkobylarz commented Jul 21, 2015

jkbradley commented Jul 21, 2015

jkbradley commented Jul 21, 2015

dkobylarz commented Jul 21, 2015

dkobylarz commented Jul 21, 2015

jkbradley commented Jul 21, 2015

jkbradley commented Jul 24, 2015

dkobylarz commented Jul 24, 2015

yu-iskw commented Aug 7, 2015

jkbradley commented Aug 7, 2015

jkbradley commented Aug 7, 2015

dkobylarz commented Aug 7, 2015

jkbradley commented Aug 7, 2015

yu-iskw commented Aug 7, 2015

[SPARK-8481] [MLlib] GaussianMixtureModel.predict, GaussianMixtureModel.predictSoft variants for a single vector #6906

[SPARK-8481] [MLlib] GaussianMixtureModel.predict, GaussianMixtureModel.predictSoft variants for a single vector #6906

Conversation

dkobylarz commented Jun 19, 2015

srowen commented Jun 19, 2015

mengxr commented Jun 19, 2015

SparkQA commented Jun 19, 2015

dkobylarz commented Jun 19, 2015

JoshRosen commented Jun 19, 2015

JoshRosen commented Jun 19, 2015

SparkQA commented Jun 19, 2015

SparkQA commented Jun 19, 2015

jkbradley commented Jun 24, 2015

jkbradley commented Jul 1, 2015

dkobylarz commented Jul 2, 2015

jkbradley commented Jul 5, 2015

SparkQA commented Jul 6, 2015

dkobylarz commented Jul 6, 2015

hadfield commented Jul 8, 2015

jkbradley commented Jul 8, 2015

jkbradley Jul 8, 2015

Choose a reason for hiding this comment

SparkQA commented Jul 17, 2015

jkbradley commented Jul 21, 2015

dkobylarz commented Jul 21, 2015

jkbradley commented Jul 21, 2015

jkbradley commented Jul 21, 2015

dkobylarz commented Jul 21, 2015

dkobylarz commented Jul 21, 2015

jkbradley commented Jul 21, 2015

jkbradley commented Jul 24, 2015

dkobylarz commented Jul 24, 2015

yu-iskw commented Aug 7, 2015

jkbradley commented Aug 7, 2015

jkbradley commented Aug 7, 2015

dkobylarz commented Aug 7, 2015

jkbradley commented Aug 7, 2015

yu-iskw commented Aug 7, 2015