[SPARK-5012][MLLib][PySpark]Python API for Gaussian Mixture Model #4059

FlytxtRnD · 2015-01-15T12:12:16Z

Python API for the Gaussian Mixture Model clustering algorithm in MLLib.

AmplabJenkins · 2015-01-15T12:17:09Z

Can one of the admins verify this patch?

mengxr · 2015-01-15T19:05:13Z

add to whitelist

SparkQA · 2015-01-15T19:07:43Z

Test build #25609 has started for PR 4059 at commit 5c83825.

This patch merges cleanly.

SparkQA · 2015-01-15T19:08:37Z

Test build #25609 has finished for PR 4059 at commit 5c83825.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class GaussianMixtureModel(object):
- class GaussianMixtureEM(object):

AmplabJenkins · 2015-01-15T19:08:38Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25609/
Test FAILed.

jkbradley · 2015-01-15T20:36:14Z

@FlytxtRnD Are you still running into the py4j serialization issue you mentioned on the JIRA?

FlytxtRnD · 2015-01-16T04:05:36Z

@jkbradley py4j serialization issue has been solved by the commit 8ead999

SparkQA · 2015-01-16T04:07:34Z

Test build #25634 has started for PR 4059 at commit f82750b.

This patch merges cleanly.

SparkQA · 2015-01-16T05:05:36Z

Test build #25634 has finished for PR 4059 at commit f82750b.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class GaussianMixtureModel(object):
- class GaussianMixtureEM(object):

AmplabJenkins · 2015-01-16T05:05:40Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25634/
Test FAILed.

…ter' into PythonGmmWrapper

…nGmmWrapper' into PythonGmmWrapper Conflicts: examples/src/main/python/mllib/gaussian_mixture_model.py

SparkQA · 2015-01-16T08:42:41Z

Test build #25648 has started for PR 4059 at commit c1d4c71.

This patch merges cleanly.

SparkQA · 2015-01-16T09:51:36Z

Test build #25648 has finished for PR 4059 at commit c1d4c71.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class GaussianMixtureModel(object):
- class GaussianMixtureEM(object):

AmplabJenkins · 2015-01-16T09:51:40Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25648/
Test PASSed.

…ter' into PythonGmmWrapper

mengxr · 2015-01-22T18:57:48Z

examples/src/main/python/mllib/gaussian_mixture_model.py

+import random
+import argparse
+import numpy as np
+from pyspark import SparkConf, SparkContext


separate spark imports from python imports

FlytxtRnD · 2015-01-23T04:31:49Z

@mengxr Thank you for the review and comments. I am changing the code according to #3923 (tgaloppo).

SparkQA · 2015-02-02T10:22:47Z

Test build #26509 has started for PR 4059 at commit d5b36ab.

This patch merges cleanly.

SparkQA · 2015-02-02T11:26:23Z

Test build #26509 has finished for PR 4059 at commit d5b36ab.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class GaussianMixtureModel(object):
- class GaussianMixture(object):
- class MultivariateGaussian(object):

AmplabJenkins · 2015-02-02T11:26:27Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26509/
Test FAILed.

FlytxtRnD · 2015-02-02T11:59:18Z

Is it possible to start a test build in Jenkins without updating the PR? TestResults shows "no failures" but the console output shows errors. All tests were passed when ran locally.

SparkQA · 2015-02-02T12:22:36Z

Test build #26515 has started for PR 4059 at commit fa0a142.

This patch merges cleanly.

SparkQA · 2015-02-02T13:33:35Z

Test build #26515 has finished for PR 4059 at commit fa0a142.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-02T13:33:39Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26515/
Test PASSed.

FlytxtRnD · 2015-02-02T13:38:59Z

Please review and merge..

mengxr · 2015-02-02T17:56:58Z

mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala

+      .setK(k)
+      .setConvergenceTol(convergenceTol)
+      .setMaxIterations(maxIterations)
+      .setSeed(seed)


The default seed is not generated. If users don't specify seed on the Python side, it will pass in as null on the JVM side.

if (seed != null) gmmAlg.setSeed(seed)

In python side I have added a default value None ..So is it required to add this statement

…nMixture

…ter' into PythonGmmWrapper

SparkQA · 2015-02-03T05:37:29Z

Test build #26607 has started for PR 4059 at commit c973ab3.

This patch merges cleanly.

SparkQA · 2015-02-03T06:43:24Z

Test build #26607 has finished for PR 4059 at commit c973ab3.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class GaussianMixtureModel(object):
- class GaussianMixture(object):
- class MultivariateGaussian(namedtuple('MultivariateGaussian', ['mu', 'sigma'])):

AmplabJenkins · 2015-02-03T06:43:27Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26607/
Test PASSed.

mengxr · 2015-02-03T07:05:19Z

LGTM. Merged into master. Thanks for adding GMM Python API!

FlytxtRnD · 2015-02-03T07:06:17Z

Thanks @mengxr for your help.

FlytxtRnD added 2 commits January 15, 2015 17:30

Python API for Gaussian Mixture Model

fda60f3

Split input file with space delimiter

5c83825

Fixed style issues

f82750b

FlytxtRnD added 3 commits January 16, 2015 12:00

Merge branch 'PythonGmmWrapper', remote-tracking branch 'upstream/mas…

332bad1

…ter' into PythonGmmWrapper

Added random seed parameter

426d130

Merge branch 'PythonGmmWrapper', remote-tracking branch 'origin/Pytho…

c1d4c71

…nGmmWrapper' into PythonGmmWrapper Conflicts: examples/src/main/python/mllib/gaussian_mixture_model.py

Merge branch 'PythonGmmWrapper', remote-tracking branch 'upstream/mas…

3464d19

…ter' into PythonGmmWrapper

mengxr reviewed Jan 22, 2015
View reviewed changes

Changed argument names to lowercase

d5b36ab

New line added

fa0a142

mengxr reviewed Feb 2, 2015
View reviewed changes

FlytxtRnD added 2 commits February 3, 2015 10:46

Added MultivariateGaussian namedtuple and Arraybuffer in trainGaussia…

339b09c

…nMixture

Merge branch 'PythonGmmWrapper', remote-tracking branch 'upstream/mas…

c973ab3

…ter' into PythonGmmWrapper

asfgit closed this in 50a1a87 Feb 3, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-5012][MLLib][PySpark]Python API for Gaussian Mixture Model #4059

[SPARK-5012][MLLib][PySpark]Python API for Gaussian Mixture Model #4059

FlytxtRnD commented Jan 15, 2015

AmplabJenkins commented Jan 15, 2015

mengxr commented Jan 15, 2015

SparkQA commented Jan 15, 2015

SparkQA commented Jan 15, 2015

AmplabJenkins commented Jan 15, 2015

jkbradley commented Jan 15, 2015

FlytxtRnD commented Jan 16, 2015

SparkQA commented Jan 16, 2015

SparkQA commented Jan 16, 2015

AmplabJenkins commented Jan 16, 2015

SparkQA commented Jan 16, 2015

SparkQA commented Jan 16, 2015

AmplabJenkins commented Jan 16, 2015

mengxr Jan 22, 2015

FlytxtRnD commented Jan 23, 2015

SparkQA commented Feb 2, 2015

SparkQA commented Feb 2, 2015

AmplabJenkins commented Feb 2, 2015

FlytxtRnD commented Feb 2, 2015

SparkQA commented Feb 2, 2015

SparkQA commented Feb 2, 2015

AmplabJenkins commented Feb 2, 2015

FlytxtRnD commented Feb 2, 2015

mengxr Feb 2, 2015

FlytxtRnD Feb 3, 2015

SparkQA commented Feb 3, 2015

SparkQA commented Feb 3, 2015

AmplabJenkins commented Feb 3, 2015

mengxr commented Feb 3, 2015

FlytxtRnD commented Feb 3, 2015

[SPARK-5012][MLLib][PySpark]Python API for Gaussian Mixture Model #4059

[SPARK-5012][MLLib][PySpark]Python API for Gaussian Mixture Model #4059

Conversation

FlytxtRnD commented Jan 15, 2015

AmplabJenkins commented Jan 15, 2015

mengxr commented Jan 15, 2015

SparkQA commented Jan 15, 2015

SparkQA commented Jan 15, 2015

AmplabJenkins commented Jan 15, 2015

jkbradley commented Jan 15, 2015

FlytxtRnD commented Jan 16, 2015

SparkQA commented Jan 16, 2015

SparkQA commented Jan 16, 2015

AmplabJenkins commented Jan 16, 2015

SparkQA commented Jan 16, 2015

SparkQA commented Jan 16, 2015

AmplabJenkins commented Jan 16, 2015

mengxr Jan 22, 2015

Choose a reason for hiding this comment

FlytxtRnD commented Jan 23, 2015

SparkQA commented Feb 2, 2015

SparkQA commented Feb 2, 2015

AmplabJenkins commented Feb 2, 2015

FlytxtRnD commented Feb 2, 2015

SparkQA commented Feb 2, 2015

SparkQA commented Feb 2, 2015

AmplabJenkins commented Feb 2, 2015

FlytxtRnD commented Feb 2, 2015

mengxr Feb 2, 2015

Choose a reason for hiding this comment

FlytxtRnD Feb 3, 2015

Choose a reason for hiding this comment

SparkQA commented Feb 3, 2015

SparkQA commented Feb 3, 2015

AmplabJenkins commented Feb 3, 2015

mengxr commented Feb 3, 2015

FlytxtRnD commented Feb 3, 2015