[SPARK-4586][MLLIB] Python API for ML pipeline and parameters #4151

mengxr · 2015-01-22T00:58:35Z

This PR adds Python API for ML pipeline and parameters. The design doc can be found on the JIRA page. It includes transformers and an estimator to demo the simple text classification example code.

TODO:

handle parameters in LRModel
unit tests
missing some docs

CC: @davies @jkbradley

SparkQA · 2015-01-22T01:02:47Z

Test build #25935 has started for PR 4151 at commit 56de571.

This patch merges cleanly.

SparkQA · 2015-01-22T02:15:00Z

Test build #25935 has finished for PR 4151 at commit 56de571.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class PipelineStage(Params):
- class Estimator(PipelineStage):
- class Transformer(PipelineStage):
- class Pipeline(Estimator):
- class PipelineModel(Transformer):
- class JavaWrapper(object):
- class JavaEstimator(Estimator, JavaWrapper):
- class JavaTransformer(Transformer, JavaWrapper):
- class LogisticRegression(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol, HasMaxIter,
- class LogisticRegressionModel(Transformer):
- class Tokenizer(JavaTransformer, HasInputCol, HasOutputCol):
- class HashingTF(JavaTransformer, HasInputCol, HasOutputCol):
- class Param(object):
- class Params(Identifiable):
- return """class Has%s(Params):
- class HasMaxIter(Params):
- class HasRegParam(Params):
- class HasFeaturesCol(Params):
- class HasLabelCol(Params):
- class HasPredictionCol(Params):
- class HasInputCol(Params):
- class HasOutputCol(Params):
- class Identifiable(object):

AmplabJenkins · 2015-01-22T02:15:03Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25935/
Test PASSed.

optimize pipeline.fit impl

SparkQA · 2015-01-26T21:12:52Z

Test build #26113 has started for PR 4151 at commit d3e8dbe.

This patch merges cleanly.

SparkQA · 2015-01-26T21:22:42Z

Test build #26114 has started for PR 4151 at commit 05e3e40.

This patch merges cleanly.

SparkQA · 2015-01-26T22:15:47Z

Test build #26113 has finished for PR 4151 at commit d3e8dbe.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class PipelineStage(Params):
- class Estimator(PipelineStage):
- class Transformer(PipelineStage):
- class Pipeline(Estimator):
- class PipelineModel(Transformer):
- class JavaWrapper(Params):
- class JavaEstimator(Estimator, JavaWrapper):
- class JavaTransformer(Transformer, JavaWrapper):
- class LogisticRegression(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol, HasMaxIter,
- class LogisticRegressionModel(Transformer):
- class Tokenizer(JavaTransformer, HasInputCol, HasOutputCol):
- class HashingTF(JavaTransformer, HasInputCol, HasOutputCol):
- class Param(object):
- class Params(Identifiable):
- return """class Has%s(Params):
- class HasMaxIter(Params):
- class HasRegParam(Params):
- class HasFeaturesCol(Params):
- class HasLabelCol(Params):
- class HasPredictionCol(Params):
- class HasInputCol(Params):
- class HasOutputCol(Params):
- class Identifiable(object):

AmplabJenkins · 2015-01-26T22:15:51Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26113/
Test PASSed.

SparkQA · 2015-01-26T22:27:36Z

Test build #26114 has finished for PR 4151 at commit 05e3e40.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class PipelineStage(Params):
- class Estimator(PipelineStage):
- class Transformer(PipelineStage):
- class Pipeline(Estimator):
- class PipelineModel(Transformer):
- class JavaWrapper(Params):
- class JavaEstimator(Estimator, JavaWrapper):
- class JavaTransformer(Transformer, JavaWrapper):
- class LogisticRegression(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol, HasMaxIter,
- class LogisticRegressionModel(Transformer):
- class Tokenizer(JavaTransformer, HasInputCol, HasOutputCol):
- class HashingTF(JavaTransformer, HasInputCol, HasOutputCol):
- class Param(object):
- class Params(Identifiable):
- return """class Has%s(Params):
- class HasMaxIter(Params):
- class HasRegParam(Params):
- class HasFeaturesCol(Params):
- class HasLabelCol(Params):
- class HasPredictionCol(Params):
- class HasInputCol(Params):
- class HasOutputCol(Params):
- class Identifiable(object):

AmplabJenkins · 2015-01-26T22:27:39Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26114/
Test PASSed.

davies · 2015-01-27T00:43:51Z

python/pyspark/ml/util.py

+    def __init__(self):
+        #: A unique id for the object. The default implementation
+        #: concatenates the class name, "-", and 8 random hex chars.
+        self.uid = type(self).__name__ + "-" + uuid.uuid4().hex[:8]


The id(obj) will be the memory address of obj, it should be used as part of uid.

The memory address could be reused, which may not be unique.

For all live objects, the id (memory address) will be unique, but the random one (uuid) may not.

What if one gets dereferenced and a new object gets created? I try to make the random part of the id short while maintaining a tiny collision rate. With 8 hex chars, one gets selected with equal chances from more than 4 billion values.

That makes sense, thanks!

SparkQA · 2015-01-28T03:22:14Z

Test build #26202 has finished for PR 4151 at commit fc59a02.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-28T03:22:18Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26202/
Test PASSed.

SparkQA · 2015-01-28T18:42:43Z

Test build #26238 has started for PR 4151 at commit dd1256b.

This patch merges cleanly.

SparkQA · 2015-01-28T19:51:54Z

Test build #26238 has finished for PR 4151 at commit dd1256b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class BlockMatrix(
- class PipelineStage(Params):
- class Estimator(PipelineStage):
- class Transformer(PipelineStage):
- class Model(Transformer):
- class Pipeline(Estimator):
- class PipelineModel(Model):
- class JavaWrapper(Params):
- class JavaEstimator(Estimator, JavaWrapper):
- class JavaTransformer(Transformer, JavaWrapper):
- class JavaModel(JavaTransformer):
- class LogisticRegression(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol, HasMaxIter,
- class LogisticRegressionModel(JavaModel):
- class Tokenizer(JavaTransformer, HasInputCol, HasOutputCol):
- class HashingTF(JavaTransformer, HasInputCol, HasOutputCol, HasNumFeatures):
- class Param(object):
- class Params(Identifiable):
- template = '''class Has$Name(Params):
- class HasMaxIter(Params):
- class HasRegParam(Params):
- class HasFeaturesCol(Params):
- class HasLabelCol(Params):
- class HasPredictionCol(Params):
- class HasInputCol(Params):
- class HasOutputCol(Params):
- class HasNumFeatures(Params):
- class Identifiable(object):

AmplabJenkins · 2015-01-28T19:51:58Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26238/
Test PASSed.

refactor

SparkQA · 2015-01-28T20:32:45Z

Test build #26244 has started for PR 4151 at commit 44c2405.

This patch merges cleanly.

mengxr · 2015-01-28T20:43:16Z

@davies I merged your changes and moved Identifiable to util.py. Could you make a final pass?

SparkQA · 2015-01-28T20:47:52Z

Test build #26246 has started for PR 4151 at commit edbd6fe.

This patch merges cleanly.

SparkQA · 2015-01-28T21:42:36Z

Test build #26244 has finished for PR 4151 at commit 44c2405.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class BlockMatrix(
- class LogisticRegression(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol, HasMaxIter,
- class LogisticRegressionModel(JavaModel):
- class Tokenizer(JavaTransformer, HasInputCol, HasOutputCol):
- class HashingTF(JavaTransformer, HasInputCol, HasOutputCol, HasNumFeatures):
- class Identifiable(object):
- class Param(object):
- class Params(Identifiable):
- template = '''class Has$Name(Params):
- class HasMaxIter(Params):
- class HasRegParam(Params):
- class HasFeaturesCol(Params):
- class HasLabelCol(Params):
- class HasPredictionCol(Params):
- class HasInputCol(Params):
- class HasOutputCol(Params):
- class HasNumFeatures(Params):
- class Estimator(Params):
- class Transformer(Params):
- class Pipeline(Estimator):
- class PipelineModel(Transformer):
- class JavaWrapper(Params):
- class JavaEstimator(Estimator, JavaWrapper):
- class JavaTransformer(Transformer, JavaWrapper):
- class JavaModel(JavaTransformer):

AmplabJenkins · 2015-01-28T21:42:40Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26244/
Test PASSed.

SparkQA · 2015-01-28T21:57:58Z

Test build #26246 has finished for PR 4151 at commit edbd6fe.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class LogisticRegression(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol, HasMaxIter,
- class LogisticRegressionModel(JavaModel):
- class Tokenizer(JavaTransformer, HasInputCol, HasOutputCol):
- class HashingTF(JavaTransformer, HasInputCol, HasOutputCol, HasNumFeatures):
- class Param(object):
- class Params(Identifiable):
- template = '''class Has$Name(Params):
- class HasMaxIter(Params):
- class HasRegParam(Params):
- class HasFeaturesCol(Params):
- class HasLabelCol(Params):
- class HasPredictionCol(Params):
- class HasInputCol(Params):
- class HasOutputCol(Params):
- class HasNumFeatures(Params):
- class Estimator(Params):
- class Transformer(Params):
- class Pipeline(Estimator):
- class PipelineModel(Transformer):
- class Identifiable(object):
- class JavaWrapper(Params):
- class JavaEstimator(Estimator, JavaWrapper):
- class JavaTransformer(Transformer, JavaWrapper):
- class JavaModel(JavaTransformer):

AmplabJenkins · 2015-01-28T21:58:02Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26246/
Test PASSed.

davies · 2015-01-28T22:27:12Z

python/pyspark/ml/param/shared.py

+
+    def __init__(self):
+        super(HasMaxIter, self).__init__()
+        #: param for max number of iterations


Minor： Does this appear in the generated doc? I did see that

No, but this is the official Sphinx way to document instance attributes.

But pyspark.ml.rst, we disable the doc for members:

:undoc-members:

davies · 2015-01-28T22:30:48Z

@mengxr After remove inherit_doc from pipeline.py , I think it's OK to merge.

mengxr · 2015-01-28T23:54:47Z

Great! Waiting for Jenkins ...

SparkQA · 2015-01-28T23:57:33Z

Test build #26266 has started for PR 4151 at commit 415268e.

This patch merges cleanly.

SparkQA · 2015-01-29T01:10:25Z

Test build #26266 has finished for PR 4151 at commit 415268e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-29T01:10:29Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26266/
Test PASSed.

mengxr · 2015-01-29T01:14:45Z

Merged into master.

mengxr added 10 commits December 22, 2014 16:41

a working LR

33b68e0

a pipeline in python

46eea43

add Estimator and Transformer

a3015cf

add base classes and docs

dadd84e

make the example working

c18dca1

update doc

d9ea77c

code gen for shared params

17ecfb9

Merge remote-tracking branch 'apache/master' into SPARK-4586

bce72f4

a working copy

d0c5bb8

fix style

56de571

more docs

d3e8dbe

optimize pipeline.fit impl

update example

05e3e40

davies reviewed Jan 27, 2015
View reviewed changes

mengxr added 6 commits January 27, 2015 10:05

use LabeledDocument and Document in example

f4d0fe6

update doc conf and move embedded param map to instance attribute

d5efd34

make params a property

f66ba0c

update code gen and make param appear in the doc

1dcc17e

update mllib/pom.xml to include python files in the assembly

46fa147

gen numFeatures

036ca04

Davies Liu and others added 4 commits January 27, 2015 19:23

Merge branch 'SPARK-4586' of github.com:mengxr/spark into ml

78638df

fix tests

54ca7df

fix docs

14ae7e2

Merge remote-tracking branch 'apache/master' into SPARK-4586

dd1256b

Merge pull request #2 from davies/ml

44c2405

refactor

move Identifiable to ml.util

edbd6fe

davies reviewed Jan 28, 2015
View reviewed changes

remove inherit_doc from __init__

415268e

asfgit closed this in e80dc1c Jan 29, 2015

[SPARK-4586][MLLIB] Python API for ML pipeline and parameters #4151

[SPARK-4586][MLLIB] Python API for ML pipeline and parameters #4151

Conversation

mengxr commented Jan 22, 2015

SparkQA commented Jan 22, 2015

SparkQA commented Jan 22, 2015

AmplabJenkins commented Jan 22, 2015

SparkQA commented Jan 26, 2015

SparkQA commented Jan 26, 2015

SparkQA commented Jan 26, 2015

AmplabJenkins commented Jan 26, 2015

SparkQA commented Jan 26, 2015

AmplabJenkins commented Jan 26, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jan 28, 2015

AmplabJenkins commented Jan 28, 2015

SparkQA commented Jan 28, 2015

SparkQA commented Jan 28, 2015

AmplabJenkins commented Jan 28, 2015

SparkQA commented Jan 28, 2015

mengxr commented Jan 28, 2015

SparkQA commented Jan 28, 2015

SparkQA commented Jan 28, 2015

AmplabJenkins commented Jan 28, 2015

SparkQA commented Jan 28, 2015

AmplabJenkins commented Jan 28, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davies commented Jan 28, 2015

mengxr commented Jan 28, 2015

SparkQA commented Jan 28, 2015

SparkQA commented Jan 29, 2015

AmplabJenkins commented Jan 29, 2015

mengxr commented Jan 29, 2015