[SPARK-8704] [ML] [PySpark] Add missing methods in StandardScaler #7086

MechCoder · 2015-06-29T13:33:53Z

Add std, mean to StandardScalerModel
getVectors, findSynonyms to Word2Vec Model
setFeatures and getFeatures to hashingTF

AmplabJenkins · 2015-06-29T13:37:12Z

Merged build triggered.

AmplabJenkins · 2015-06-29T13:37:18Z

Merged build started.

SparkQA · 2015-06-29T13:37:49Z

Test build #35988 has started for PR 7086 at commit ac9397b.

MechCoder · 2015-06-29T13:46:50Z

This breaks model.transform(doc).collect() for Word2VecModel but I do not understand why.

SparkQA · 2015-06-29T14:05:50Z

Test build #35988 has finished for PR 7086 at commit ac9397b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-06-29T14:05:58Z

Merged build finished. Test FAILed.

MechCoder · 2015-06-29T17:26:17Z

It seems that the 'JMap[String, JList[Float]]' is not serializable.

AmplabJenkins · 2015-06-29T17:37:13Z

Merged build triggered.

AmplabJenkins · 2015-06-29T17:37:22Z

Merged build started.

MechCoder · 2015-06-29T17:38:33Z

I pushed in a hack that works, but now the Scala version looks ugly.

SparkQA · 2015-06-29T17:40:24Z

Test build #36007 has started for PR 7086 at commit 5b01465.

SparkQA · 2015-06-29T18:20:44Z

Test build #36007 has finished for PR 7086 at commit 5b01465.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-06-29T18:21:11Z

Merged build finished. Test PASSed.

MechCoder · 2015-06-29T19:09:54Z

Is there are a way to make a scala method private such that it is accessible only to the python API? It seems private[python] works only with classes.

AmplabJenkins · 2015-06-29T19:12:13Z

Merged build triggered.

AmplabJenkins · 2015-06-29T19:12:20Z

Merged build started.

SparkQA · 2015-06-29T19:15:48Z

Test build #36015 has started for PR 7086 at commit 51b1f24.

SparkQA · 2015-06-29T19:49:39Z

Test build #36015 has finished for PR 7086 at commit 51b1f24.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-06-29T19:50:04Z

Merged build finished. Test PASSed.

JoshRosen · 2015-06-30T06:28:25Z

Jenkins, retest this please.

AmplabJenkins · 2015-06-30T06:32:13Z

Merged build triggered.

AmplabJenkins · 2015-06-30T06:32:19Z

Merged build started.

SparkQA · 2015-06-30T06:35:27Z

Test build #36102 has started for PR 7086 at commit 51b1f24.

SparkQA · 2015-06-30T07:06:03Z

Test build #36102 has finished for PR 7086 at commit 51b1f24.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-06-30T07:06:09Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-06-30T13:07:12Z

Merged build triggered.

AmplabJenkins · 2015-06-30T13:07:22Z

Merged build started.

SparkQA · 2015-06-30T13:08:31Z

Test build #36135 has started for PR 7086 at commit 6180b60.

SparkQA · 2015-06-30T13:43:14Z

Test build #36135 has finished for PR 7086 at commit 6180b60.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-06-30T13:43:21Z

Merged build finished. Test FAILed.

SparkQA · 2015-07-01T15:24:19Z

Test build #36274 has started for PR 7086 at commit 2715b25.

MechCoder · 2015-07-01T15:24:40Z

@mengxr

I just checked that HashingTF inherits from Has NumFeatures so there is not really anything to do for the first point.

I've just added the methods in scala and python in this PR and it should be open for review now.

SparkQA · 2015-07-01T15:55:30Z

Test build #36274 has finished for PR 7086 at commit 2715b25.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-07-01T15:55:37Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-07-01T16:03:14Z

Merged build triggered.

AmplabJenkins · 2015-07-01T16:03:21Z

Merged build started.

SparkQA · 2015-07-01T16:06:41Z

Test build #36279 has started for PR 7086 at commit 3334d31.

SparkQA · 2015-07-01T16:47:18Z

Test build #36279 has finished for PR 7086 at commit 3334d31.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-07-01T16:47:47Z

Merged build finished. Test PASSed.

MechCoder · 2015-07-07T17:40:48Z

@mengxr updated. I thought it might be slightly overkill to add the scala and python version separately :P

AmplabJenkins · 2015-07-07T17:43:13Z

Merged build triggered.

AmplabJenkins · 2015-07-07T17:43:20Z

Merged build started.

SparkQA · 2015-07-07T17:44:31Z

Test build #36700 has started for PR 7086 at commit 6e3d6b2.

SparkQA · 2015-07-07T17:46:21Z

Test build #36700 has finished for PR 7086 at commit 6e3d6b2.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-07-07T17:46:22Z

Merged build finished. Test FAILed.

mengxr · 2015-07-07T18:37:35Z

mllib/src/main/scala/org/apache/spark/ml/feature/StandardScaler.scala

need the explicit type info for public methods

mengxr · 2015-07-07T18:38:17Z

LGTM except the Scala style issue.

AmplabJenkins · 2015-07-07T18:53:15Z

Merged build triggered.

AmplabJenkins · 2015-07-07T18:53:20Z

Merged build started.

SparkQA · 2015-07-07T18:54:01Z

Test build #36706 has started for PR 7086 at commit 9fbae90.

SparkQA · 2015-07-07T19:26:35Z

Test build #36706 has finished for PR 7086 at commit 9fbae90.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DecisionTreeClassificationModel(DecisionTreeModel):
- class RandomForestClassificationModel(TreeEnsembleModels):
- class GBTClassificationModel(TreeEnsembleModels):
- class DecisionTreeModel(JavaModel):
- class TreeEnsembleModels(JavaModel):
- class DecisionTreeRegressionModel(DecisionTreeModel):
- class RandomForestRegressionModel(TreeEnsembleModels):
- class GBTRegressionModel(TreeEnsembleModels):
- case class InSet(child: Expression, hset: Set[Any])

AmplabJenkins · 2015-07-07T19:27:03Z

Merged build finished. Test PASSed.

mengxr · 2015-07-07T19:37:15Z

Merged into master. Thanks!

MechCoder force-pushed the missing_model_methods branch from 2715b25 to 3334d31 Compare July 1, 2015 16:00

[SPARK-8704] Add missing methods in StandardScaler (ML and PySpark)

6e3d6b2

MechCoder force-pushed the missing_model_methods branch from 3334d31 to 6e3d6b2 Compare July 7, 2015 17:39

MechCoder changed the title ~~[SPARK-8704] [ML] [PySpark] Add additional methods to wrappers in ml.pyspark.feature~~ [SPARK-8704] [ML] [PySpark] Add missing methods in StandardScaler Jul 7, 2015

mengxr reviewed Jul 7, 2015
View reviewed changes

mllib/src/main/scala/org/apache/spark/ml/feature/StandardScaler.scala Outdated

Copy link

Contributor

mengxr Jul 7, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need the explicit type info for public methods

Add type

9fbae90

asfgit closed this in 35d781e Jul 7, 2015

MechCoder deleted the missing_model_methods branch July 7, 2015 19:39

[SPARK-8704] [ML] [PySpark] Add missing methods in StandardScaler #7086

[SPARK-8704] [ML] [PySpark] Add missing methods in StandardScaler #7086

Uh oh!

Conversation

MechCoder commented Jun 29, 2015

Uh oh!

AmplabJenkins commented Jun 29, 2015

Uh oh!

AmplabJenkins commented Jun 29, 2015

Uh oh!

SparkQA commented Jun 29, 2015

Uh oh!

MechCoder commented Jun 29, 2015

Uh oh!

SparkQA commented Jun 29, 2015

Uh oh!

AmplabJenkins commented Jun 29, 2015

Uh oh!

MechCoder commented Jun 29, 2015

Uh oh!

AmplabJenkins commented Jun 29, 2015

Uh oh!

AmplabJenkins commented Jun 29, 2015

Uh oh!

MechCoder commented Jun 29, 2015

Uh oh!

SparkQA commented Jun 29, 2015

Uh oh!

SparkQA commented Jun 29, 2015

Uh oh!

AmplabJenkins commented Jun 29, 2015

Uh oh!

MechCoder commented Jun 29, 2015

Uh oh!

AmplabJenkins commented Jun 29, 2015

Uh oh!

AmplabJenkins commented Jun 29, 2015

Uh oh!

SparkQA commented Jun 29, 2015

Uh oh!

SparkQA commented Jun 29, 2015

Uh oh!

AmplabJenkins commented Jun 29, 2015

Uh oh!

JoshRosen commented Jun 30, 2015

Uh oh!

AmplabJenkins commented Jun 30, 2015

Uh oh!

AmplabJenkins commented Jun 30, 2015

Uh oh!

SparkQA commented Jun 30, 2015

Uh oh!

SparkQA commented Jun 30, 2015

Uh oh!

AmplabJenkins commented Jun 30, 2015

Uh oh!

AmplabJenkins commented Jun 30, 2015

Uh oh!

AmplabJenkins commented Jun 30, 2015

Uh oh!

SparkQA commented Jun 30, 2015

Uh oh!

SparkQA commented Jun 30, 2015

Uh oh!

AmplabJenkins commented Jun 30, 2015

Uh oh!

SparkQA commented Jul 1, 2015

Uh oh!

MechCoder commented Jul 1, 2015

Uh oh!

SparkQA commented Jul 1, 2015

Uh oh!

AmplabJenkins commented Jul 1, 2015

Uh oh!

AmplabJenkins commented Jul 1, 2015

Uh oh!

AmplabJenkins commented Jul 1, 2015

Uh oh!

SparkQA commented Jul 1, 2015

Uh oh!