[SPARK-9654][ML][PYSPARK] Add IndexToString to PySpark #7976

holdenk · 2015-08-05T23:15:04Z

Adds IndexToString to PySpark.

SparkQA · 2015-08-05T23:37:31Z

Test build #39934 has finished for PR 7976 at commit 91e6ff3.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StringIndexerInverse(JavaTransformer):

SparkQA · 2015-08-06T00:23:55Z

Test build #39947 has finished for PR 7976 at commit 6c73ff8.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StringIndexerInverse(JavaTransformer):

SparkQA · 2015-08-06T00:59:01Z

Test build #39953 has finished for PR 7976 at commit 9b19da4.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StringIndexerInverse(JavaTransformer):

SparkQA · 2015-08-06T01:48:48Z

Test build #39970 has finished for PR 7976 at commit 1745fd4.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StringIndexerInverse(JavaTransformer):

SparkQA · 2015-08-06T02:29:06Z

Test build #39977 has finished for PR 7976 at commit 846984b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StringIndexerInverse(JavaTransformer):

jkbradley · 2015-08-11T19:01:49Z

python/pyspark/ml/feature.py

+        return StringIndexerInverse(self._java_obj.invert(inputCol, outputCol))
+
+
+class StringIndexerInverse(JavaTransformer):


This should also extend HasInputCol, HasOutputCol and have a "labels" Param.

holdenk · 2015-08-14T20:45:50Z

Note: Since the underlying Scala code has changed this is not currently good to merge. I'm waiting until @jkbradley's change to make the labels visible goes in before updating this.

…converting to regular string

holdenk · 2015-08-14T21:42:54Z

And the required PR is now in master.

SparkQA · 2015-08-14T22:33:34Z

Test build #40925 has finished for PR 7976 at commit e95b61b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StringIndexerModel (
- class IndexToString(JavaTransformer, HasInputCol, HasOutputCol):

jkbradley · 2015-08-18T00:21:51Z

mllib/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala

+  /**
+   * The labels used for applying this transformation
+   */
+  private[spark] def getLabels() = labels


no longer needed since "label" is a public val

feynmanliang · 2015-09-01T01:20:36Z

python/pyspark/ml/feature.py

+@inherit_doc
+class IndexToString(JavaTransformer, HasInputCol, HasOutputCol):
+    """
+    .. note:: Experimental


nit: line break

feynmanliang · 2015-09-01T01:27:34Z

LGTM, some really tiny nits

holdenk · 2015-09-01T04:34:22Z

@feynmanliang I'm not sure I agree with the changes to the param docs your asking for.

SparkQA · 2015-09-01T04:54:02Z

Test build #41854 has finished for PR 7976 at commit 28afcfd.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class IndexToString(JavaTransformer, HasInputCol, HasOutputCol):

feynmanliang · 2015-09-01T06:38:53Z

There's still a "." on L952.

I agree my wording is not the clearest, do you have any thoughts about improving it? In any case, I think this issue is quite small and I'm fine leaving it as. I just thought it was weird to use "is supplied" on one line and "the empty array is ignored" in another as if they were two different things; is it reasonable to expect new users to assume that not passing in any params leads to labels = []?

holdenk · 2015-09-01T06:48:12Z

Shouldn't L952 have a . on it?

SparkQA · 2015-09-01T06:56:44Z

Test build #41860 has finished for PR 7976 at commit f19445d.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DCT(JavaTransformer, HasInputCol, HasOutputCol):
- class SQLTransformer(JavaTransformer):
- class IndexToString(JavaTransformer, HasInputCol, HasOutputCol):

feynmanliang · 2015-09-01T07:29:18Z

Yes, sorry I was trying to point out that it currently doesn't have one and needs one added.

I like the doc change!

SparkQA · 2015-09-01T19:55:08Z

Test build #41887 has finished for PR 7976 at commit 3ef852f.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class IndexToString(JavaTransformer, HasInputCol, HasOutputCol):
- class StopWordsRemover(JavaTransformer, HasInputCol, HasOutputCol):

SparkQA · 2015-09-01T21:29:33Z

Test build #41895 has finished for PR 7976 at commit 41d0d27.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class JavaTrainValidationSplitExample
- class KMeans @Since("1.5.0") (
- class DCT(JavaTransformer, HasInputCol, HasOutputCol):
- class SQLTransformer(JavaTransformer):
- class IndexToString(JavaTransformer, HasInputCol, HasOutputCol):
- class StopWordsRemover(JavaTransformer, HasInputCol, HasOutputCol):
- case class LimitNode(limit: Int, child: LocalNode) extends UnaryLocalNode
- case class UnionNode(children: Seq[LocalNode]) extends LocalNode

jkbradley · 2015-09-08T21:26:18Z

python/pyspark/ml/feature.py

@@ -931,6 +937,63 @@ class StringIndexerModel(JavaModel):
    """
    Model fitted by StringIndexer.
    """
+    @property
+    def labels(self):


copy Scala doc: "Ordered list of labels, corresponding to indices to be assigned"

jkbradley · 2015-09-08T21:26:44Z

That should be it. Thanks!

SparkQA · 2015-09-09T01:04:14Z

Test build #42169 has finished for PR 7976 at commit 4f56b17.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class IndexToString(JavaTransformer, HasInputCol, HasOutputCol):

jkbradley · 2015-09-09T05:12:31Z

Merging with master. Thanks!

holdenk changed the title ~~[WIP][SPARK-9654][ML][PYSPARK] Add string indexer inverse in PySpark~~ [SPARK-9654][ML][PYSPARK] Add string indexer inverse in PySpark Aug 6, 2015

jkbradley reviewed Aug 11, 2015
View reviewed changes

holdenk changed the title ~~[SPARK-9654][ML][PYSPARK] Add string indexer inverse in PySpark~~ [SPARK-9654][ML][PYSPARK][WIP] Add string indexer inverse in PySpark Aug 14, 2015

holdenk added 9 commits August 14, 2015 14:18

SPARK-9654 Add string indexer inverse in PySpark

1dc4579

doc fix

0445fcc

Don't changge the base class init, fill out the doctest for the invert.

af2f869

remove extra blank line

510bce5

get rid of unicude specificers in doctest

c6da160

Deal with the difference between 2.X and 3.X with the output by just …

9f5af3a

…converting to regular string

Use the standard constructor method for the StringIndexInverse

7b3b5ca

Update for index to string changeover

244e083

Move the property on to the model, remove references to old class name

e95b61b

holdenk force-pushed the SPARK-9654-add-string-indexer-inverse-in-pyspark branch from ea655f3 to e95b61b Compare August 14, 2015 21:42

holdenk changed the title ~~[SPARK-9654][ML][PYSPARK][WIP] Add string indexer inverse in PySpark~~ [SPARK-9654][ML][PYSPARK] Add string indexer inverse in PySpark Aug 14, 2015

holdenk changed the title ~~[SPARK-9654][ML][PYSPARK] Add string indexer inverse in PySpark~~ [SPARK-9654][ML][PYSPARK] Add IndexToString to PySpark Aug 14, 2015

jkbradley reviewed Aug 18, 2015
View reviewed changes

feynmanliang reviewed Sep 1, 2015
View reviewed changes

Some CR feedback (note: still sorting our one of the params)

28afcfd

Change description text

f19445d

holdenk added 4 commits September 1, 2015 11:54

merge in master

51ae7ee

moar merge

ed0ca91

punctuation

8fca8b3

remove unrelated change

3ef852f

long line fix

41d0d27

jkbradley reviewed Sep 8, 2015
View reviewed changes

holdenk added 2 commits September 8, 2015 17:11

Add missing period

cd5d418

Fix link to transformer class, copy scala doc for labels

4f56b17

asfgit closed this in 2f6fd52 Sep 9, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-9654][ML][PYSPARK] Add IndexToString to PySpark #7976

[SPARK-9654][ML][PYSPARK] Add IndexToString to PySpark #7976

holdenk commented Aug 5, 2015

SparkQA commented Aug 5, 2015

SparkQA commented Aug 6, 2015

SparkQA commented Aug 6, 2015

SparkQA commented Aug 6, 2015

SparkQA commented Aug 6, 2015

jkbradley Aug 11, 2015

holdenk commented Aug 14, 2015

holdenk commented Aug 14, 2015

SparkQA commented Aug 14, 2015

jkbradley Aug 18, 2015

feynmanliang Sep 1, 2015

feynmanliang commented Sep 1, 2015

holdenk commented Sep 1, 2015

SparkQA commented Sep 1, 2015

feynmanliang commented Sep 1, 2015

holdenk commented Sep 1, 2015

SparkQA commented Sep 1, 2015

feynmanliang commented Sep 1, 2015

SparkQA commented Sep 1, 2015

SparkQA commented Sep 1, 2015

jkbradley Sep 8, 2015

jkbradley commented Sep 8, 2015

SparkQA commented Sep 9, 2015

jkbradley commented Sep 9, 2015

		return StringIndexerInverse(self._java_obj.invert(inputCol, outputCol))


		class StringIndexerInverse(JavaTransformer):

[SPARK-9654][ML][PYSPARK] Add IndexToString to PySpark #7976

[SPARK-9654][ML][PYSPARK] Add IndexToString to PySpark #7976

Conversation

holdenk commented Aug 5, 2015

SparkQA commented Aug 5, 2015

SparkQA commented Aug 6, 2015

SparkQA commented Aug 6, 2015

SparkQA commented Aug 6, 2015

SparkQA commented Aug 6, 2015

jkbradley Aug 11, 2015

Choose a reason for hiding this comment

holdenk commented Aug 14, 2015

holdenk commented Aug 14, 2015

SparkQA commented Aug 14, 2015

jkbradley Aug 18, 2015

Choose a reason for hiding this comment

feynmanliang Sep 1, 2015

Choose a reason for hiding this comment

feynmanliang commented Sep 1, 2015

holdenk commented Sep 1, 2015

SparkQA commented Sep 1, 2015

feynmanliang commented Sep 1, 2015

holdenk commented Sep 1, 2015

SparkQA commented Sep 1, 2015

feynmanliang commented Sep 1, 2015

SparkQA commented Sep 1, 2015

SparkQA commented Sep 1, 2015

jkbradley Sep 8, 2015

Choose a reason for hiding this comment

jkbradley commented Sep 8, 2015

SparkQA commented Sep 9, 2015

jkbradley commented Sep 9, 2015