[SPARK-9679][ML][PYSPARK] Add Python API for Stop Words Remover #8118

holdenk · 2015-08-12T00:22:47Z

Add a python API for the Stop Words Remover.

holdenk · 2015-08-12T00:40:11Z

jenkins, retest this please.

SparkQA · 2015-08-12T23:18:51Z

Test build #40679 has finished for PR 8118 at commit ec6baab.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- * Set thresholds in multiclass (or binary) classification to adjust the probability of
- class StopWordsRemover(JavaTransformer, HasInputCol, HasOutputCol):

holdenk · 2015-08-13T22:10:23Z

jenkins, retest this please

holdenk · 2015-08-13T23:42:43Z

jenkins, retest this please.

SparkQA · 2015-08-14T00:10:17Z

Test build #40830 has finished for PR 8118 at commit b98bb1d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-18T23:10:20Z

Test build #41169 has finished for PR 8118 at commit d8e3672.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

feynmanliang · 2015-08-25T23:07:18Z

python/pyspark/ml/feature.py

+    @keyword_only
+    def __init__(self, inputCol=None, outputCol=None, stopWords=[]):
+        """
+        Initialize this instace of the StopWordsRemover.


Is there a reason why this __init__ doc string breaks the pattern of just repeating the method with default args seen elsewhere in feature.py?

… use english stop words as default

SparkQA · 2015-08-26T22:46:48Z

Test build #41654 has finished for PR 8118 at commit 84bc507.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StopWordsRemover(JavaTransformer, HasInputCol, HasOutputCol):

feynmanliang · 2015-08-27T02:39:29Z

python/pyspark/ml/feature.py

+                                   "sensitive comparison over the stop words")
+        stopWordsObj = _jvm().org.apache.spark.ml.feature.StopWords
+        defaultStopWords = stopWordsObj.ENGLISH_STOP_WORDS()
+        print "Constructing java param pair for value "+str(defaultStopWords)


Are these prints intentional?

oh no, I was checking the type when debugging something

feynmanliang · 2015-08-27T02:42:14Z

some small comments, LGTM after they're fixed

SparkQA · 2015-08-27T05:14:01Z

Test build #41672 has finished for PR 8118 at commit acfc9fe.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StopWordsRemover(JavaTransformer, HasInputCol, HasOutputCol):

holdenk · 2015-08-27T08:00:30Z

jenkins, retest this please.

SparkQA · 2015-08-27T08:44:56Z

Test build #41679 has finished for PR 8118 at commit 53f97b7.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StopWordsRemover(JavaTransformer, HasInputCol, HasOutputCol):

mengxr · 2015-08-28T05:10:33Z

mllib/src/main/scala/org/apache/spark/ml/feature/StopWordsRemover.scala

@@ -29,14 +29,14 @@ import org.apache.spark.sql.types.{ArrayType, StringType, StructField, StructTyp
 /**
 * stop words list
 */
-private object StopWords {
+protected[spark] object StopWords {


private[spark] should be the same but appears more often

mengxr · 2015-08-28T05:10:37Z

mllib/src/main/scala/org/apache/spark/ml/feature/StopWordsRemover.scala


  /**
   * Use the same default stopwords list as scikit-learn.
   * The original list can be found from "Glasgow Information Retrieval Group"
   * [[http://ir.dcs.gla.ac.uk/resources/linguistic_utils/stop_words]]
   */
-  val EnglishStopWords = Array( "a", "about", "above", "across", "after", "afterwards", "again",
+  val ENGLISH_STOP_WORDS = Array( "a", "about", "above", "across", "after", "afterwards", "again",


minor: Since the object is already StopWords, would English be sufficient? We didn't use ENGLISH_STOP_WORDS because it is a mutable array.

SparkQA · 2015-08-28T21:55:46Z

Test build #41761 has finished for PR 8118 at commit 7767df0.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StopWordsRemover(JavaTransformer, HasInputCol, HasOutputCol):

SparkQA · 2015-08-28T22:50:53Z

Test build #41764 has finished for PR 8118 at commit 345bde2.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StopWordsRemover(JavaTransformer, HasInputCol, HasOutputCol):

mengxr · 2015-08-31T23:03:44Z

LGTM except a minor issue on the test code style.

SparkQA · 2015-09-01T01:40:45Z

Test build #41848 has finished for PR 8118 at commit 62b821a.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class DCT(JavaTransformer, HasInputCol, HasOutputCol):
- class SQLTransformer(JavaTransformer):
- class StopWordsRemover(JavaTransformer, HasInputCol, HasOutputCol):

mengxr · 2015-09-01T17:51:34Z

Merged into master. Thanks!

holdenk added 3 commits August 11, 2015 16:19

Start work towards adding stopwordpython interface.

78c0362

pep8 fix (add second blank line between classes)

7a65dc3

Make the stopwrods param settable from Python & Java, fix test

c634fa1

Merge in master

ec6baab

merge

b98bb1d

merge in master

d8e3672

feynmanliang mentioned this pull request Aug 25, 2015

[SPARK-9680][MLlib][Doc] StopWordsRemovers user guide and Java compatibility test #8436

Closed

feynmanliang reviewed Aug 25, 2015
View reviewed changes

holdenk added 3 commits August 26, 2015 13:49

Some progress on code review, add caseSensitive option for stopwords,…

12fb73e

… use english stop words as default

Merge branch 'master' into SPARK-9679-python-StopWordsRemover

c366fd6

Use the english stop words from scala by default

84bc507

feynmanliang reviewed Aug 27, 2015
View reviewed changes

CR feedback (whitespace and remove prints)

acfc9fe

new constant for english stop words

53f97b7

mengxr reviewed Aug 28, 2015
View reviewed changes

CR feedback from mengxr

7767df0

holdenk added 2 commits August 28, 2015 14:59

Merge branch 'master' into SPARK-9679-python-StopWordsRemover

e36703e

Add an extra blank line that ended up missing after the automatic merge

345bde2

holdenk added 2 commits August 31, 2015 17:43

merge in master

c6dee1b

CR feedback

62b821a

asfgit closed this in e6e483c Sep 1, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-9679][ML][PYSPARK] Add Python API for Stop Words Remover #8118

[SPARK-9679][ML][PYSPARK] Add Python API for Stop Words Remover #8118

holdenk commented Aug 12, 2015

holdenk commented Aug 12, 2015

SparkQA commented Aug 12, 2015

holdenk commented Aug 13, 2015

holdenk commented Aug 13, 2015

SparkQA commented Aug 14, 2015

SparkQA commented Aug 18, 2015

feynmanliang Aug 25, 2015

SparkQA commented Aug 26, 2015

feynmanliang Aug 27, 2015

holdenk Aug 27, 2015

feynmanliang commented Aug 27, 2015

SparkQA commented Aug 27, 2015

holdenk commented Aug 27, 2015

SparkQA commented Aug 27, 2015

mengxr Aug 28, 2015

mengxr Aug 28, 2015

SparkQA commented Aug 28, 2015

SparkQA commented Aug 28, 2015

mengxr commented Aug 31, 2015

SparkQA commented Sep 1, 2015

mengxr commented Sep 1, 2015

[SPARK-9679][ML][PYSPARK] Add Python API for Stop Words Remover #8118

[SPARK-9679][ML][PYSPARK] Add Python API for Stop Words Remover #8118

Conversation

holdenk commented Aug 12, 2015

holdenk commented Aug 12, 2015

SparkQA commented Aug 12, 2015

holdenk commented Aug 13, 2015

holdenk commented Aug 13, 2015

SparkQA commented Aug 14, 2015

SparkQA commented Aug 18, 2015

feynmanliang Aug 25, 2015

Choose a reason for hiding this comment

SparkQA commented Aug 26, 2015

feynmanliang Aug 27, 2015

Choose a reason for hiding this comment

holdenk Aug 27, 2015

Choose a reason for hiding this comment

feynmanliang commented Aug 27, 2015

SparkQA commented Aug 27, 2015

holdenk commented Aug 27, 2015

SparkQA commented Aug 27, 2015

mengxr Aug 28, 2015

Choose a reason for hiding this comment

mengxr Aug 28, 2015

Choose a reason for hiding this comment

SparkQA commented Aug 28, 2015

SparkQA commented Aug 28, 2015

mengxr commented Aug 31, 2015

SparkQA commented Sep 1, 2015

mengxr commented Sep 1, 2015