[Spark-8169] [ML] Add StopWordsRemover as a transformer

[hhbyyh]

jira: https://issues.apache.org/jira/browse/SPARK-8169

stop words: http://en.wikipedia.org/wiki/Stop_words

StopWordsRemover takes a string array column and outputs a string array column with all defined stop words removed. The transformer should also come with a standard set of stop words as default.

Currently I used a minimum stop words set since on some case, small set of stop words is preferred.
ASCII char has been tested, Yet I cannot check it in due to style check.

Further thought,

1. Maybe I should use OpenHashSet. Is it recommended?
2. Currently I leave the null in input array untouched, i.e. Array(null, null) => Array(null, null).
3. If the current stop words set looks too limited, any suggestion for replacement? We can have something similar to the one in SKlearn.

[SparkQA]

Test build #34592 has finished for PR 6742 at commit b3aa957.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class StopWordsRemover(override val uid: String)

[feynmanliang]

It would be great if this was some standard stopwords list like the SKLearn list you mentioned or this one from CoreNLP. Please include a reference in the docs if you use it.

[mengxr]

Please address @feynmanliang 's comment.

[hhbyyh]

I will use the one from SKLearn. Please let me know if there's other suggestion.

[feynmanliang]

1. Maybe I should use OpenHashSet. Is it recommended?

   In my opinion set should suffice. OpenHashSet optimizes for Long and Int keys, but you are using String keys here. Perhaps some perf benchmarks may help here.

2. Currently I leave the null in input array untouched, i.e. Array(null, null) => Array(null, null).

   I imagine this will be used downstream of Tokenizer, in which case you should not need to worry about nulls. Perhaps document the behavior in the ScalaDocs.

3. If the current stop words set looks too limited, any suggestion for replacement? We can have something similar to the one in SKlearn.

   See in-line comments.

Also, it may be useful to think how this will interact with #7084 since we could just supply a vocabulary without stop words.

[mengxr]

Please make this private. We only need to mention where we get the default list of stop words.

[hhbyyh]

OK. I'm just thinking users may write
val stopWords = StopWords.EnglishStopWords ++ Array("python", "scala") in some occasion.

[mengxr]

Users can get the default value from the StopWordsRemover instance.

[mengxr]

@hhbyyh Sorry for late review! If you don't have time to address my comments before the feature freeze, please let me know:)

[hhbyyh]

@mengxr I've started working on it and will try to send an update within one hour.

[hhbyyh]

Sorry for the delay due to time difference. Update sent.

[SparkQA]

Test build #39141 has finished for PR 6742 at commit f190217.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class StopWordsRemover(override val uid: String)

[mengxr]

space before {

[mengxr]

Further thought,

1. Maybe I should use OpenHashSet. Is it recommended?

It might be faster. We can compare the performance during QA. Let's use Set for simplicity in this PR.

1. Currently I leave the null in input array untouched, i.e. Array(null, null) => Array(null, null).

See my inline comments. I would recommend treating null as a normal word.

[hhbyyh]

@mengxr Update sent: Separating udf, adding note about preserving null, and other style fix. Thanks.

[SparkQA]

Test build #39310 has finished for PR 6742 at commit f190217.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class StopWordsRemover(override val uid: String)

[SparkQA]

Test build #39314 has finished for PR 6742 at commit fa959d8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class StopWordsRemover(override val uid: String)

[mengxr]

LGTM. Merged into master. Thanks!

[hhbyyh]

Thanks for taking time finish the review. It must have been a long day.

Shall I create jira for Python and document? @mengxr

[mengxr]

Yes, please create JIRAs for Python API and doc. Thanks!