[SPARK-18374][ML]Incorrect words in StopWords/english.txt

[hhbyyh]

What changes were proposed in this pull request?

Currently English stop words list in MLlib contains only the argumented words after removing all the apostrophes, so "wouldn't" become "wouldn" and "t". Yet by default Tokenizer and RegexTokenizer don't split on apostrophes or quotes.

Adding original form to stop words list to match the behavior of Tokenizer and StopwordsRemover. Also remove "won" from list.

see more discussion in the jira: https://issues.apache.org/jira/browse/SPARK-18374

How was this patch tested?

existing ut

[hhbyyh]

@srowen Thanks for the comments in jira.

[SparkQA]

Test build #69489 has finished for PR 16103 at commit efeae8c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

Oh @hhbyyh, I think I am not used to this area but actually I just wanted leave a comment just for stop words list that I considered for my elasticsearch cluster before. I just made a list for missing ones via

target = open("mllib/src/main/resources/org/apache/spark/ml/feature/stopwords/english.txt").read().split("\n")
source = open("mylist.txt").read().split("\n")
for w in source:
    if w not in target:
        print w

and I got these below:

cannot
could
here's
how's
let's
ought
that's
there's
what's
when's
where's
who's
why's
would

I am not sure if they should be added here or not but I just wanted to let you know what I considered before for stop words.

[srowen]

Seems reasonable to me too.

[hhbyyh]

Thanks @HyukjinKwon. I'll add them.

[SparkQA]

Test build #69583 has finished for PR 16103 at commit 4cec57a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

You would then remove the other stems like "wasn" "weren" etc right?

[hhbyyh]

I'm fine with both options, leaving them or removing them.

[srowen]

I think we should remove the spurious stems; the info on this PR suggests they aren't supposed to be there.

[SparkQA]

Test build #69675 has finished for PR 16103 at commit aa8c72a.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #69677 has finished for PR 16103 at commit bd62396.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[hhbyyh]

Thanks for the review.

[srowen]

Merged to master