Unnecessary blank lines found in stopwords.txt of SmartChineseAnalyzer

[JerryChin]

Description

Hi team,

This issue is a spin-off from the java-user list thread.

The stopwords.txt of SmartChineseAnalyzer contains two blank lines at L56 & L58. As a result, SmartChineseAnalyzer.getDefaultStopSet() will produce an empty string stop word, but it makes no sense to have empty string as a stop word.

Maybe we can improve it?

[tang-hi]

Good Catch! Could you submit a PR to fix that?

[uschindler]

In general I'd suggest to figure out if we should not change the stopword file parser to strip blankempty lines like comments?

[mikemccand]

I think the stoplist loader already ignores comment lines, but, does not ignore empty lines! Darned empty string rears its head at us again...

[JerryChin]

Hi @tang-hi,

I can summit a PR to fix this issue.

How about skipping blank line before

lucene/lucene/core/src/java/org/apache/lucene/analysis/WordlistLoader.java

Line 56 in 5d203f8

result.add(word.trim());

and remove the blank lines from stopwords.txt.

What do you think?

[uschindler]

Also here:

lucene/lucene/core/src/java/org/apache/lucene/analysis/WordlistLoader.java

Line 119 in 5d203f8

if (word.startsWith(comment) == false) {