Improvements to StemmerTokenFilter #6452
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This looks great: I especially like the cleanup to the code organization and the doc links. |
| @@ -85,90 +89,116 @@ public TokenStream create(TokenStream tokenStream) { | |||
| return new CzechStemFilter(tokenStream); | |||
| } else if ("danish".equalsIgnoreCase(language)) { | |||
| return new SnowballFilter(tokenStream, new DanishStemmer()); | |||
|
|
|||
There was a problem hiding this comment.
can you remove this additional line? ;)
There was a problem hiding this comment.
The extra lines are there to break up the language blocks, eg English, vs French etc. I've added comments for each block with multiple languages
|
@jpountz Please could you check that I'm using the index versions correctly here |
|
@clintongormley It looks good to me. I really like the fact that we now recommend on a stemmer! |
* `english` returned the slow snowball English stemmer * `porter2` returned the snowball Porter stemmer (v1) * `portuguese` was used twice, preventing the second version from working Changes: * `english` now returns the fast PorterStemmer (for indices created from v1.3.0 onwards) * `porter2` now returns the snowball English stemmer (for indices created from v1.3.0 onwards) * `light_english` now returns the `kstem` stemmer (`kstem` still works) * `portuguese_rslp` returns the PortugueseStemmer * `dutch_kp` is a synonym for `kp` Tests and docs updated Fixes #6345 Fixes #6213 Fixes #6330
clintongormley
changed the title
rboulton
pushed a commit
to alphagov/search-api
that referenced
this pull request
Jun 8, 2015
Elasticsearch 1.3.0 changed the stemmer implementation used when the "english" stemmer is requested. Instead of using the stemmer as revised by Martin Porter in 2001/2002 to avoid lots of incorrect over-stemming and other problems, elasticsearch now defaults to using the algorithm defined in 1980. This is an entirely inferior algorithm in terms of the output it produces. To get the better algorithm back, we now need to request the "porter2" algorithm. This has made me grumpy. The reason that snowball returns the "english" stemmer instead of the "porter" stemmer when it's asked for a stemmer for "english" is exactly because the "porter" stemmer is inferior in every case where it differs from the "english" stemmer. However, elasticsearch elastic/elasticsearch#6452 changed the default, apparently because the implementation of the 1980 Porter stemmer available is faster. In particular, this change has made a search for "news" on GOV.UK return anything with the word "new" in it.
This was referenced Jun 8, 2015
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The StemmerTokenFilter had a number of issues:
englishreturned the slow snowball English stemmerporter2returned the snowball Porter stemmer (v1)portuguesewas used twice, preventing the second version from workingChanges:
englishnow returns the fast PorterStemmer (for indices created from v1.3.0 onwards)porter2now returns the snowball English stemmer (for indices created from v1.3.0 onwards)light_englishnow returns thekstemstemmer (kstemstill works)portuguese_rslpreturns the PortugueseStemmerdutch_kpis a synonym forkpTests and docs updated