Add additional Analyzers, Tokenizers, and TokenFilters from Lucene #6693
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… Lucene Add `irish` analyzer Add `sorani` analyzer (Kurdish) Add `classic` tokenizer: specific to english text and tries to recognize hostnames, companies, acronyms, etc. Add `thai` tokenizer: segments thai text into words. Add `classic` tokenfilter: cleans up acronyms and possessives from classic tokenizer Add `apostrophe` tokenfilter: removes text after apostrophe and the apostrophe itself Add `german_normalization` tokenfilter: umlaut/sharp S normalization Add `hindi_normalization` tokenfilter: accounts for hindi spelling differences Add `indic_normalization` tokenfilter: accounts for different unicode representations in Indian languages Add `sorani_normalization` tokenfilter: normalizes kurdish text Add `scandinavian_normalization` tokenfilter: normalizes Norwegian, Danish, Swedish text Add `scandinavian_folding` tokenfilter: much more aggressive form of `scandinavian_normalization` Add additional languages to stemmer tokenfilter: `galician`, `minimal_galician`, `irish`, `sorani`, `light_nynorsk`, `minimal_nynorsk` Add support access to default Thai stopword set "_thai_" Fix some bugs and broken links in documentation. Closes elastic#5935
|
Yay! |
|
LGTM this is great! |
|
LGTM good stuff |
rmuir
added a commit
to rmuir/elasticsearch-definitive-guide
that referenced
this pull request
Jul 3, 2014
clintongormley
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add
irishanalyzerAdd
soranianalyzer (Kurdish)Add
classictokenizer: specific to english text and tries to recognize hostnames, companies, acronyms, etc.Add
thaitokenizer: segments thai text into words.Add
classictokenfilter: cleans up acronyms and possessives from classic tokenizerAdd
apostrophetokenfilter: removes text after apostrophe and the apostrophe itselfAdd
german_normalizationtokenfilter: umlaut/sharp S normalizationAdd
hindi_normalizationtokenfilter: accounts for hindi spelling differencesAdd
indic_normalizationtokenfilter: accounts for different unicode representations in Indian languagesAdd
sorani_normalizationtokenfilter: normalizes kurdish textAdd
scandinavian_normalizationtokenfilter: normalizes Norwegian, Danish, Swedish textAdd
scandinavian_foldingtokenfilter: much more aggressive form ofscandinavian_normalizationAdd additional languages to stemmer tokenfilter:
galician,minimal_galician,irish,sorani,light_nynorsk,minimal_nynorskAdd support access to default Thai stopword set "thai"
Fix some bugs and broken links in documentation.
Closes #5935