Prune stop words for text index #5297
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kishoreg
reviewed
Apr 24, 2020
| @@ -50,6 +58,15 @@ | |||
| private final Directory _indexDirectory; | |||
| private final IndexWriter _indexWriter; | |||
|
|
|||
| public static final CharArraySet ENGLISH_STOP_WORDS_SET = | |||
| new CharArraySet(Arrays.asList( | |||
There was a problem hiding this comment.
Dont we have this in Lucene StopAnalyzer?
StopAnalyzer.ENGLISH_STOP_WORDS_SET
There was a problem hiding this comment.
No this is not in StopAnalyzer. That also takes a passed list of stopwords.
The list is borrowed from EnglishAnalyzer. The reason for not directly using EnglishAnalyzer is because it does word stemming and reduces words to their root form. I don't think we need that.
There was a problem hiding this comment.
StopAnalyzer can't be used either because it tokenizes input text at each character. So we stick to StandardAnalyzer which uses StandardTokenizer (based on unicode general purpose text segmentation algorithm)
There was a problem hiding this comment.
Thanks add this comment in the code.
kishoreg
approved these changes
Apr 27, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For general english text, indexing the stop words can easily lead to explosion in the size of text index and consequently impact the performance of queries. For now use a predefined list of general stop words that can be passed to the text analyzer both during indexing and querying. The analyzer ensures that stop words in input text are pruned and not indexed. Similarly, on the query path, if stop words are present in search expression, the matching is done without stop words.
This is something that can possibly be specified on a per use case (per text index basis). Currently we have a hardcoded list of common english stop words.