Add inter segment tests for text search and fix bug for Lucene query parser creation #5226
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
and fix bug for lucene query parser creation Parser is JavaCC based. So it has to be created per query. Also added text search tests for the SQL path
siddharthteotia
force-pushed
the
lucene_parser
branch
from
facc49c
to
43dc5da
Compare
Jackie-Jiang
approved these changes
Apr 8, 2020
| @@ -133,7 +131,8 @@ public MutableRoaringBitmap getDocIds(Object value) { | |||
| MutableRoaringBitmap docIds = new MutableRoaringBitmap(); | |||
| Collector docIDCollector = new LuceneDocIdCollector(docIds, _docIdTranslator); | |||
| try { | |||
| Query query = _queryParser.parse(searchQuery); | |||
| QueryParser parser = new QueryParser(_column, new StandardAnalyzer()); | |||
There was a problem hiding this comment.
Is there any way to reuse some components here? Will this cause too much garbage?
Please also add some comments so that future developers know that it cannot be reused.
There was a problem hiding this comment.
So the parser is JavaCC based and so it is stateful. We have to instantiate it per query (this is the bug). Analyzer on the other hand seems stateless so I reverted that part to be instantiated just once.
|
Ran the benchmark again locally to compare the numbers to previous run (when the doc id cache optimizations were implemented #5199). No difference. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add inter segment tests for text search and fix bug for Lucene query parser creation. Parser should be instantiated per query (search expression). Also added text search tests for the SQL path