Wildcard query support with suffix trie #2932
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
👍
Added some comments. My main concerns is about assumption of NULL terminated strings. Maybe we should just organise and document those assumptions.
Also, I did not see tests that verify it on FT.AGGREAGTEA and on coordinator, I agree it should not make any different but lets verify.
Also, any documentation that we need to update?
MeirShpilraien
previously approved these changes
Aug 4, 2022
MeirShpilraien
approved these changes
Aug 8, 2022
oshadmi
pushed a commit
that referenced
this pull request
Aug 9, 2022
* parser work * wip * TEXT field is working * wip * TAG works * benchmark * wip * working w/o params * with params * clean params * cleanup + streamline * added TM_WILDCARD_FIXED_LEN_MODE * fix tests * llvm warning clear * add ft.profile tests * change to Wildcard_MatchType * add unit tests * add comments * suffix support for TAG * wip * quit early with no stars * per meir review * update comment * wip * wildcard Suffix TEXT * clean * fix * cleanup * clean * add score by suffix token index * fix leak * better func names * add upper lower case test * per review comments * fallback from suffix to brute force * remove redundant function * remove extra environment * fix test
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR complements PR #2886 and adds an optimization to
wildcard queriesutilizing a suffix trie which was introduced in PR #2774 forcontains queries.The wildcard pattern is broken into tokens at '*' character and an estimation is made to find the token which would require the least processing.
The suffix trie is then iterated, and the payload of the relative nodes, which contains a list of words, is returned.
Each word in the list is checked against the pattern, and all matches are added to the union iterator.
If a wildcard pattern does not support using suffix trie (tokens must be 2 chars or more), the brute-force function is used instead.
Main changes:
Suffix_StarBreakfunctions receive either char or rune pattern and break it into tokens at*. A heuristic is used to determine the best token to use on the suffix trie. The initial score is the length of the token. Each '?' reduces a point, and having a '*' at the end reduces further 5 points (since all children have to be iterated).SuffixCtxwas added for ease of use.TrieRangeCallback, the callback of the iterator ofTrie, now return the payload. It is used to access the payload, which contains theSuffixDatastruct with all matching words.recursiveAddto return an error. The error can be produced by the callback function ifmaxPrefixExpansionsis reached.related to #2886
MOD-3284
MOD-3752