Fix exponential query sharding time of binops #3027
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When sharding operations like `expr1 or expr2 or ... or expr_n`, the last expression has to be cloned 2^n times by the `canShardAllVectorSelectors` check. We can avoid that by caching the results of that function for each expr, making the execution time linear again. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
pracucci
deleted the
fix-query-sharding-exponential-processing-time-of-binary-expressions
branch
pracucci
pushed a commit
that referenced
this pull request
Sep 23, 2022
* Fix exponential query sharding time of binops When sharding operations like `expr1 or expr2 or ... or expr_n`, the last expression has to be cloned 2^n times by the `canShardAllVectorSelectors` check. We can avoid that by caching the results of that function for each expr, making the execution time linear again. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Update CHANGELOG.md Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Update test to check 30 expressions Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
pracucci
pushed a commit
that referenced
this pull request
Sep 23, 2022
* Fix exponential query sharding time of binops When sharding operations like `expr1 or expr2 or ... or expr_n`, the last expression has to be cloned 2^n times by the `canShardAllVectorSelectors` check. We can avoid that by caching the results of that function for each expr, making the execution time linear again. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Update CHANGELOG.md Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Update test to check 30 expressions Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
krajorama
reviewed
Sep 23, 2022
| if can, ok := summer.canShardAllVectorSelectorsCache[query]; ok { | ||
| return can | ||
| } | ||
| defer func() { summer.canShardAllVectorSelectorsCache[query] = can }() |
There was a problem hiding this comment.
how is this caching the result of the function which is c.stats.GetShardedQueries() == countVectorSelectors(mappedExpr)
There was a problem hiding this comment.
can is a named return value. So return statement assigns the value to it, and then the defer statement is executed.
pracucci
pushed a commit
that referenced
this pull request
Sep 23, 2022
* Fix exponential query sharding time of binops When sharding operations like `expr1 or expr2 or ... or expr_n`, the last expression has to be cloned 2^n times by the `canShardAllVectorSelectors` check. We can avoid that by caching the results of that function for each expr, making the execution time linear again. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Update CHANGELOG.md Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Update test to check 30 expressions Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
3 tasks
andyasp
pushed a commit
that referenced
this pull request
Sep 26, 2022
* Fix exponential query sharding time of binops When sharding operations like `expr1 or expr2 or ... or expr_n`, the last expression has to be cloned 2^n times by the `canShardAllVectorSelectors` check. We can avoid that by caching the results of that function for each expr, making the execution time linear again. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Update CHANGELOG.md Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Update test to check 30 expressions Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com>
|
The backport to To backport manually, run these commands in your terminal: # Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-3027-to-release-2.3 origin/release-2.3
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x 0e45c39fa33ddefe0ceea402a6d15a21b46f4752
# Push it to GitHub
git push --set-upstream origin backport-3027-to-release-2.3
git switch main
# Remove the local backport branch
git branch -D backport-3027-to-release-2.3Then, create a pull request where the |
treid314
pushed a commit
that referenced
this pull request
Sep 26, 2022
* Fix exponential query sharding time of binops When sharding operations like `expr1 or expr2 or ... or expr_n`, the last expression has to be cloned 2^n times by the `canShardAllVectorSelectors` check. We can avoid that by caching the results of that function for each expr, making the execution time linear again. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Update CHANGELOG.md Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Update test to check 30 expressions Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> (cherry picked from commit 0e45c39)
treid314
pushed a commit
that referenced
this pull request
Sep 26, 2022
* Fix exponential query sharding time of binops When sharding operations like `expr1 or expr2 or ... or expr_n`, the last expression has to be cloned 2^n times by the `canShardAllVectorSelectors` check. We can avoid that by caching the results of that function for each expr, making the execution time linear again. Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Update CHANGELOG.md Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> * Update test to check 30 expressions Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> Signed-off-by: Oleg Zaytsev <mail@olegzaytsev.com> (cherry picked from commit 0e45c39) Co-authored-by: Oleg Zaytsev <mail@olegzaytsev.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does
When sharding operations like
expr1 or expr2 or ... or expr_n, the last expression has to be cloned2^ntimes by thecanShardAllVectorSelectorscheck.We can avoid that by caching the results of that function for each expr, making the execution time linear again.
Which issue(s) this PR fixes or relates to
Fixes an internal issue.
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]