select/pairs: don't ignore provided bucket_id #222
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Totktonada
force-pushed
the
Totktonada/gh-220-bucket-id-is-ignored
branch
from
fa62b8c to
124ba3e
Compare
ligurio
suggested changes
Oct 14, 2021
ligurio
suggested changes
Oct 14, 2021
Totktonada
force-pushed
the
Totktonada/gh-220-bucket-id-is-ignored
branch
from
124ba3e to
78e96e6
Compare
AnaNek
approved these changes
Oct 15, 2021
ligurio
suggested changes
Oct 18, 2021
ligurio
suggested changes
Oct 18, 2021
ligurio
suggested changes
Oct 18, 2021
I don't want to lean on box.stat() information here, because I don't control all iproto calls to storages: say, vshard rebalancer may perform them in background. Instead, I wrapped particular storage function I'm interested in. The goal is to be able to determine how much storages are involved into a select/pairs request. It is implemented as a helper for testing, but hopefully we'll implement some nice statistics as part of the module in a future (see #224). Part of #220
Totktonada
added a commit
that referenced
this pull request
Oct 18, 2021
The bug is simple: crud ignores provided bucket_id, when unable to determine it itself. For example, when no conditions are given or when given condition involves a secondary index, which is not entirely in the primary index. It leads to incorrect select/pairs result: tuples are collected from all replicasets, while should be collected from one replicaset pointed by bucket_id. Second, it involves all replicasets into the request processing (performs map-reduce) that may dramatically drop performance. One existing test case was changed: 'test_opts_not_damaged' in ipairs_test.lua. The crud.pairs() request in this test case was affected by the problem and incorrect result was expected. The idea of the fix is suggested by Michael Filonenko in PR #221. Nice suggestions were given by Sergey Bronnikov (see PR #222). Fixes #220
Totktonada
force-pushed
the
Totktonada/gh-220-bucket-id-is-ignored
branch
from
78e96e6 to
473c899
Compare
The bug is simple: crud ignores provided bucket_id, when unable to determine it itself. For example, when no conditions are given or when given condition involves a secondary index, which is not entirely in the primary index. It leads to incorrect select/pairs result: tuples are collected from all replicasets, while should be collected from one replicaset pointed by bucket_id. Second, it involves all replicasets into the request processing (performs map-reduce) that may dramatically drop performance. One existing test case was changed: 'test_opts_not_damaged' in ipairs_test.lua. The crud.pairs() request in this test case was affected by the problem and incorrect result was expected. The idea of the fix is suggested by Michael Filonenko in PR #221. Nice suggestions were given by Sergey Bronnikov (see PR #222). Fixes #220
Totktonada
force-pushed
the
Totktonada/gh-220-bucket-id-is-ignored
branch
from
473c899 to
7371998
Compare
This was referenced Nov 11, 2021
Open
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The bug is simple: crud ignores provided bucket_id, when unable to determine it itself. For example, when no conditions are given or when given condition involves a secondary index, which is not entirely in the primary index.
It leads to incorrect select/pairs result: tuples are collected from all replicasets, while should be collected from one storage pointed by bucket_id. Second, it involves all replicasets into the request processing (performs map-reduce) that may dramatically drop performance.
The idea of the fix is suggested by Michael Filonenko in PR #221.
As a side effect a set of simple testing utils for collecting select statistics from storages was added.
See details in the commit messages.
Fixes #220