feat: enable consistent hash for materialize and table scan #3558
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
3 tasks
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
BugenZhao
commented
Jun 30, 2022
| @@ -95,13 +98,19 @@ impl ExecutorBuilder for ArrangeExecutorBuilder { | |||
| .map(|key| *key as usize) | |||
| .collect(); | |||
|
|
|||
| // FIXME: Lookup is now implemented without cell-based table API and relies on all vnodes | |||
There was a problem hiding this comment.
This will be resolved by #3418. cc @BowenXiao1999
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
xxchan
approved these changes
Jun 30, 2022
BowenXiao1999
approved these changes
Jun 30, 2022
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Codecov Report
@@ Coverage Diff @@
## main #3558 +/- ##
==========================================
- Coverage 74.39% 74.36% -0.03%
==========================================
Files 771 771
Lines 108714 108713 -1
==========================================
- Hits 80875 80844 -31
- Misses 27839 27869 +30
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more |
xx01cyx
reviewed
Jun 30, 2022
| @@ -184,7 +184,7 @@ mod tests { | |||
| #[test] | |||
| fn test_vec_decode_encode() { | |||
| let mut result = vec![]; | |||
| let value_meta = ValueMeta { vnode: 63492 }; | |||
| let value_meta = ValueMeta { vnode: 233 }; | |||
There was a problem hiding this comment.
Do we still have vnode in value meta now?
There was a problem hiding this comment.
Will remove soon. Check #3316
BugenZhao
added
the
mergify/can-merge
huangjw806
pushed a commit
that referenced
this pull request
Jul 5, 2022
* encode vnode in mview Signed-off-by: Bugen Zhao <i@bugenzhao.com> * use vnode in scan & batch query executor Signed-off-by: Bugen Zhao <i@bugenzhao.com> * support singleton materialize Signed-off-by: Bugen Zhao <i@bugenzhao.com> * ordered batch iterator Signed-off-by: Bugen Zhao <i@bugenzhao.com> * parallelized chain Signed-off-by: Bugen Zhao <i@bugenzhao.com> * disable time index & time window Signed-off-by: Bugen Zhao <i@bugenzhao.com> * introduce distribution struct Signed-off-by: Bugen Zhao <i@bugenzhao.com> * remove without distribution Signed-off-by: Bugen Zhao <i@bugenzhao.com> * disable delta join Signed-off-by: Bugen Zhao <i@bugenzhao.com> * recover delta join Signed-off-by: Bugen Zhao <i@bugenzhao.com> * update proto test Signed-off-by: Bugen Zhao <i@bugenzhao.com> * fix clippy Signed-off-by: Bugen Zhao <i@bugenzhao.com> * concurrent iter creation Signed-off-by: Bugen Zhao <i@bugenzhao.com> * use 256 vnodes Signed-off-by: Bugen Zhao <i@bugenzhao.com> * fix tests Signed-off-by: Bugen Zhao <i@bugenzhao.com> * fix tests Signed-off-by: Bugen Zhao <i@bugenzhao.com> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the Singularity Data, Inc. Contributor License Agreement.
What's changed and what's your intention?
This PR enables consistent hash for materialize executor, so that the batch query executor can really parallelize the scanning now. Besides, the
RowSeqScanis now ready to support parallel scan as well (#3251).There leave several issues to resolve:
CREATE INDEXare disabled due to bug: should use order key instead of primary key for cell based table desc ofINDEX#3572Checklist
./risedev check(or alias,./risedev c)Refer to a related PR or issue link (optional)