feat(batch): index scan & point get #3014

xxchan · 2022-06-06T20:30:58Z

What's changed and what's your intention?

As title. Mainly added ScanRange, which is built in LogicalScan::to_batch and used in RowSeqScan for different types of scan.

Checklist

I have written necessary docs and comments
I have added necessary unit tests and integration tests
All checks passed in ./risedev check (or alias, ./risedev c)

Refer to a related PR or issue link (optional)

part of #2429

github-actions

license-eye has totally checked 826 files.

Valid	Invalid	Ignored	Fixed
824	1	1	0

Click to see the invalid file list

src/frontend/src/utils/scan_range.rs

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

codecov · 2022-06-07T10:05:52Z

Codecov Report

Merging #3014 (a1ca321) into main (97cb014) will decrease coverage by 0.07%.
The diff coverage is 57.65%.

@@            Coverage Diff             @@
##             main    #3014      +/-   ##
==========================================
- Coverage   73.48%   73.40%   -0.08%     
==========================================
  Files         733      734       +1     
  Lines       99548    99973     +425     
==========================================
+ Hits        73157    73390     +233     
- Misses      26391    26583     +192

Flag	Coverage Δ
rust	`73.40% <57.65%> (-0.08%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/common/src/util/ordered/serde.rs	`91.06% <0.00%> (-1.98%)`	⬇️
src/frontend/src/utils/mod.rs	`66.66% <ø> (ø)`
src/batch/src/executor/row_seq_scan.rs	`23.27% <3.27%> (-20.28%)`	⬇️
src/storage/src/table/cell_based_table.rs	`72.97% <4.08%> (-17.73%)`	⬇️
src/frontend/src/expr/mod.rs	`86.40% <61.53%> (-5.81%)`	⬇️
src/frontend/src/utils/scan_range.rs	`80.00% <80.00%> (ø)`
src/frontend/src/utils/condition.rs	`93.59% <80.76%> (-5.28%)`	⬇️
...frontend/src/optimizer/plan_node/batch_seq_scan.rs	`95.76% <98.73%> (+3.30%)`	⬆️
src/common/src/catalog/physical_table.rs	`30.00% <100.00%> (+30.00%)`	⬆️
...c/frontend/src/optimizer/plan_node/logical_scan.rs	`100.00% <100.00%> (+0.37%)`	⬆️
... and 4 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

wcy-fdu · 2022-06-08T03:42:29Z

#3008 merged.

BugenZhao · 2022-06-08T04:43:00Z

Should we make PointGet a separate executor? 🤔

github-actions

license-eye has totally checked 833 files.

Valid	Invalid	Ignored	Fixed
831	1	1	0

Click to see the invalid file list

src/batch/src/executor/row_seq_scan.rs

xxchan · 2022-06-08T07:33:15Z

Should we make PointGet a separate executor? 🤔

Why bother? 🤔

proto/batch_plan.proto

liurenjie1024 · 2022-06-08T08:34:58Z

src/compute/tests/integration_tests.rs

@@ -436,7 +442,7 @@ async fn test_row_seq_scan() -> Result<()> {

    let executor = Box::new(RowSeqScanExecutor::new(
        table.schema().clone(),
-        table.iter_with_pk(u64::MAX, pk_descs).await.unwrap(),
+        ScanType::TableScan(table.iter_with_pk(u64::MAX, pk_descs).await.unwrap()),


Add tests for other ScanType?

e2e test is enough?

liurenjie1024 · 2022-06-08T08:46:00Z

src/frontend/src/utils/scan_range.rs

+    pub eq_conds: Vec<Literal>,
+    /// ((lb, inclusive), (ub, inclusive))
+    #[allow(clippy::type_complexity)]
+    pub range: Option<(Option<(Literal, bool)>, Option<(Literal, bool)>)>,


Is it possible to change it to T: Range<Literal>

xxchan · 2022-06-08T09:21:43Z

😕
thread 'tokio-runtime-worker' panicked at 'not implemented: excluded begin key is not supported', /Users/xxchan/Projects/risingwave/src/storage/src/hummock/iterator/forward_user.rs:221:28

@BugenZhao Any idea how to fix it?

xxchan · 2022-06-08T09:25:25Z

And Unbounded in storage not handled correctly 😕

xxchan · 2022-06-08T09:41:44Z

finished except excluded start key 😇

fuyufjh

LGTM!

fuyufjh · 2022-06-08T11:00:55Z

src/storage/src/table/cell_based_table.rs

+        pk_prefix: &Row,
+        next_col_bound: Bound<&Datum>,
+        is_start_bound: bool,
+    ) -> StorageResult<Bound<Vec<u8>>> {


So it seems unnecessary to return StorageResult<...>?

serialize_pk returns Result

I think serialize_pk should not return StorageResult as well...

Yes it returns Ok 😇 Let's get rid of it

skyzh · 2022-06-08T11:07:27Z

😕 thread 'tokio-runtime-worker' panicked at 'not implemented: excluded begin key is not supported', /Users/xxchan/Projects/risingwave/src/storage/src/hummock/iterator/forward_user.rs:221:28

@BugenZhao Any idea how to fix it?

Currently we only implemented some of the bound. If you need exclude bound, use next_key to convert it into an include bound.

included end key -> excluded next key

xxchan · 2022-06-08T12:00:57Z

If you need exclude bound, use next_key to convert it into an include bound.

Ok I did it. Btw I also found I need to convert included end key -> excluded next key 😇

cc @wcy-fdu Maybe #3008 also needs some fix? I'm not sure

xxchan · 2022-06-08T12:40:48Z

No idea why it fails 😇

Oh my local machine, ./risedev slt -p 4566 'e2e_test/batch/range_scan.slt' -d dev with ci-3cn-1fe also fails, but then I can see the correct result using psql.

xxchan · 2022-06-08T19:16:38Z

I feel confused about whether and where to add wait_epoch 😇

hengm3467 · 2022-07-11T09:28:54Z

@xxchan Does this PR contain user-facing changes? Thanks.

xxchan · 2022-07-11T09:38:59Z

@xxchan Does this PR contain user-facing changes? Thanks.

This is a perfermance optimization. No behavior changes.

github-actions bot added the type/feature label Jun 6, 2022

feat(batch): index scan & point get

75ac491

xxchan force-pushed the xxchan/point branch from 81fce49 to 75ac491 Compare June 6, 2022 20:36

xxchan added 2 commits June 7, 2022 10:25

implement split_to_scan_range

54f373e

use rust type for scan_range for pretty print

d07c1ae

github-actions bot reviewed Jun 7, 2022

View reviewed changes

xxchan and others added 2 commits June 7, 2022 11:50

Update src/frontend/src/utils/scan_range.rs

80f28f5

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

buf format

ab2fb96

xxchan added 5 commits June 7, 2022 13:58

add e2e test

2eb6bbb

Merge branch 'main' into xxchan/point

2c64039

fix: use order_column_ids instead of pk_ids

f54342f

add ordered_row_serializer

9579ef9

fmt

b974276

xxchan marked this pull request as ready for review June 7, 2022 16:07

xxchan force-pushed the xxchan/point branch from e109b31 to b974276 Compare June 8, 2022 07:20

xxchan and others added 2 commits June 8, 2022 09:21

Merge branch 'main' into xxchan/point

418aaed

range scan

7be297a

github-actions bot reviewed Jun 8, 2022

View reviewed changes

fmt

26940bf

risingwavelabs deleted a comment from github-actions bot Jun 8, 2022

liurenjie1024 reviewed Jun 8, 2022

View reviewed changes

fix

244ac44

clippy

27ed7b1

xxchan and others added 3 commits June 8, 2022 11:31

fix unbounded

e0024f2

Merge branch 'main' into xxchan/point

9618048

add more e2e test & clippy

cf1d6a5

xxchan requested review from fuyufjh, skyzh and st1page June 8, 2022 09:57

fuyufjh approved these changes Jun 8, 2022

View reviewed changes

excluded start key -> included next key,

b532bfe

included end key -> excluded next key

fix: wait epoch

3da89ea

xxchan and others added 6 commits June 8, 2022 21:17

Merge branch 'main' into xxchan/point

7afdd8c

Merge branch 'main' into xxchan/point

bf1bff5

fix wait_epoch???

db20181

style: tweak the representation of range

99a32e8

no pub

323606b

rerun ci

a1ca321

xxchan merged commit 0e313b1 into main Jun 9, 2022

xxchan deleted the xxchan/point branch June 9, 2022 01:02

xxchan mentioned this pull request Jun 9, 2022

RFC: use pk of the table to accelerate the query #2429

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(batch): index scan & point get #3014

feat(batch): index scan & point get #3014

xxchan commented Jun 6, 2022 •

edited

github-actions bot left a comment

codecov bot commented Jun 7, 2022 •

edited

wcy-fdu commented Jun 8, 2022

BugenZhao commented Jun 8, 2022

github-actions bot left a comment

xxchan commented Jun 8, 2022

liurenjie1024 Jun 8, 2022

xxchan Jun 8, 2022

liurenjie1024 Jun 8, 2022

xxchan Jun 9, 2022

xxchan commented Jun 8, 2022 •

edited

xxchan commented Jun 8, 2022

xxchan commented Jun 8, 2022

fuyufjh left a comment

fuyufjh Jun 8, 2022

xxchan Jun 8, 2022

fuyufjh Jun 9, 2022

xxchan Jun 9, 2022 •

edited

skyzh commented Jun 8, 2022

xxchan commented Jun 8, 2022 •

edited

xxchan commented Jun 8, 2022

xxchan commented Jun 8, 2022 •

edited

hengm3467 commented Jul 11, 2022

xxchan commented Jul 11, 2022

feat(batch): index scan & point get #3014

feat(batch): index scan & point get #3014

Conversation

xxchan commented Jun 6, 2022 • edited

What's changed and what's your intention?

Checklist

Refer to a related PR or issue link (optional)

github-actions bot left a comment

Choose a reason for hiding this comment

codecov bot commented Jun 7, 2022 • edited

Codecov Report

wcy-fdu commented Jun 8, 2022

BugenZhao commented Jun 8, 2022

github-actions bot left a comment

Choose a reason for hiding this comment

xxchan commented Jun 8, 2022

liurenjie1024 Jun 8, 2022

Choose a reason for hiding this comment

xxchan Jun 8, 2022

Choose a reason for hiding this comment

liurenjie1024 Jun 8, 2022

Choose a reason for hiding this comment

xxchan Jun 9, 2022

Choose a reason for hiding this comment

xxchan commented Jun 8, 2022 • edited

xxchan commented Jun 8, 2022

xxchan commented Jun 8, 2022

fuyufjh left a comment

Choose a reason for hiding this comment

fuyufjh Jun 8, 2022

Choose a reason for hiding this comment

xxchan Jun 8, 2022

Choose a reason for hiding this comment

fuyufjh Jun 9, 2022

Choose a reason for hiding this comment

xxchan Jun 9, 2022 • edited

Choose a reason for hiding this comment

skyzh commented Jun 8, 2022

xxchan commented Jun 8, 2022 • edited

xxchan commented Jun 8, 2022

xxchan commented Jun 8, 2022 • edited

hengm3467 commented Jul 11, 2022

xxchan commented Jul 11, 2022

xxchan commented Jun 6, 2022 •

edited

codecov bot commented Jun 7, 2022 •

edited

xxchan commented Jun 8, 2022 •

edited

xxchan Jun 9, 2022 •

edited

xxchan commented Jun 8, 2022 •

edited

xxchan commented Jun 8, 2022 •

edited