-
Notifications
You must be signed in to change notification settings - Fork 525
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
perf(executor): implement dedup pk decoding for BatchQueryExecutor #3060
Conversation
e734233
to
341e0e0
Compare
Codecov Report
@@ Coverage Diff @@
## main #3060 +/- ##
=======================================
Coverage 73.54% 73.54%
=======================================
Files 735 735
Lines 100309 100340 +31
=======================================
+ Hits 73769 73793 +24
- Misses 26540 26547 +7
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more |
7ebf9c2
to
af5505a
Compare
rebased |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM except the naming. By the way, I found that every time we need to access a table, our executors only get parts of its information. e.g., in this case, BatchExecutor only get column desc without ordered columns. We should have a single struct to describe a storage table in the future, maybe after the state table refactor.
related: #2212
cc @BowenXiao1999 @wcy-fdu
schema_ref_id: Default::default(), | ||
table_desc: Some(CellBasedTableDesc { | ||
table_id: self.logical.table_desc().table_id.into(), | ||
pk: self |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The field name should be order_key instead? pk is different from order key in the streaming engine.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ineed. Maybe Meta/FE should embed this info into Stream Plan, so executor can just read it and build state table.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated: 2206ad5 Sorry realized it should be order_key
rather than order_keys
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
StateTable will need:
|
af5505a
to
2206ad5
Compare
What's changed and what's your intention?
BatchQueryExecutor
.pk_descs
to pass by reference toDedupPkCellBasedTableIter
. This avoids aclone
ofpk_descs
each time we call theexecute_inner
method ofBatchQueryExecutor
.Checklist
./risedev check
(or alias,./risedev c
)Refer to a related PR or issue link (optional)
related #588