Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(state_table): pre-compute column id mapping #3201

Merged
merged 3 commits into from
Jun 14, 2022

Conversation

skyzh
Copy link
Contributor

@skyzh skyzh commented Jun 14, 2022

Signed-off-by: Alex Chi iskyzh@gmail.com

What's changed and what's your intention?

As title

Checklist

  • I have written necessary docs and comments
  • I have added necessary unit tests and integration tests
  • All checks passed in ./risedev check (or alias, ./risedev c)

Refer to a related PR or issue link (optional)

Signed-off-by: Alex Chi <iskyzh@gmail.com>
@skyzh skyzh requested review from BugenZhao and wcy-fdu June 14, 2022 06:40
@skyzh skyzh marked this pull request as ready for review June 14, 2022 06:40
@codecov
Copy link

codecov bot commented Jun 14, 2022

Codecov Report

Merging #3201 (d0a8a69) into main (1b5fe56) will decrease coverage by 0.00%.
The diff coverage is 90.97%.

@@            Coverage Diff             @@
##             main    #3201      +/-   ##
==========================================
- Coverage   73.56%   73.55%   -0.01%     
==========================================
  Files         745      745              
  Lines      102223   102242      +19     
==========================================
+ Hits        75199    75208       +9     
- Misses      27024    27034      +10     
Flag Coverage Δ
rust 73.55% <90.97%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/stream/src/executor/lookup/sides.rs 29.16% <ø> (ø)
src/stream/src/executor/managed_state/join/mod.rs 77.77% <ø> (+4.70%) ⬆️
src/storage/src/cell_based_row_deserializer.rs 95.38% <87.17%> (-2.80%) ⬇️
src/storage/src/table/cell_based_table.rs 71.30% <88.57%> (+1.52%) ⬆️
src/storage/src/table/state_table.rs 83.15% <88.57%> (+1.39%) ⬆️
src/common/src/util/ordered/serde.rs 91.63% <100.00%> (+0.56%) ⬆️
src/stream/src/executor/lookup/impl_.rs 95.58% <100.00%> (ø)
src/stream/src/executor/managed_state/top_n/mod.rs 100.00% <100.00%> (ø)
...ecutor/managed_state/top_n/top_n_bottom_n_state.rs 77.63% <100.00%> (ø)
...am/src/executor/managed_state/top_n/top_n_state.rs 93.02% <100.00%> (ø)
... and 6 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

Signed-off-by: Alex Chi <iskyzh@gmail.com>
@skyzh skyzh enabled auto-merge (squash) June 14, 2022 07:24
@@ -88,7 +116,7 @@ impl CellBasedRowDeserializer {
pub fn take(&mut self) -> Option<(Vec<u8>, Row)> {
let cur_pk_bytes = self.pk_bytes.take();
cur_pk_bytes.map(|bytes| {
let ret = self.data.iter_mut().map(Option::take).collect::<Vec<_>>();
let ret = std::mem::replace(&mut self.data, vec![None; self.columns.len()]);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this lead to extra allocation?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe no. Previously, .collect_vec() will lead to one allocation. Now only vec![None; self.columns.len()] has one allocation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sense.

Comment on lines -196 to +210
#[allow(dead_code)]
pub async fn get<'a, 'b>(&'a mut self, key: &'b K) -> Option<&'a HashValueType<S>> {
let state = self.inner.get(key);
// TODO: we should probably implement a entry function for `LruCache`
match state {
Some(_) => self.inner.get(key),
None => {
let remote_state = self.fetch_cached_state(key).await.unwrap();
remote_state.map(|rv| {
self.inner.put(key.clone(), rv);
self.inner.get(key).unwrap()
})
}
}
}
// #[allow(dead_code)]
// pub async fn get<'a, 'b>(&'a mut self, key: &'b K) -> Option<&'a HashValueType<S>> {
// let state = self.inner.get(key);
// // TODO: we should probably implement a entry function for `LruCache`
// match state {
// Some(_) => self.inner.get(key),
// None => {
// let remote_state = self.fetch_cached_state(key).await.unwrap();
// remote_state.map(|rv| {
// self.inner.put(key.clone(), rv);
// self.inner.get(key).unwrap()
// })
// }
// }
// }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revert this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not used. And it triggers Rust compiler's false negative very often. So I commented it out. Anyone needs this can simply revert the comment.

@BugenZhao BugenZhao disabled auto-merge June 14, 2022 07:26
@BugenZhao BugenZhao merged commit 941bde6 into main Jun 14, 2022
@BugenZhao BugenZhao deleted the skyzh/avoid-full-clone-state-table branch June 14, 2022 07:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants