-
Notifications
You must be signed in to change notification settings - Fork 210
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(array): introduce DataChunkBuilder and split chunks in HashJoinExecutor #466
Conversation
…xecutor Signed-off-by: TennyZhuang <zty0826@gmail.com>
Signed-off-by: TennyZhuang <zty0826@gmail.com>
Signed-off-by: TennyZhuang <zty0826@gmail.com>
Signed-off-by: TennyZhuang <zty0826@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have two suggestions:
- Don't consume the builder when the size reaches the limit, so that users will no longer need to rebuild it by themselves.
impl DataChunkBuilder {
#[must_use = "should yield the DataChunk when size limit is reached"]
pub fn push_row(&mut self, row: impl IntoIterator<Item = DataValue>) -> Option<DataChunk>;
/// Generate a [`DataChunk`] with the remaining rows, leaving the builder empty.
#[must_use]
pub fn take(&mut self) -> Option<DataChunk>;
}
- Provide a helper macro to simplify the usage.
macro_rules! yield_if_some {
($maybe_chunk:expr) => {
if let Some(chunk) = $maybe_chunk {
yield chunk;
}
}
}
Then the usage should be something like:
let mut builder = DataChunkBuilder::new(...);
loop {
// ...
yield_if_some!(builder.push_row(values));
// if necessary
yield_if_some!(builder.take());
}
yield_if_some!(builder.take());
But it can force users to handle the finished DataChunk correctly by compiler error instead of just lint ( |
Unfortunately, macros does not work well with |
Signed-off-by: TennyZhuang <zty0826@gmail.com>
@wangrunji0408 PTAL |
Signed-off-by: TennyZhuang <zty0826@gmail.com>
Signed-off-by: TennyZhuang <zty0826@gmail.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM in general. Sorry for the late response ==
Signed-off-by: TennyZhuang <zty0826@gmail.com>
Signed-off-by: TennyZhuang <zty0826@gmail.com>
The PR introduces
DataChunkBuilder
, which is responsible for two things:capacity
.Signed-off-by: TennyZhuang zty0826@gmail.com