-
Notifications
You must be signed in to change notification settings - Fork 696
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Deduplicate strings/binarys when building view types #6005
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @XiangpengHao -- I think this PR looks great.
I think we need to extend the tests to cover 2 more cases (I left a comment) but otherwise it is ready to go
I left some suggestions that might improve the code, but I think they are optional
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @XiangpengHao 🙏
|
||
let mut builder = StringViewBuilder::new() | ||
.with_deduplicate_strings() | ||
.with_block_size(value_1.len() as u32 * 2); // so that we will have multiple buffers |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
💯
Which issue does this PR close?
Closes #5910.
Rationale for this change
As discussed in #5910, when the array of strings have low number of unique values, it makes sense to deduplicate them while building it, especially when invoking
gc
This pr implements an optional mechanism to deduplicate the strings. This is turned off by default, as it may slow down building array if the array is mostly unique.
However, the
gc
will by default enable it, so that gc generates an optimal layout.What changes are included in this PR?
Are there any user-facing changes?