-
Notifications
You must be signed in to change notification settings - Fork 112
feat: add new uninit_range API for PrimitiveBuilder
#2443
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
CodSpeed Performance ReportMerging #2443 will not alter performanceComparing Summary
|
| builder | ||
| .values | ||
| .extend_from_slice(transmute(&decoded[offset..])); | ||
| uninit.copy_from_init(out_idx, 1024 - offset, transmute(&decoded[offset..])); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The uninit isn't stateful, the whole point is that you have access to the whole range while the uninit is live. The downside is we need to keep track of the out_idx ourself
d085e22 to
3fc68d8
Compare
| &mut transmute_mut(builder.values.deref_mut()) | ||
| [my_offset_in_builder + i * 1024 - offset..][..1024], | ||
| ); | ||
| // SAFETY: &[T] and &[MaybeUninit<T>] have the same layout |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
and also we promise not to read the values right?
|
Documenting some discussion from Slack:
|
The PrimitiveBuilder currently exposes its internal components as
pub. This puts the onus of handling the state of the null buffer and values buffers on the client.The PrimitiveBuilder is built around append-only access to values and buffers. But for several array variants, random access is better suited. For example, unpacking FastLanes vectors, or scattering the patches of a SpareArray.
This PR removes
valuesandnullsfrom the public API for the PrimitiveBuilder, and replaces it with anuninit_range(len: usize)method. This returns a mutable handle that provides scoped access to the nulls and the values. For simplicity, it impls Deref/DerefMut to&[MaybeUninit<T>]to make random access of values easy and also gives direct access to nice std::slice methods likefillorcopy_from_slice.The returned handle has a
finish(self)method which should be called after value initialization, and will advance the internal buffer length to finish a chunk.As an example of usage:
You can also copy into the output from an already initialized set of values: