Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ScalarBuffer::pad_nulls Zeros Out Padded Data Unnecessarily #1279

Closed
tustvold opened this issue Feb 7, 2022 · 0 comments · Fixed by #1280
Closed

ScalarBuffer::pad_nulls Zeros Out Padded Data Unnecessarily #1279

tustvold opened this issue Feb 7, 2022 · 0 comments · Fixed by #1280
Labels
enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate performance

Comments

@tustvold
Copy link
Contributor

tustvold commented Feb 7, 2022

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

ScalarBuffer::pad_nulls iterates backwards through the values, moving them into their final positions. It does this by swapping them with the zero-initialized value in the destination position.

Describe the solution you'd like

As far as I understand the specification, primitive arrays can have arbitrary values at null positions. Therefore it makes sense, at least to me, to simply copy the value to its destination and not zero out the source position.

Describe alternatives you've considered

The current logic is not incorrect and could remain

@tustvold tustvold added the enhancement Any new improvement worthy of a entry in the changelog label Feb 7, 2022
@alamb alamb changed the title ScalarBuffer::pad_nulls Zeros Out Padded Data ScalarBuffer::pad_nulls Zeros Out Padded Data Unnecessarily Feb 8, 2022
@alamb alamb added arrow Changes to the arrow crate performance parquet Changes to the parquet crate and removed arrow Changes to the arrow crate labels Feb 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Any new improvement worthy of a entry in the changelog parquet Changes to the parquet crate performance
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants