-
Notifications
You must be signed in to change notification settings - Fork 669
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
parquet: improve BOOLEAN writing logic and report error on encoding fail #443
Merged
Merged
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
db8af9b
improve BOOLEAN writing logic and report error on encoding fail
garyanaplan a4ec2d7
only manipulate the bit_writer for BOOLEAN data
garyanaplan eb11b2b
better isolation of changes
garyanaplan 06d9a33
add test for boolean writer
garyanaplan da8c665
fix capacity calculation error in bool encoding
garyanaplan File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -112,6 +112,7 @@ pub struct PlainEncoder<T: DataType> { | |
buffer: ByteBuffer, | ||
bit_writer: BitWriter, | ||
desc: ColumnDescPtr, | ||
bw_bytes_written: usize, | ||
_phantom: PhantomData<T>, | ||
} | ||
|
||
|
@@ -124,6 +125,7 @@ impl<T: DataType> PlainEncoder<T> { | |
buffer: byte_buffer, | ||
bit_writer: BitWriter::new(256), | ||
desc, | ||
bw_bytes_written: 0, | ||
_phantom: PhantomData, | ||
} | ||
} | ||
|
@@ -153,7 +155,11 @@ impl<T: DataType> Encoder<T> for PlainEncoder<T> { | |
|
||
#[inline] | ||
fn put(&mut self, values: &[T::T]) -> Result<()> { | ||
if self.bw_bytes_written + values.len() >= self.bit_writer.capacity() { | ||
self.bit_writer.extend(256); | ||
} | ||
T::T::encode(values, &mut self.buffer, &mut self.bit_writer)?; | ||
self.bw_bytes_written += values.len(); | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm going to add a comment myself! :-) I just realised that I only want to do this checking if the encoding is for a Boolean, otherwise it's wasted work/memory. I'll think of the best way to achieve that. |
||
Ok(()) | ||
} | ||
} | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since
put_value
returns false if there isn't enough space, you might be able to avoid errors with something like:Rather than returning an error
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yea, we can either do this or make sure up front that there's enough capacity to write. One minor concern is putting the if branch inside the for loop might hurt the performance.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I found it hard to think of a good way to test this with the fix in place.
I preferred the "don't auto expand memory at the point of failure" approach because I'm fairly conservative and didn't want to make a change that was too wide in impact without a better understanding of the code. i.e.: my fix specifically targeted the error I reported and made it possible to detect in other locations.
I think a better fix would be to (somehow) pre-size the vector or avoid having to size a vector for all the bytes that could be written, but that would be a much bigger scope to the fix.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
leaving the code as is seems fine to me