Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Decimal and List ArrayData Validation (#1813) (#1814) #1816

Merged
merged 2 commits into from
Jun 7, 2022

Conversation

tustvold
Copy link
Contributor

@tustvold tustvold commented Jun 7, 2022

Which issue does this PR close?

Closes #1813
Closes #1814

Rationale for this change

See tickets

What changes are included in this PR?

See tickets

Are there any user-facing changes?

No

@github-actions github-actions bot added the arrow Changes to the arrow crate label Jun 7, 2022
&'a self,
buffer: &'a Buffer,
) -> Result<&'a [T]> {
fn typed_offsets<T: ArrowNativeType + num::Num>(&self) -> Result<&[T]> {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found the way this accepted a buffer, but then used information from self to interpret that data, deeply confusing. I therefore changed this

@tustvold tustvold force-pushed the array-data-validation-fixes branch 3 times, most recently from c201841 to 76bc699 Compare June 7, 2022 18:18
Fix offset validation for sliced children of list arrays (apache#1814)
@tustvold tustvold force-pushed the array-data-validation-fixes branch from 76bc699 to bdaba34 Compare June 7, 2022 18:20
@tustvold tustvold requested a review from viirya June 7, 2022 18:20
@@ -819,12 +826,12 @@ impl ArrayData {
match &self.data_type {
DataType::List(field) | DataType::Map(field, _) => {
let values_data = self.get_single_valid_child_data(field.data_type())?;
self.validate_offsets::<i32>(&self.buffers[0], values_data.len)?;
self.validate_offsets::<i32>(values_data.len)?;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the fix for #1814

};
let value = i128::from_le_bytes(raw_val.try_into().unwrap());
validate_decimal_precision(value, *p)?;
let values_buffer: &[i128] = self.typed_buffer(0, self.len)?;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the fix for #1813

@codecov-commenter
Copy link

codecov-commenter commented Jun 7, 2022

Codecov Report

Merging #1816 (eb632ea) into master (96a4d6c) will increase coverage by 0.01%.
The diff coverage is 88.46%.

@@            Coverage Diff             @@
##           master    #1816      +/-   ##
==========================================
+ Coverage   83.42%   83.44%   +0.01%     
==========================================
  Files         199      199              
  Lines       56632    56639       +7     
==========================================
+ Hits        47244    47260      +16     
+ Misses       9388     9379       -9     
Impacted Files Coverage Δ
arrow/src/array/data.rs 84.16% <88.46%> (+0.18%) ⬆️
parquet/src/arrow/async_reader.rs 0.00% <0.00%> (ø)
parquet_derive/src/parquet_field.rs 66.21% <0.00%> (+0.22%) ⬆️
parquet/src/file/footer.rs 98.11% <0.00%> (+2.74%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 96a4d6c...eb632ea. Read the comment docs.


if (buffer.len() / std::mem::size_of::<T>()) < required_offsets {
if buffer.len() < required_len {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, shouldn't it be

let required_len = len * std::mem::size_of::<T>();
if buffer.len() < required_len {
  ...
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No? You need to take the offset into account, otherwise you will potentially panic a few lines down?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, looks correct. ArrayData.offset is offset into buffer, different than buffer's offset.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it's pretty confusing. FWIW #1799 may eventually allow us to get rid of this source of bugs

arrow/src/array/data.rs Outdated Show resolved Hide resolved
Copy link
Member

@viirya viirya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @tustvold. The fix looks good to me. The refactoring also makes the code more clear.

Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com>
@tustvold tustvold merged commit 9c40a87 into apache:master Jun 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate
Projects
None yet
3 participants