Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrect Offset Validation for Sliced List Array Children #1814

Closed
tustvold opened this issue Jun 7, 2022 · 0 comments · Fixed by #1816
Closed

Incorrect Offset Validation for Sliced List Array Children #1814

tustvold opened this issue Jun 7, 2022 · 0 comments · Fixed by #1816
Labels
arrow Changes to the arrow crate bug

Comments

@tustvold
Copy link
Contributor

tustvold commented Jun 7, 2022

Describe the bug

fn test_sliced_array_child() {
    let values = Int32Array::from_iter_values([1, 2, 3]);
    let values_sliced = values.slice(1, 2);
    let offsets = Buffer::from_iter([1_i32, 3_i32]);

    let list_field = Field::new("element", DataType::Int32, false);
    let data_type = DataType::List(Box::new(list_field));

    let data = unsafe {
        ArrayData::new_unchecked(
            data_type,
            1,
            None,
            None,
            0,
            vec![offsets],
            vec![values_sliced.data().clone()],
        )
    };

    let err = data.validate_dictionary_offset().unwrap_err();
    assert_eq!(err.to_string(), "Invalid argument error: Offset invariant failure: offset at position 1 out of bounds: 3 > 2");
}

Should pass, however, validate_dictionary_offset does not use the correct max offset during validation.

The good news is that the logic in validate() is correct, and will catch that the final offset is too large, and so this will be caught by full_validate() (i.e. this is not a soundness issue)

To Reproduce

See test

Expected behavior

The test should pass

Additional context
Add any other context about the problem here.

@tustvold tustvold added the bug label Jun 7, 2022
tustvold added a commit to tustvold/arrow-rs that referenced this issue Jun 7, 2022
Fix offset validation for sliced children of list arrays (apache#1814)
tustvold added a commit to tustvold/arrow-rs that referenced this issue Jun 7, 2022
Fix offset validation for sliced children of list arrays (apache#1814)
tustvold added a commit to tustvold/arrow-rs that referenced this issue Jun 7, 2022
Fix offset validation for sliced children of list arrays (apache#1814)
tustvold added a commit to tustvold/arrow-rs that referenced this issue Jun 7, 2022
Fix offset validation for sliced children of list arrays (apache#1814)
tustvold added a commit to tustvold/arrow-rs that referenced this issue Jun 7, 2022
Fix offset validation for sliced children of list arrays (apache#1814)
tustvold added a commit that referenced this issue Jun 7, 2022
* Fix DecimalArray validation (#1813)

Fix offset validation for sliced children of list arrays (#1814)

* Update arrow/src/array/data.rs

Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com>

Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com>
@alamb alamb added the arrow Changes to the arrow crate label Jun 9, 2022
@alamb alamb changed the title Incorrect Offset Validation for Sliced List Array Children Fix Incorrect Offset Validation for Sliced List Array Children Jun 9, 2022
@alamb alamb changed the title Fix Incorrect Offset Validation for Sliced List Array Children Incorrect Offset Validation for Sliced List Array Children Jun 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants