Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

filter produces invalid sparse UnionArrays #1547

Closed
Tracked by #1546
alamb opened this issue Apr 12, 2022 · 0 comments · Fixed by #1567
Closed
Tracked by #1546

filter produces invalid sparse UnionArrays #1547

alamb opened this issue Apr 12, 2022 · 0 comments · Fixed by #1567
Labels
arrow Changes to the arrow crate bug

Comments

@alamb
Copy link
Contributor

alamb commented Apr 12, 2022

Describe the bug
The force_validate check added in #1546 revealed that the UnionArrays produced from the filter kernel fails validation.

It is not clear if the validation is incorrect or the filter kernel is incorrect

To Reproduce
Uncomment the

    #[cfg(not(feature = "force_validate"))]

And then run the test like:

cargo test --features=force_validate -p arrow

lines for the following tests and the will fail:

---- compute::kernels::filter::tests::test_filter_union_array_sparse stdout ----
thread 'compute::kernels::filter::tests::test_filter_union_array_sparse' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidArgumentError("Expected 1 buffers in array of type Union([Field { name: \"A\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: \"B\", data_type: Float64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }], Sparse), got 2")', arrow/src/array/data.rs:301:34

---- compute::kernels::filter::tests::test_filter_union_array_sparse_with_nulls stdout ----
thread 'compute::kernels::filter::tests::test_filter_union_array_sparse_with_nulls' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidArgumentError("Expected 1 buffers in array of type Union([Field { name: \"A\", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }, Field { name: \"B\", data_type: Float64, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: None }], Sparse), got 2")', arrow/src/array/data.rs:301:34

Expected behavior
The tests should pass when force_validate is on

Additional context
Add any other context about the problem here.

@alamb alamb added bug arrow Changes to the arrow crate labels Apr 12, 2022
alamb added a commit to alamb/arrow-rs that referenced this issue Apr 12, 2022
alamb added a commit that referenced this issue Apr 14, 2022
* Add force_validate feature

* Disable some redundant checks

* Add issue link

* Add test with force_validate feature flag

* fix up message

* disable due to #1547

* disable ipc test failure

* fix clippy

* Fix doctest to pass with force_validate enabled
@alamb alamb changed the title Output of filter on UnionArrays fails validation filter produces invalid sparse UnionArray fails validation Apr 15, 2022
@alamb alamb changed the title filter produces invalid sparse UnionArray fails validation filter produces invalid sparse UnionArrays Apr 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant