Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

arrow::compute::concat panics for dense union arrays with non-trivial type IDs #4578

Closed
kawadakk opened this issue Jul 28, 2023 · 2 comments · Fixed by #4598
Closed

arrow::compute::concat panics for dense union arrays with non-trivial type IDs #4578

kawadakk opened this issue Jul 28, 2023 · 2 comments · Fixed by #4598
Labels
arrow Changes to the arrow crate bug

Comments

@kawadakk
Copy link
Contributor

Describe the bug
arrow::compute::concat panics when called with dense union arrays with type IDs that don't start at zero.

thread 'proptest::tests::qc_row' panicked at 'index out of bounds: the len is 1 but the index is 35', ...\.cargo\registry\src\index.crates.io-6f17d22bba15001f\arrow-data-43.0.0\src\transform\union.rs:53:39
stack backtrace:
 ...
   3: arrow_data::transform::union::build_extend_dense::{{closure}}::{{closure}}
   4: core::iter::traits::iterator::Iterator::for_each::call::{{closure}}
             at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643\library\core\src\iter\traits/iterator.rs:853:29
   5: core::iter::traits::iterator::Iterator::fold
             at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643\library\core\src\iter\traits/iterator.rs:2481:21
   6: core::iter::traits::iterator::Iterator::for_each
             at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643\library\core\src\iter\traits/iterator.rs:856:9
   7: arrow_data::transform::union::build_extend_dense::{{closure}}
             at ...\.cargo\registry\src\index.crates.io-6f17d22bba15001f\arrow-data-43.0.0\src\transform\union.rs:50:13
   8: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643\library\alloc\src/boxed.rs:2021:9
   9: arrow_data::transform::MutableArrayData::extend
             at ...\.cargo\registry\src\index.crates.io-6f17d22bba15001f\arrow-data-43.0.0\src\transform\mod.rs:631:9
  10: arrow_select::concat::concat
             at ...\.cargo\registry\src\index.crates.io-6f17d22bba15001f\arrow-select-43.0.0\src\concat.rs:89:9

To Reproduce

Call arrow::compute::concat and pass the following arrays:

[
    UnionArray(Dense)
    [
        -- type id buffer:
        ScalarBuffer([35])
        -- offsets buffer:
        ScalarBuffer([0])
        -- child 35: "" (Null)
        NullArray(1)
    ],
    UnionArray(Dense)
    [
        -- type id buffer:
        ScalarBuffer([35])
        -- offsets buffer:
        ScalarBuffer([0])
        -- child 35: "" (Null)
        NullArray(1)
    ],
]

of the following data type:

Union(
    [
        (
            35,
            Field {
                name: "",
                data_type: Null,
                nullable: false,
                dict_id: 0,
                dict_is_ordered: false,
                metadata: {},
            },
        ),
    ],
    Dense,
)

Expected behavior

Producing the following array:

UnionArray(Dense)
[
    -- type id buffer:
    ScalarBuffer([35, 35])
    -- offsets buffer:
    ScalarBuffer([0, 1])
    -- child 35: "" (Null)
    NullArray(2)
]

Additional context

@kawadakk kawadakk added the bug label Jul 28, 2023
@tustvold
Copy link
Contributor

MutableArrayData likely needs to be updated to properly understand UnionArray with non-trivial type ids

@tustvold
Copy link
Contributor

label_issue.py automatically added labels {'arrow'} from #4598

@tustvold tustvold added the arrow Changes to the arrow crate label Aug 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants