Describe the bug
concat_elements_fixed_size_binary will panic with some invalid parameters
In theory, these invalid values can come from the FixedSizeBinary data type, which may originate in user controlled schema.
To Reproduce
Full reproducer:
issue1_repo.zip
With arrow-string 59.2.0:
[dependencies]
arrow-array = "59.2.0"
arrow-buffer = "59.2.0"
arrow-string = "59.2.0"
use arrow_array::FixedSizeBinaryArray;
use arrow_buffer::Buffer;
use arrow_string::concat_elements::concat_elements_fixed_size_binary;
fn main() {
let w = 0x7000_0000_i32;
let a = FixedSizeBinaryArray::try_new(w, Buffer::from(Vec::<u8>::new()), None).unwrap();
let b = FixedSizeBinaryArray::try_new(w, Buffer::from(Vec::<u8>::new()), None).unwrap();
let _ = concat_elements_fixed_size_binary(&a, &b);
}
This panics on both release and debug builds
andrewlamb@Andrews-MacBook-Pro-3:/tmp/issue1_repo$ cargo run
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.03s
Running `target/debug/issue1_repo`
calling concat_elements_fixed_size_binary with widths 1879048192 and 1879048192
thread 'main' (44570022) panicked at /Users/andrewlamb/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/arrow-array-59.2.0/src/builder/fixed_size_binary_builder.rs:60:9:
value length (-536870912) of the array must >= 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
andrewlamb@Andrews-MacBook-Pro-3:/tmp/issue1_repo$ cargo run --release
Blocking waiting for file lock on build directory
Finished `release` profile [optimized] target(s) in 5.88s
Running `target/release/issue1_repo`
calling concat_elements_fixed_size_binary with widths 1879048192 and 1879048192
thread 'main' (44570754) panicked at /Users/andrewlamb/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/arrow-array-59.2.0/src/builder/fixed_size_binary_builder.rs:60:9:
value length (-536870912) of the array must >= 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Expected behavior
concat_elements_fixed_size_binary should return an ArrowError when the combined width exceeds i32::MAX, rather than panicking, and should not attempt the allocation first.
Additional context
Describe the bug
concat_elements_fixed_size_binarywill panic with some invalid parametersIn theory, these invalid values can come from the
FixedSizeBinarydata type, which may originate in user controlled schema.To Reproduce
Full reproducer:
issue1_repo.zip
With arrow-string 59.2.0:
This panics on both release and debug builds
andrewlamb@Andrews-MacBook-Pro-3:/tmp/issue1_repo$ cargo run Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.03s Running `target/debug/issue1_repo` calling concat_elements_fixed_size_binary with widths 1879048192 and 1879048192 thread 'main' (44570022) panicked at /Users/andrewlamb/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/arrow-array-59.2.0/src/builder/fixed_size_binary_builder.rs:60:9: value length (-536870912) of the array must >= 0 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace andrewlamb@Andrews-MacBook-Pro-3:/tmp/issue1_repo$ cargo run --release Blocking waiting for file lock on build directory Finished `release` profile [optimized] target(s) in 5.88s Running `target/release/issue1_repo` calling concat_elements_fixed_size_binary with widths 1879048192 and 1879048192 thread 'main' (44570754) panicked at /Users/andrewlamb/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/arrow-array-59.2.0/src/builder/fixed_size_binary_builder.rs:60:9: value length (-536870912) of the array must >= 0 note: run with `RUST_BACKTRACE=1` environment variable to display a backtraceExpected behavior
concat_elements_fixed_size_binaryshould return anArrowErrorwhen the combined width exceedsi32::MAX, rather than panicking, and should not attempt the allocation first.Additional context
arrow-string); earlier versions likely also affected.