Describe the bug
The upper bound of the output buffer size for binary view is estimated using its memory buffer size.
|
encoding.decode_array::<_, i32>(&array, array.get_buffer_memory_size()) |
This is incorrect as it can underestimate required memory when the BinaryView contains significant duplication and views point to overlapping regions in the buffers.
base64 decode will fail with Output slice too small due to this bad estimate.
To Reproduce
The following test case could trigger this bug
datafusion/datafusion/functions/src/encoding/inner.rs
#[test]
fn test_estimate_view_size() {
let mut builder = BinaryViewBuilder::new().with_deduplicate_strings();
for _ in 0..1000 {
builder.append_value([65u8; 64]);
}
let arr = ArrayBuilder::finish(&mut builder);
println!("{}", arr.get_buffer_memory_size());
Encoding::Base64Padded
.decode_array::<_, i32>(&arr.as_binary_view(), arr.get_buffer_memory_size())
.unwrap();
}
Expected behavior
No response
Additional context
This bug might have a much larger blast radius.
Describe the bug
The upper bound of the output buffer size for binary view is estimated using its memory buffer size.
datafusion/datafusion/functions/src/encoding/inner.rs
Line 297 in 050046a
This is incorrect as it can underestimate required memory when the BinaryView contains significant duplication and views point to overlapping regions in the buffers.
base64 decode will fail with
Output slice too smalldue to this bad estimate.To Reproduce
The following test case could trigger this bug
datafusion/datafusion/functions/src/encoding/inner.rsExpected behavior
No response
Additional context
This bug might have a much larger blast radius.