Skip to content

Incorrect size estimate for base64 decode buffer for BinaryView #23763

Description

@liningpan

Describe the bug

The upper bound of the output buffer size for binary view is estimated using its memory buffer size.

encoding.decode_array::<_, i32>(&array, array.get_buffer_memory_size())

This is incorrect as it can underestimate required memory when the BinaryView contains significant duplication and views point to overlapping regions in the buffers.

base64 decode will fail with Output slice too small due to this bad estimate.

To Reproduce

The following test case could trigger this bug

datafusion/datafusion/functions/src/encoding/inner.rs

    #[test]
    fn test_estimate_view_size() {
        let mut builder = BinaryViewBuilder::new().with_deduplicate_strings();
        for _ in 0..1000 {
            builder.append_value([65u8; 64]);
        }
        let arr = ArrayBuilder::finish(&mut builder);
        println!("{}", arr.get_buffer_memory_size());
        Encoding::Base64Padded
            .decode_array::<_, i32>(&arr.as_binary_view(), arr.get_buffer_memory_size())
            .unwrap();
    }

Expected behavior

No response

Additional context

This bug might have a much larger blast radius.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions