feat(ipc): Remove zero-initialization in IPC reader

[pchintar]

Which issue does this PR close?

• Closes Remove redundant zero-initialization in Arrow IPC reader hot paths #9777 .

Rationale for this change

In arrow-ipc/src/reader.rs, buffers are currently allocated with MutableBuffer::from_len_zeroed(len) and then passed to read_exact, which fully overwrites the entire buffer contents.

This results in an unnecessary full memory pass for all data bytes:

• first zero-initialization
• then complete overwrite by the incoming data

Since read_exact guarantees that the provided slice is fully written on success, the initial zeroing step is redundant and can be safely avoided by allocating with capacity and setting the length before the read.

What changes are included in this PR?

Affected locations

• read_block (file reader path)
• MessageReader::maybe_next (stream reader path, message body buffer)

Before/Current Approach

let mut buf = MutableBuffer::from_len_zeroed(len);
reader.read_exact(&mut buf)?;

After Approach

let mut buf = MutableBuffer::with_capacity(len);
unsafe { buf.set_len(len) };
reader.read_exact(buf.as_slice_mut())?;

Rationale

read_exact fully overwrites the provided slice on success, so the initial zero-fill is redundant.
This change removes one full memory write pass per read without altering behavior.

Safety

The unsafe set_len usage is sound because:

• the buffer is not accessed between set_len and read_exact
• read_exact either fully initializes the buffer or returns an error
• on error, the buffer is not used and is dropped immediately
• no partially initialized data is exposed to safe Rust

Crash or interruption scenarios (panic, early return, process termination) do not introduce unsoundness, as the buffer is never observed unless read_exact completes successfully by returning Ok(()).

Are these changes tested?

Yes the changes were tested successfully by running:

cargo test -p arrow-ipc
cargo fmt --all
cargo clippy --all-targets --all-features -- -D warnings

Are there any user-facing changes?

No, there are no changes made to any public api code.

[jhorstmann]

Strictly speaking, this is not sound for arbitrary Read implementations. Unfortunately, nothing would prevent a Read impl from reading the uninitialized bytes in buf, causing undefined behavior. For a known implementation like File this pattern could be fine though.

An alternative could be to combine take and read_to_end to read into a Vec, but note that this changes the alignment of the resulting Buffer:

    let mut buf = Vec::with_capacity(total_len);
    reader.take(total_len as u64).read_to_end(&mut buf)?;
    if buf.len() != total_len {
        return Err(...)
    }
    Ok(buf.into())

[pchintar]

Thanks for the reply @jhorstmann , so I dug into this a bit more and agree the set_len + read_exact approach isn’t suitable for generic Read.

While looking into alternatives, I came across the newer read_buf API using BorrowedBuf, which is designed specifically for reading into uninitialized memory safely. It seems like it would let us avoid the zero-fill while still preserving alignment via MutableBuffer.

Something like:

use std::io::{Read, BorrowedBuf};

let mut buf = MutableBuffer::with_capacity(total_len);

{
    let spare = buf.spare_capacity_mut();
    let mut borrowed = BorrowedBuf::from(spare);
    reader.read_buf_exact(borrowed)?;
}

unsafe { buf.set_len(total_len) };

[pchintar]

so it seems to be that the read_buf is a nightly api and so may not be compatible with stable rust, so I'm unsure about it.