Skip to content

[arrow-buffer] Add fallible APIs for overflow-prone buffer growth #9843

@alamb

Description

@alamb

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

I made a PR to change MutableBuffer to panic on length overflow instead of relying on unchecked arithmetic.

That is clearly better than undefined behavior, but it still leaves downstream users without a fallible API when they need to handle very large requested lengths safely.

In particular, callers of APIs such as MutableBuffer::reserve, MutableBuffer::extend_zeros, and higher-level BufferBuilder methods like reserve, advance, and append_n_zeroed currently have to either:

  • duplicate the overflow checks themselves before calling into Arrow, or
  • accept that these paths may panic on invalid or unexpectedly large input sizes.

This came up in the review discussion on PR #9820 with @etseidl and @scovich :
#9820 (comment)

Describe the solution you'd like

Add fallible variants (aka return Result for relevant MutableBuffer APIs, so callers that need to code defensively can opt into Result-based handling instead of panic-based handling.

  • add low-level APIs such as MutableBuffer::try_reserve and MutableBuffer::try_extend_zeros
  • optionally add matching try_ variants on BufferBuilder, such as try_reserve, try_advance, and try_append_n_zeroed

The exact error type can be decided separately, but the goal would be to report size/length overflow as a normal error rather than panicking.

Describe alternatives you've considered

One alternative is to keep the current panic-only behavior and improve documentation. That would help users understand the behavior, but it would still force defensive callers to reimplement overflow checks outside Arrow.

I am personally not super convinced many of the call paths in this low level will use Result based handling, but I don't see how it would hurt.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementAny new improvement worthy of a entry in the changelog

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions