Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I made a PR to change MutableBuffer to panic on length overflow instead of relying on unchecked arithmetic.
That is clearly better than undefined behavior, but it still leaves downstream users without a fallible API when they need to handle very large requested lengths safely.
In particular, callers of APIs such as MutableBuffer::reserve, MutableBuffer::extend_zeros, and higher-level BufferBuilder methods like reserve, advance, and append_n_zeroed currently have to either:
- duplicate the overflow checks themselves before calling into Arrow, or
- accept that these paths may panic on invalid or unexpectedly large input sizes.
This came up in the review discussion on PR #9820 with @etseidl and @scovich :
#9820 (comment)
Describe the solution you'd like
Add fallible variants (aka return Result for relevant MutableBuffer APIs, so callers that need to code defensively can opt into Result-based handling instead of panic-based handling.
- add low-level APIs such as
MutableBuffer::try_reserve and MutableBuffer::try_extend_zeros
- optionally add matching
try_ variants on BufferBuilder, such as try_reserve, try_advance, and try_append_n_zeroed
The exact error type can be decided separately, but the goal would be to report size/length overflow as a normal error rather than panicking.
Describe alternatives you've considered
One alternative is to keep the current panic-only behavior and improve documentation. That would help users understand the behavior, but it would still force defensive callers to reimplement overflow checks outside Arrow.
I am personally not super convinced many of the call paths in this low level will use Result based handling, but I don't see how it would hurt.
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I made a PR to change
MutableBufferto panic on length overflow instead of relying on unchecked arithmetic.MutableBuffer::extend_zeros#9820That is clearly better than undefined behavior, but it still leaves downstream users without a fallible API when they need to handle very large requested lengths safely.
In particular, callers of APIs such as
MutableBuffer::reserve,MutableBuffer::extend_zeros, and higher-levelBufferBuildermethods likereserve,advance, andappend_n_zeroedcurrently have to either:This came up in the review discussion on PR #9820 with @etseidl and @scovich :
#9820 (comment)
Describe the solution you'd like
Add fallible variants (aka return
Resultfor relevant MutableBuffer APIs, so callers that need to code defensively can opt intoResult-based handling instead of panic-based handling.MutableBuffer::try_reserveandMutableBuffer::try_extend_zerostry_variants onBufferBuilder, such astry_reserve,try_advance, andtry_append_n_zeroedThe exact error type can be decided separately, but the goal would be to report size/length overflow as a normal error rather than panicking.
Describe alternatives you've considered
One alternative is to keep the current panic-only behavior and improve documentation. That would help users understand the behavior, but it would still force defensive callers to reimplement overflow checks outside Arrow.
I am personally not super convinced many of the call paths in this low level will use Result based handling, but I don't see how it would hurt.