Skip to content

perf(arrow/ipc): slice already-zero-based offset buffers - #1191

Draft
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/ipc-zero-based-offset-slices
Draft

perf(arrow/ipc): slice already-zero-based offset buffers#1191
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/ipc-zero-based-offset-slices

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Rationale for this change

IPC currently allocates a new offsets buffer whenever a variable-length array has unused trailing offsets. For a prefix slice, the first logical offset is already zero, so the loop only copies every offset and subtracts zero.

Arrow C++ handles this case by slicing the existing offsets buffer.

What changes are included in this PR?

  • Read the first logical offset instead of treating every sliced array as needing rebasing.
  • Rebase only when that offset is non-zero.
  • Use memory.SliceBuffer when the required offsets are already zero-based.
  • Add coverage for int32 and int64 offsets, middle slices, and slices with a non-zero array offset whose first logical value offset is zero.

The benchmark uses a 524,288-value slice from a 1,048,576-value backing array. Medians from 6 runs on an Apple M1 Pro were:

prefix slice main this PR
int32 time/op 238 us 40.7 ns
int32 B/op 2,106,000 80
int64 time/op 283 us 40.7 ns
int64 B/op 4,203,153 80
allocs/op 8 1

Unsliced buffers remain allocation-free. Middle slices that need rebasing are unchanged.

Are these changes tested?

  • go test ./arrow/ipc -count=1
  • go test ./arrow/... -count=1
  • go test ./arrow/ipc -run "^$" -bench "^BenchmarkGetZeroBasedValueOffsets$" -benchmem -count=6

Are there any user-facing changes?

No. The IPC output and public API are unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant