Skip to content

perf(arrow/array): bulk append fixed-width empty values - #1180

Open
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-fixed-width-empty-values
Open

perf(arrow/array): bulk append fixed-width empty values#1180
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-fixed-width-empty-values

Conversation

@fallintoplace

@fallintoplace fallintoplace commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

What changed

AppendEmptyValues used to call AppendEmptyValue in a loop. For larger batches, each value repeated the reserve, bitmap, and storage work.

This change makes fixed-width empty appends bulk operations:

  • reserve space once
  • zero the appended value range in one operation
  • set validity bits in bulk
  • clear boolean value bits separately from the validity bitmap
  • keep the existing single-value path for count 1

The numeric generated source and template are updated together. The covered builders are numeric, temporal, interval, float16, decimal, fixed-size binary, and boolean.

Correctness

  • Empty values stay valid, including when the builder reuses storage.
  • Reused numeric and boolean storage is explicitly tested.
  • bufferBuilder.Advance now zeroes skipped bytes as documented.
  • BooleanBuilder.Resize now preserves the requested logical length when truncating.
  • Zero and negative counts remain no-ops.

Benchmark

AppendEmptyValues(1024) on an Apple M1 Pro. Three runs were used for each result.

Builder Before After Allocs
Int32 5.15 us 1.50 us 11 -> 5
Int64 6.44 us 2.52 us 11 -> 5
Timestamp 6.50 us 3.22 us 12 -> 6
Boolean 4.04 us 0.30 us 7 -> 5
Fixed-size binary 10.1 us 5.69 us 16 -> 7

Small counts keep the single-value path, so the biggest improvement is for larger batches.

Tests

  • go test ./arrow/array
  • go vet ./arrow/array
  • Added focused correctness tests and representative benchmarks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant