Skip to content

perf(arrow/array): remove map validity slice - #1228

Merged
zeroshade merged 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-mapbuilder-validity
Aug 28, 2026
Merged

perf(arrow/array): remove map validity slice#1228
zeroshade merged 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-mapbuilder-validity

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • Replace the temporary all-true validity slice in MapBuilder.adjustStructBuilderLen.
  • Reuse the existing all-valid validity bitmap fast path.
  • Add a benchmark for NewMapArray after bulk key/item appends.

The benchmark stops the timer while setting up the map entries, so it focuses on the finalization path where the temporary slice was created.

Benchmark

Measured on Apple M1 Pro, macOS arm64, Go 1.26.3. Median of 3 runs. Each result is time/op, B/op, and allocs/op.

Bulk entries Before After
32 6.59 us, 4,392 B, 54 allocs 6.20 us, 4,392 B, 54 allocs
1,024 9.00 us, 5,608 B, 55 allocs 7.69 us, 4,584 B, 54 allocs
65,536 92.9 us, 88.2 KB, 55 allocs 31.0 us, 22.7 KB, 54 allocs

For 65,536 entries, this reduces bytes allocated by about 74% and removes one allocation per operation. The 1,024-entry case removes the temporary 1,024-byte slice as well.

Command:

go test -vet=off ./arrow/array -run '^$' -bench '^BenchmarkMapBuilderNewArrayAfterBulkChildren$' -benchmem -benchtime=500ms -count=3 -cpu=1

Tests

  • go test ./arrow/array -run 'TestMap' -count=1
  • go test ./... -count=1 with the pinned arrow-testing and parquet-testing submodules initialized

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the map validity fast path preserves output semantics and raw buffers across bitmap boundaries, builder reuse, mixed builder workflows, nested maps, invalid-state handling, and randomized cases. Array, race, vet, LSP, allocator, and performance checks pass. The allocation reduction is reproducible.

Nice improvement—thanks!

@zeroshade
zeroshade merged commit d72b313 into apache:main Aug 28, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants