Skip to content

perf(arrow/array): copy same-dictionary indices directly - #1225

Merged
zeroshade merged 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-dictionary-concat
Aug 28, 2026
Merged

perf(arrow/array): copy same-dictionary indices directly#1225
zeroshade merged 1 commit into
apache:mainfrom
fallintoplace:perf/arrow-dictionary-concat

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What does this change?

  • Same-dictionary concatenation used to create one temporary buffer wrapper per input chunk.
  • Use concatFixedWidthBuffers directly for the dictionary indices.
  • Keep the dictionary-unification path unchanged for different dictionaries.

Benchmark

Command:

go test ./arrow/array -run '^$' -bench '^BenchmarkConcatenateSameDictionary$' -benchmem -benchtime=200ms -count=3

Apple M1 Pro, Go 1.26.3. The benchmark concatenates 65,536 rows with the same four-value string dictionary, split across different chunk counts.

Chunks Before ns/op After ns/op Before B/op After B/op Allocs before/after
1 33,736 33,262 271,049 270,962 11 / 9
8 33,791 33,809 272,353 271,650 26 / 17
64 42,134 39,250 282,785 277,153 138 / 73
1,024 198,280 157,632 462,626 371,234 2,058 / 1,033
8,192 1,202,369 928,859 1,778,211 1,057,314 16,394 / 8,201

Tests

  • go test ./arrow/array -count=1
  • go test ./arrow/... ./internal/...

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change safely reuses the fixed-width concatenation path for same-dictionary indices and substantially reduces allocations for large chunk counts. I verified parity across index widths, slices, offsets, nulls, empty inputs, equal-but-distinct dictionaries, and dictionary unification. Array, repeated, race, vet, and LSP checks pass; CI is green.

@zeroshade
zeroshade merged commit 4c667c9 into apache:main Aug 28, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants