Skip to content

perf(arrow/array): batch dictionary empty appends - #1223

Merged
zeroshade merged 2 commits into
apache:mainfrom
fallintoplace:perf/arrow-dictionary-empty-append
Aug 28, 2026
Merged

perf(arrow/array): batch dictionary empty appends#1223
zeroshade merged 2 commits into
apache:mainfrom
fallintoplace:perf/arrow-dictionary-empty-append

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What does this change?

  • AppendEmptyValues used to build an n-element values array and dictionary-encode every empty value.
  • Resolve the empty value once, reserve the index builder once, and append the same dictionary index for the whole batch.
  • Keep the existing null dictionary path unchanged.

Benchmark

Command:

go test ./arrow/array -run '^$' -bench '^BenchmarkDictionaryBuilderAppendEmptyValues$' -benchmem -benchtime=200ms -count=3

Apple M1 Pro, Go 1.26.3. The benchmark reuses the dictionary builder and resets the output length between iterations.

Type Count Before ns/op After ns/op Before B/op After B/op
int32 64 1,684 653 1,168 784
int32 4,096 81,910 14,193 42,576 784
int32 65,536 1,305,061 221,358 551,382 784
string 64 2,573 868 1,568 1,056
string 4,096 121,191 14,577 42,912 1,056
string 65,536 1,916,937 230,499 551,714 1,056

Tests

  • go test ./arrow/array -count=1
  • go test ./arrow/... ./internal/...

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The batching change preserves behavior in the tested cases and reproduces the expected performance and allocation improvements. Array, repeated dictionary, race, vet, and LSP checks pass. One lint failure must be fixed before merge.

Comment thread arrow/array/dictionary.go Outdated
panic(err)
}

b.idxBuilder.Builder.Reserve(n)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The repository’s lint hook rewrites this to b.idxBuilder.Reserve(n), causing the required Lint check to fail because the hook modifies the file. Please use the promoted method and rerun lint.

@zeroshade
zeroshade merged commit a0dcf38 into apache:main Aug 28, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants