Skip to content

perf(parquet): batch delta binary packed decoding - #1169

Draft
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-batch-delta-unpack
Draft

perf(parquet): batch delta binary packed decoding#1169
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/parquet-batch-delta-unpack

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Rationale for this change

DELTA_BINARY_PACKED decoding called BitReader.GetBatch once for every value in a miniblock. The miniblock is already the unit decoded by this code, so this missed the existing batch unpackers.

What changes are included in this PR?

  • decode each miniblock into reusable scratch with one GetBatch call
  • keep 33 to 64 bit values on the generic uint64 path instead of the 32 bit SIMD unpackers
  • read SIMD input directly into pooled scratch so batched calls do not add allocations
  • add int32 and int64 benchmarks with nonzero packed widths
  • add coverage for batched values from 33 through 64 bits

Are these changes tested?

Yes. I ran the full Parquet test tree with the parquet-testing fixtures, the internal utility and encoding tests with noasm, and a linux/amd64 cross-build.

Apple M1 Pro results for 65,536 values with GOMAXPROCS=1 and 10 samples:

Workload Time
int32 small deltas -58.86%
int32 alternating wide deltas -66.04%
int64 timestamp-like deltas -59.72%
int64 deltas wider than 32 bits -33.89%

Allocations stay at 2 allocs/op in all four benchmarks.

@fallintoplace
fallintoplace marked this pull request as draft August 11, 2026 16:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant