Skip to content

perf(table): size partition row indices adaptively - #1769

Closed
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/adaptive-partition-row-capacity
Closed

perf(table): size partition row indices adaptively#1769
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/adaptive-partition-row-capacity

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

What changed

  • Keep the existing maximum starting capacity of 128 row indexes.
  • Reduce the starting capacity as more partitions are discovered in a batch.
  • Use batch rows / discovered partitions as the estimate, with a minimum of one.
  • Keep grouping and row ordering unchanged.

Why

Every new partition currently reserves space for 128 row indexes. This works well for low-cardinality batches, but a batch with one partition per row reserves about 1 KiB for every single row.

The adaptive estimate keeps the current behavior for a few large partitions. It gradually moves toward one row of capacity for high-cardinality batches.

Benchmark

Apple M1 Pro, 32,768 rows, median of 5 runs:

go test ./table -run '^$' -bench '^BenchmarkPartitionRowCapacity$' -benchmem -benchtime=1s -count=5 -cpu=1
Distribution main this PR Change
1 partition 1,160,120 B/op 1,160,120 B/op unchanged
16 partitions 951,784 B/op 951,784 B/op unchanged
1,024 partitions 1,759,064 B/op 1,348,728 B/op -23%
1 partition per row 50,331,068 B/op 18,241,259 B/op -64%
90% in one partition 5,994,673 B/op 3,580,624 B/op -40%

Allocation counts are unchanged in every case. The unique-partition case was also about 14% faster in the repeated comparison.

Tests

  • go test ./...
  • go test -race ./table -run '^TestFanoutWriter$' -count=1
  • go vet ./...
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant