Skip to content

perf(table): group deletion vector reads by Puffin file - #1764

Closed
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/group-dv-reads-by-puffin-file
Closed

perf(table): group deletion vector reads by Puffin file#1764
fallintoplace wants to merge 1 commit into
apache:mainfrom
fallintoplace:perf/group-dv-reads-by-puffin-file

Conversation

@fallintoplace

Copy link
Copy Markdown
Contributor

Summary

  • group deletion vector loads by physical Puffin path
  • open and decode each shared Puffin file footer once
  • keep a fast path for one deletion vector
  • keep concurrency across different Puffin files

Why

The DV writer stores one blob per data file in a shared Puffin file. The scanner was calling ReadDV once per blob, so 100 DVs in one Puffin opened and decoded the same file 100 times.

Benchmarks

BenchmarkReadAllDeletionVectorsSharedPuffin on Apple M1 Pro:

DVs main this PR speedup
1 49.98 us 29.92 us 1.67x
10 672.21 us 97.03 us 6.93x
100 19.35 ms 558.17 us 34.66x

At 100 DVs, allocations drop from 19.3 MB and 125,737 allocs to 278 KB and 2,793 allocs. The shared Puffin file is opened once instead of 100 times.

Tests

  • go test ./...
  • go test -race ./table/dv ./table -run "TestReadDV|TestReadDVs|TestReadAllDeletionVectors" -count=1
  • go vet ./...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant