You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
bench/ loads every engine with a single serial COPY. We ship pgcolumnar.parallel_copy(target regclass, filename text, workers int DEFAULT NULL),
which is 7x faster at 16 workers with zero compression cost, and none of our published
load numbers reflect it. Our own benchmark understates our loader by a factor of seven.
Measured on 2M ClickBench hits rows (105 cols, 1.4 GB), pg18n, every arm asserted at
exactly 2,000,000 rows:
arm
load
vs serial
size
plain COPY
87.2s
1.00x
251 MB
parallel_copy 1w
87.7s
1.00x
251 MB
parallel_copy 2w
46.2s
1.89x
251 MB
parallel_copy 4w
25.2s
3.46x
251 MB
parallel_copy 8w
16.6s
5.25x
250 MB
parallel_copy 16w
12.4s
7.03x
251 MB
1 worker matches plain COPY (87.7s vs 87.2s), so the coordinator and 2PC add no
measurable overhead — the speedup is real, not a base-inflation artifact.
What to do
Add a parallel load arm to bench/, reported alongside the serial number rather than
replacing it. The two measure different capabilities and both are worth publishing:
serial COPY = the single-connection ingest path
parallel_copy N = the bulk-load path
Related: bench_run.sh already forces serial + index-off, and that decision deserves the
same treatment — see the note in this repo about it being a capability line, not the
only number worth showing.
Requirements
A parallel arm must fail loudly, never silently degrade to serial.parallel_copy
prepares one transaction per worker; with the stock max_prepared_transactions = 0 every parallel arm errors out instantly. My first harness printed those failures as 0.0s / 0.8s / 1.1s / 1.3s, which is indistinguishable from perfect scaling. The bench
must assert the loaded row count exactly and fail the run otherwise.
Preflight max_prepared_transactions >= workers and fail with a clear message
naming the required value, since raising it needs a postmaster restart.
Report the worker count in the results table. A load time without its degree is
uninterpretable.
Keep the fixture a stride sample, not a head prefix — a head of hits.tsv has 1
distinct date and 7 counters against 17 and 4,220 for a stride.
Out of scope
Changing pgcolumnar.fsst_min_gain_percent or any encoder default. That trade is being
priced separately in #445 and must not ride in on a harness change.
Problem
bench/loads every engine with a single serialCOPY. We shippgcolumnar.parallel_copy(target regclass, filename text, workers int DEFAULT NULL),which is 7x faster at 16 workers with zero compression cost, and none of our published
load numbers reflect it. Our own benchmark understates our loader by a factor of seven.
Measured on 2M ClickBench
hitsrows (105 cols, 1.4 GB), pg18n, every arm asserted atexactly 2,000,000 rows:
COPYparallel_copy1wparallel_copy2wparallel_copy4wparallel_copy8wparallel_copy16w1 worker matches plain
COPY(87.7s vs 87.2s), so the coordinator and 2PC add nomeasurable overhead — the speedup is real, not a base-inflation artifact.
What to do
Add a parallel load arm to
bench/, reported alongside the serial number rather thanreplacing it. The two measure different capabilities and both are worth publishing:
COPY= the single-connection ingest pathparallel_copy N= the bulk-load pathRelated:
bench_run.shalready forces serial + index-off, and that decision deserves thesame treatment — see the note in this repo about it being a capability line, not the
only number worth showing.
Requirements
parallel_copyprepares one transaction per worker; with the stock
max_prepared_transactions = 0every parallel arm errors out instantly. My first harness printed those failures as
0.0s / 0.8s / 1.1s / 1.3s, which is indistinguishable from perfect scaling. The benchmust assert the loaded row count exactly and fail the run otherwise.
max_prepared_transactions >= workersand fail with a clear messagenaming the required value, since raising it needs a postmaster restart.
uninterpretable.
columnar parallelises same-table
COPYtoo (5.91x at 16 sessions); a cross-engine tablemust match degree on both sides or it is marketing, not measurement. See Citus columnar loads the same data 3x faster than we do, which contradicts why #300 was closed #445.
headofhits.tsvhas 1distinct date and 7 counters against 17 and 4,220 for a stride.
Out of scope
Changing
pgcolumnar.fsst_min_gain_percentor any encoder default. That trade is beingpriced separately in #445 and must not ride in on a harness change.