docs: the 2026-08-09 ClickBench run, and the bulk claim was wrong in sign (#445)

[jdatcmd]

The published ClickBench section carried a single 2026-08-05 run with two arms.
It reported columnar loading 3.8x slower than heap, and said nothing about Citus
or about parallel_copy. On the exact question #445 was opened about, that
understates the engine.

This PR has been rewritten around the 2026-08-09 re-run. The bulk claim it
originally made was wrong, and wrong in sign. What follows is the corrected
version; the history of this branch has the original.

The correction, which goes against us

An earlier revision of this branch said the parallel loader was 2.01x faster
than Citus on the bulk path, and said a fair bulk comparison was unmeasured.
That comparison put our sixteen worker path against a single Citus COPY
connection. Citus accepts concurrent writers into one columnar table and scales
well, so the claim rested on a premise nobody had checked. @ChronicallyJD asked
the question, measured it, and built the fair arm in #529: both bulk arms now
split the same file at the same boundaries using our own
pgcolumnar.file_split_offsets, with the same worker count, so they differ in
engine and in nothing else.

arm	connections	load	size
heap	1	148.8 s	7,818,592,256
columnar, serial COPY	1	447.4 s	1,479,745,536
citus columnar, serial COPY	1	187.2 s	1,662,558,208
columnar, parallel_copy	16	86.6 s	1,478,057,984
citus columnar, parallel COPY	16	49.1 s	1,663,025,152

Like for like at sixteen workers we are 1.76x slower than Citus, not 2.01x
faster. We are 1.72x faster than heap and 11.1 percent smaller than Citus. Our
serial to bulk speedup is 5.17x against their 3.81x, which is a statement about
how our loader scales and not a win over them on load time.

The query analysis is re-derived from this run

Not carried over. Two things changed beyond the numbers.

The loss table is ordered on the milliseconds columnar adds, not on the ratio.
The ratio does not order the six losses the same way, and the reason is the
baseline rather than anything about columnar. q29 and q21 both touch one column
and add 433 ms and 671 ms, but sit on baselines of 8,001 ms and 675 ms, so
comparable overheads read as 1.05 and 1.99. The previous text used q29's low
ratio as the evidence for an ordering by columns materialised. On this run that
argument does not hold in the ratio column; it holds in the added milliseconds.

"Every one of these predicates has a leading wildcard" was false. q28 is
URL <> '' and q29 is Referer <> ''. Both are still unprunable by a min and
max statistic, but for a different reason, and the text now says which: a group
is prunable for <> '' only when its minimum and maximum are both the empty
string. The conclusion that all six are #452 is unchanged.

Win and loss counts

Taken from the times, not from the printed ratio: 33 wins, 6 losses, 4 ties
against heap, and 39 wins against Citus. A tie is a query whose two arms are
closer to each other than the run to run scatter of their own repeated tries.
The harness itself counted this wrong, which is #531 and is fixed in #532. This
page should merge after that one so the document and the tool agree.

Two limits stated rather than discovered

Every arm was vacuumed and analyzed, which is not the normal state for a
columnar table because autovacuum cannot reach it (#415). And the columnar arm
runs with the analytical accelerators off, which is what a user gets by default.

The 2026-08-05 run is kept and labelled, as a second independent run of the same
benchmark.

Refs #445, #529, #531

[ChronicallyJD]

Checked the arithmetic and the provenance, and I can corroborate the build claim from an independent direction.

The numbers are internally consistent: 437.7 / 182.5 = 2.398, 182.5 / 90.9 = 2.008, 139.1 / 90.9 = 1.530, and 1,478,057,984 against 1,662,558,208 is 11.1% smaller. All four match the text.

"PostgreSQL 18.4 non-assert" is corroborated rather than taken. Running bench/provision.sh check on the bench earlier tonight read pg18n as non-assert from pg_config --configure — the flags, not the suffix — and confirmed citus.so and citus_columnar.so are installed in that same prefix. So the arm could have been co-loaded where you say it was, on a build whose non-assert status is a measured fact. That matters here more than usual, given #504.

Reading the two paths together instead of picking one is the right call, and the sentence saying so is the part I would keep.

One asymmetry I would name explicitly

The bulk-path row compares our 16-worker pgcolumnar.parallel_copy against Citus's single-connection COPY. I checked bench/run_clickbench.sh: parallel_copy is pgcolumnar-specific and there is no bulk Citus arm, so Citus is loaded serially in every run.

That is very likely a capability difference rather than an omission — but the text does not say which, and the two read identically to someone scanning the table:

• If Citus columnar cannot ingest one table from several writers concurrently, then "2.01x faster on the bulk path" is a fair and rather stronger claim than it currently sounds: it is a path they do not have, not a race they lost.
• If it can and simply was not measured, the ratio compares our best path against their non-best one, and 2.01x overstates it.

I do not know which is true, and it is answerable — N concurrent COPYs into one citus_columnar table either scale or serialise. One clause naming the reason would close the gap, and it costs nothing next to the run itself. Given the same question about our engine cost a week of single-writer tuning on a stale assumption, I would rather it were written down than inferred.

Not blocking: the claim as written is labelled by path and the labels are accurate.

Smaller

The loss table printing baselines beside ratios is right — a 1.08 on 8,157 ms and a 1.73 on 695 ms are not comparable quantities, and my own artifact was corrected for exactly that. And ordering by columns materialised with q29 as the counter-example (the most expensive per-value work, one column, smallest loss) is the kind of evidence that survives someone disagreeing with the conclusion.

[ChronicallyJD]

I asked above whether Citus has a bulk ingest path, and said the answer changes what 2.01x means. It does have one, and I have now measured it. The bulk comparison in this PR is not like-for-like.

Measured

Bench host, its own cluster on a free port so nothing existing was touched, citus_columnar 14.1-1 and pgcolumnar co-loaded, 4,000,000 rows of (bigint, int, text), one split shared by every arm, three runs:

arm	seconds (3 runs)
citus, serial COPY	3.06, 3.07, 3.06
citus, 8 concurrent COPY into one table	0.509, 0.518, 0.513
pgcolumnar, serial COPY	4.80, 4.80, 4.79
pgcolumnar, parallel_copy 8	1.137, 1.138, 1.135

All four arms loaded 4,000,000 rows. Scaling from concurrency: citus 5.9x, pgcolumnar 4.2x.

So Citus columnar accepts concurrent writers into a single table and scales well. The premise the published comparison rests on — that serial COPY is the only path Citus has — does not hold.

What this does and does not establish

Established: the 2.01x is our 16-worker path against their 1-connection path, and a fair bulk row needs a concurrent-COPY Citus arm. On this fixture, like-for-like, Citus is about 2.2x faster than parallel_copy, not 2.01x slower.

Not established: that 2.2x is the ClickBench number. My fixture is three narrow columns; ClickBench hits is 105. Ingest cost per row, compression work and per-column overhead all differ, and I would not publish my ratio as a correction of yours — one fixture is not the claim. The right number comes from the real fixture.

Suggested

The harness should have a bulk Citus arm so this is fair by construction rather than by review. bench/run_clickbench.sh already parameterises the pgcolumnar bulk arm with PGC_CB_PCOPY_WORKERS; the Citus equivalent is N concurrent \copy of the existing split into one columnar table, which is what I ran.

I am happy to add that arm and re-run the ClickBench load section on the bench, if you would rather it came from the harness than from a side experiment. Until there is a number for it, I would drop or qualify the "2.01x faster on the bulk path" sentence rather than publish it — the rest of the section (serial path, sizes, query latency) is unaffected and stands.

Flagging rather than blocking, but this one I would hold the merge for: it is the specific claim #445 was opened about.

[ChronicallyJD]

Reviewed the correction commit. The bulk-path fix is right and complete, and I found one defect in a neighbouring table.

The correction

Verified on the branch: no surviving "2.01x" or "faster than Citus" claim about the load path (the two remaining greps are the dimension-join table and the query latency sentence, which is like-for-like and fine). docs_style 6 checks / 0 fail, benchmarks.md ok — the new prose meets the STE rules.

Stating the measurement, naming the fixture as separate, and saying plainly that the table does not draw the comparison is more useful than a corrected ratio would have been.

One thing that will go stale: "A fair bulk comparison needs a Citus bulk arm in the harness, and that arm does not exist yet." #529 adds exactly that arm. If #529 lands first, this sentence is wrong on arrival; if this lands first, it wants a follow-up. Merge order is worth deciding rather than discovering.

The defect: a mislabelled cell in the latency table

| arm            | total across 43 queries | geometric mean against heap |
| heap           | 152.3 s                 |                             |
| columnar       | 123.3 s                 | 0.49                        |
| citus columnar | 256.9 s                 | 0.25 against citus          |

The citus row carries 0.25 against citus in a column headed against heap. As written it reads as citus columnar's geometric mean against heap, qualified by "against citus", which is not a quantity.

The totals show it cannot mean what the header says:

ratio	value
columnar / heap	0.81 (geomean 0.49 — consistent)
columnar / citus	0.48 (geomean 0.25 — consistent)
citus / heap	1.69

Citus is slower than heap across the 43 queries, so a 0.25 against heap in its row is impossible. The 0.25 is almost certainly columnar's geometric mean against citus, sitting one row down and under the wrong header.

Two ways out, and either is fine: give the citus row its own geomean against heap (which would be greater than 1), or split the column into "against heap" and "against citus" and put 0.25 in the columnar row.

Worth fixing before this merges: it is a published comparison table, and the number currently favours us in a cell that cannot support it. Same failure the bulk row had, one table down.

Everything else in the section stands, including the loss table printing baselines beside ratios and the q29 counter-example.

[ChronicallyJD]

Following up on the mislabelled cell. A complete three-arm run finished on the bench at 16:07 UTC today, and it settles what the 0.25 is.

To be exact about provenance: this is not a recomputation of your 2026-08-08 numbers. raw_timings.tsv holds one run at a time and has been overwritten, so the per-query data behind the table in this PR is gone. What follows is an independent run of the same harness on the same sample, which is why I am quoting it as corroboration and not as a correction of your arithmetic.

artifact	/srv/clickbench/raw_timings.tsv, 387 rows, complete
log	/home/jd/cb529.log, finished 2026-08-09 16:07 UTC
shape	43 queries x 3 arms x 3 tries, interleaved, 0 errors
sample	11,110,833 rows — the same every-ninth stride, 17 distinct EventDate, 5,727 distinct CounterID
aggregation	mine matches the harness: hot = min of tries 2 and 3, try 1 discarded

The two runs agree, so the 0.25 is not a run artefact

	your 2026-08-08	this run 2026-08-09	delta
heap total	152.3 s	154.9 s	+1.7%
columnar total	123.3 s	127.7 s	+3.6%
citus total	256.9 s	251.7 s	-2.0%

Sizes reproduce exactly: columnar 1,478,057,984 and citus 1,662,558,208 bytes, the same two numbers I corroborated this morning.

The geometric means, computed per query over all 43

ratio	geomean	total ratio
columnar / heap	0.53	0.82
citus / heap	1.91	1.62
columnar / citus	0.28	0.51

Your 0.49 and this run's 0.53 are the same quantity a run apart. Your 0.25 and this run's 0.28 are the same quantity a run apart. They are different quantities from each other, and the second one is columnar against citus.

So the citus row's honest value in a column headed against heap is about 1.9 — citus columnar is roughly twice as slow as heap per query, geometrically. Nothing near 0.25 can go in that cell.

Splitting the column is the better of the two exits I offered, because it lets the 0.25 stay:

arm	total across 43 queries	geomean vs heap	geomean vs citus
heap	152.3 s
columnar	123.3 s	0.49	0.25
citus columnar	256.9 s	~1.9

I have left the citus geomean as ~1.9 rather than filling in a number, because the exact value for your run is not recoverable — 1.91 is this run's. Either recompute it from a fresh run or leave the cell empty; do not copy my 1.91 into a table headed 2026-08-08.

The stale sentence is now false, not "will go stale"

A fair bulk comparison needs a Citus bulk arm in the harness, and that arm does not exist yet.

#529 merged at 14:08 UTC. The arm exists, and this run is the first one to use it — the clone is at 1607884, the merge commit, and the premise line fired: ok premise: the citus bulk arm loaded every row of the file.

It also produced the first fair bulk number, on the full ClickBench sample:

bulk arm, 16 concurrent sessions, same byte splits	load
citus_pcopy	49.1 s
columnar_pcopy (pgcolumnar.parallel_copy)	86.6 s

Both arms take their ranges from the same pgcolumnar.file_split_offsets call and feed them in with FROM PROGRAM, so they differ in the engine and not in how the file was divided. Citus is 1.76x faster than us on bulk load, measured like for like.

That is the number #445 was actually asking for, and it does not favour us. Which is exactly why the sentence must not ship saying the arm is absent: the PR would be publishing "we cannot compare this yet" on the same day the comparison landed and went against us. Whatever else this table says, that result should be in it.

Separate defect, in the harness rather than this PR

The win/loss count this PR quotes ("faster than heap on 35 of the 43") comes from a line that classifies on the rounded ratio. run_clickbench.sh does r1=$(ratio "$c" "$h") — printf "%.2f" — and then tests r1 < 1, so a query where columnar is faster by less than half a percent prints 1.00 and is counted a loss:

q13   columnar 1582.859 < heap 1588.095   ratio 0.99670 -> "1.00" -> counted a LOSS
q34   columnar 7221.889 < heap 7247.047   ratio 0.99653 -> "1.00" -> counted a LOSS
q35   columnar 7277.247 < heap 7278.204   ratio 0.99987 -> "1.00" -> counted a LOSS

This run printed "beats heap on 33 and loses on 10"; comparing the unrounded values gives 36 and 7.

I am not asking you to restate the count as 36. Three tenths of a percent is inside the noise of this harness, and calling those wins would be the same kind of overclaim as the 0.25. The defect is that a string formatted for display is being used as a number for a decision, and the fix is to compare the unrounded values — after which the honest reading of q13, q34 and q35 is "level", not "won". I will file it against the harness so it does not ride on this PR.

Everything else in my 14:08 review stands.

[ChronicallyJD]

Reviewed both commits against the run's raw timings. The rewrite is a large improvement and almost all of it verifies exactly — but the cell I filed at 13:46 is still wrong, in the same row, and the header change has made it harder to see rather than fixing it.

What I checked and could not fault

Every load number and every quantity derived from one, recomputed from /srv/clickbench/raw_timings.tsv and cb529.log:

claim	recomputed
serial: columnar 2.39x slower than Citus	447.4 / 187.2 = 2.3900	ok
16 workers: 1.76x slower than Citus	86.6 / 49.1 = 1.7637	ok
16 workers: 1.72x faster than heap	148.8 / 86.6 = 1.7182	ok
our serial→bulk speedup 5.17x	447.4 / 86.6 = 5.1663	ok
their serial→bulk speedup 3.81x	187.2 / 49.1 = 3.8126	ok
11.1 percent smaller than Citus (bulk pair)	0.11122	ok
11.0 percent on the serial pair	0.10996	ok
5.3x smaller than heap	5.2897	ok
totals 154.9 / 127.7 / 251.7 s	exact	ok
33 wins, 4 ties, 6 losses vs heap	exact	ok
ties are q13, q19, q34, q35	exact	ok
39 wins, no tie, 4 losses vs Citus (q16 q17 q19 q33)	exact	ok
all six columnar adds figures	exact on unrounded times	ok

The verdict counts land on the nose under #532's rule, including the tie set. I recomputed them independently before reading your text and got the same four queries.

The predicate correction is right, and I checked it rather than took it: queries.sql has q21-q24 on LIKE '%google%'/'%Google%', and q28 on URL <> '', q29 on Referer <> ''. Two of the six are not leading wildcards, exactly as you now say. Reading the added milliseconds rather than the ratio is the right instruction and q29 against q21 makes the case cleanly.

Still blocking: 0.28 is in the Citus row

| arm            | total across 43 queries | geometric mean      |
| heap           | 154.9 s                 |                     |
| columnar       | 127.7 s                 | 0.54 against heap   |
| citus columnar | 251.7 s                 | 0.28 against citus  |

0.28 is geomean(columnar / citus). I measured it: 0.2800. It is a property of the columnar arm, and it is sitting in the citus columnar row.

Changing the header from "geometric mean against heap" to "geometric mean" removed the visible contradiction without moving the number. Before, the cell disagreed with the column heading and a careful reader would stop. Now the cell is internally consistent and only the row is wrong, so nothing on the line tells the reader anything is off. Read literally, that row says citus columnar scores 0.28 against citus, which is 1.00 by construction.

The quantity that belongs in that row is missing entirely:

geomean(citus / heap) = 1.9103

That is the number which makes 0.28 interpretable — Citus is roughly 1.9x slower than heap per query, which is why beating it by 0.28 is a much weaker statement than beating heap by 0.53. Dropping it leaves the section with no way to see that.

Both numbers belong to arms, so put each on its own arm:

arm	total across 43 queries	geomean vs heap	geomean vs citus
heap	154.9 s
columnar	127.7 s	0.53	0.28
citus columnar	251.7 s	1.91

Unlike last time, these are all this run's numbers, so they can go in as measured — nothing is carried over from 08-08.

And the columnar geomean is 0.53, not 0.54

I could not reproduce 0.54 under any aggregation rule:

hot = min(try2, try3)   [the harness rule]  0.5348
min of all three tries                      0.5342
try 3 only                                  0.5326
try 1 only (cold)                           0.1270
arithmetic mean of ratios                   0.9207
ratio of the totals                         0.8244

Your totals match mine exactly, so we are aggregating the same way on the same data; 0.5348 rounds to 0.53.

To be fair about the direction: this one costs us. A lower ratio is a better result, so 0.54 understates columnar's advantage over heap. It does not flatter us and it is not the same class of error as the 0.28. But this section's whole subject is numbers that were wrong in sign, and a headline geomean that is off by a rounding step should not be the one thing left unchecked.

Summary

Two changes and this is ready as far as I am concerned: move 0.28 to the columnar row and give the Citus row its own 1.91, and correct 0.54 to 0.53. Everything else in the 2026-08-09 section I have verified against the raw data and it stands.