Re-measure the single-engine benchmarks on current main (#367)

[ChronicallyJD]

What

The single-engine half of the 1.0-alpha benchmark pass (#367). The cross-engine
section is not touched here — that is a separate 100M run which is still going, and
it will land as a second commit on this branch.

Those sections were one run from 2026-07-27 at commit 7a9c9f7. That predates
column projection (#339), every vectorized aggregate, and the entire planner and
fetch-cache wave (#350, #353, #354, #357, #359, #361, #362, #363). Re-run at
eb5c7ef with bench/run_bench.sh unchanged — verified, no commit touches it
between the two — so the comparison is valid modulo machine and major.

The correction that matters most

The page described the point lookup as an open regression at 1251.88 ms and
pointed readers at #171 with "treat the row as a bug report".

#171 closed on 2026-07-27 — the same day the published run was taken. So the page
has spent a week telling users that a columnar point lookup costs 1.25 seconds. It is
15.64 ms.

What moved

measurement	7a9c9f7	eb5c7ef
point lookup by indexed id	1,251.88 ms	15.64 ms
filtered agg, min/max-skippable	90.70	11.19
projection: 3 of 8 cols, 1% filter	85.83	10.85
sorted projection, after vacuum_sorted	47.72	1.61
Arrow import	17,721 ms	4,037 ms
Parquet import	17,762 ms	4,468 ms
index-only scan (covering count)	7.53	4.97
storage: heap / zstd / none	707 / 135 / 40 MB	unchanged

The ~8x on the filtered aggregate and the projection query is column projection
(#339) reading only referenced columns. The same cause makes the non-vectorized arm
of the vectorization toggle much faster (1392 ms to 270 ms), which shrinks that
speedup from 2731x to 601x — the ratio fell because the baseline got better, and the
page now says so.

The mutation table, which needed investigating rather than transcribing

The harness reported single-row UPDATE at 0.22 ms before and 71.73 ms now, which
reads as a 326x regression on an unchanged harness. It is not one.

• My first hypothesis was my own The #355 index-fetch penalty changes the estimate but not the plan: the cheaper path is already freed when it is applied #362/The index-fetch cost model sizes the decode from the emitted columns, but the fetch decodes the attribute prefix: 200x invisible to reltarget->width #363 costing the index path out of a point
  UPDATE. Disproven: the UPDATE keeps Index Scan, and penalty on/off timings
  match (19-22 against 19-23 ms).
• A/B on one machine and one major, only the code differing: 7a9c9f7 gives
  162 / 154 / 171 ms; eb5c7ef gives 19 / 17 / 20 ms. The path is ~8x
  faster. On 7a9c9f7 the plan carries no index-scan node at all, which is the
  Point lookup regressed 23.75 ms to 1251.88 ms once ANALYZE has run, and ANALYZE itself is pathologically slow on some shapes #171-era planner taking a full scan for a point lookup.
• So 0.22 ms is unreproducible on this machine with either build, and the table
  now says that rather than presenting it as a baseline this run missed.

The new table also explains the shape a reader will otherwise find odd: the columnar
cost barely moves between one row and 1000, because the row group is the unit of the
work. Heap pays per row; columnar pays per row group.

Not re-measured, and now stated

bench/run_bench_readstream.sh needs a PG18 built --with-liburing. No such build
exists on this machine, so that section keeps its earlier numbers and the preamble
says so instead of letting them read as current.

Style

test/ste_check.py passes on every user-facing document.

🤖 Generated with Claude Code

[jdatcmd]

Approved

Docs-only, CI green after the install-step flake cleared (exit 124 in "Install
PostgreSQL 18", the same package-fetch hang #352 addressed, not your change).

The point-lookup correction is the important part of this PR and it should be
said plainly: docs/benchmarks.md has spent a week telling users a columnar point
lookup costs 1,251.88 ms and inviting them to treat the row as a bug report against
#171, which closed the same day the run was taken. It is 15.64 ms. That is the kind
of stale number that costs a project users who never file anything, and it is a good
argument for the re-measure being an alpha gate rather than a nice-to-have.

The mutation table is the part I would have got wrong. A harness reporting
0.22 ms before and 71.73 ms now, on an unchanged harness, is exactly the shape that
gets transcribed into a regression. Three things you did instead:

• Formed the obvious hypothesis (your own #362/#363 costing the index path out of a
  point UPDATE) and disproved it rather than assuming it, with penalty on/off
  timings that match.
• A/B'd on one machine and one major with only the code differing: 7a9c9f7 gives
  162/154/171 ms, eb5c7ef gives 19/17/20. The path is 8x faster, and the old
  plan carries no index-scan node at all, which is the #171-era planner.
• Concluded 0.22 ms is unreproducible on this machine with either build and put
  that in the page, rather than presenting it as a baseline this run missed.

That is the right handling of a number that cannot be reproduced: say so, in the
document, where the reader is.

Two smaller things worth noting:

• The vectorization speedup shrinking from 2731x to 601x because the baseline
  improved is exactly the kind of ratio that looks like a regression in a diff and
  is not. Good that the page explains the cause rather than just carrying the new
  number.
• The read-stream section keeping its old numbers with the reason stated is
  better than silently re-running it against a build without --with-liburing and
  publishing something that measures nothing.

I have verified the diff carries no em or en dashes, per docs_style.sh.

Merging. The cross-engine section lands as your second commit; ping #367 when the
100M run finishes and I will cut VERSION and [Unreleased].

[ChronicallyJD]

Commit 2: the cross-engine section, which completes this PR

1ba0686. Four engines on the same 100,000,000 TSBS rows, at eb5c7ef, with the
index set and the parallelism stated per row. 384 timing measurements plus a 64-cell
spill pass.

Citus is measured rather than omitted. The old table dropped it with "Citus was
not re-measured and is omitted rather than carried forward". Both it and TimescaleDB
turned out to be unreproducible from the box — cpu_ts held zero rows and the
Citus cluster had no bench database at all — so both fixtures were rebuilt from
the TSBS source before anything was measured. All four engines now carry the same
(hostname, time DESC) btree; Citus columnar accepts one, so this is genuine parity
rather than a footnote.

What moved

query	previous (P=4)	now (P=4)
q1	10,335	73
q2	123,546	494
q3	161,972	861
q5	16,443	11,908
q6	2,292	2,324
q7	368	44,058

q2 and q3 are 250x and 188x — column projection, the fetch cache work and the planner
fixes compounding. pgColumnar leads q5 and q6 with workers. TimescaleDB still leads
every host-filtered query, because its columnstore segments by hostname and
pgColumnar stores in load order.

q7 went the wrong way, and that is #376

368 ms to 44,058 ms. It is not a storage property: the planner declines an index scan
this query wants, because the cost model charges the index path for the rows it
returns rather than the rows the query reads. DISTINCT ON reads one row per host.
Same query, pgcolumnar.enable_index_fetch_penalty = off: 769 ms.

Filed as #376, it is mine from #355/#362/#363, and it is on by default. The table
gives both numbers and points at the issue rather than presenting 44 s as a property
of columnar storage. I would not cut the release without deciding about it — that
is on #367 and it is your call.

Two things the table records rather than hides

TimescaleDB cannot run the parallel arm on this host. Any query that gets a
parallel plan fails in a parallel worker. Those cells say fails. They are not
reported as the time the failed statement printed — my harness was doing exactly that
until I noticed q5 reading 10,500 ms serial against 11 ms parallel, which is not a
speedup four workers can produce. psql prints a Time: line for a failed statement;
the harness now checks for ERROR: as a distinct outcome.

Spill is reported beside every number, per your #358 requirement — and it
qualifies the inference rather than confirming it. Five cells spill. Their spread
across five runs is 1.00x to 1.04x, while the widest spread in either table
(1.18x) belongs to a cell that does not spill. So "the spilling plan is the
unstable one" was a property of spilling at work_mem = 4MB, where you measured the
1.43x, not of spilling as such. At 256 MB the sort has room and the spill is cheap.
Your instruction stands and is followed; it is the inference the data corrects.

I nearly wrote "nothing spilled" off the first nineteen cells, which were all the small
host-filtered queries. The five that spill are the heavy shapes at the end.

Not re-measured, and stated

The read-stream section needs a PG18 built --with-liburing and no such build exists
on this machine, so it keeps its earlier numbers and the preamble says so.

test/ste_check.py passes on every user-facing document.