docs: refresh benchmarks.md from the 2026-08-12 run (ClickBench win 33->25, #452 trade, #595)

[ChronicallyJD]

Refresh benchmarks.md from a full 2026-08-12 run on current main

Re-ran the whole harness on main at 2fe6596, pg18n non-assert, 16 cores/62 GB: main suite (6M rows, median of 5), FSST, joins (#401, 20M), and cross-engine ClickBench (11.1M-row stride sample, heap/columnar/columnar_tuned/citus/duckdb). Every measured table is updated from the raw output; not-re-measured sections (TSBS, read-stream/AIO, parallel) keep their prior attributions.

The substantive change: ClickBench win count 33 → 25, and why

Columnar now beats heap on 25/43 (was 33), because #452 phase-2 decode gating shipped since the last run. An A/B on this build/data (gating on vs off) attributes it precisely: gating helps q24 1.8x (its target — a wide SELECT * under a selective filter, 6098→3395 ms) but regresses eight less-selective queries 1.2–2x (q11/q12/q14/q15/q25/q27/q31/q32), because it evaluates the qual per vector even when nothing is skippable. It's a deliberate trade, documented as such, with the fix (gate the gating on selectivity — runtime probe) filed as #595 (prototype in progress).

Other updates: load gap restated as a serial-path property (columnar 2.38x serial / 1.73x bulk vs Citus, still smallest on disk at 1.48 GB); parallel_flush documented as a measured opt-in, not eventually-default (#445 slice 4); anchored LIKE (#510), detoast-once (#587), FSST verdict cache (#472), bloom by distinct count (#467) in "what changed". Noted that #423 is resolved — q21 no longer errors in the fresh run.

Judgment calls flagged for review (I did not want to guess)

1. ClickBench summary totals + geomean (heap 168.2 s / columnar 140.4 s / citus 266.9 s; geomeans 0.52 / 1.77 / 0.29): the harness printed per-query times but not these aggregates, so I derived them arithmetically from the measured rows. Confirm you want them derived vs left blank.
2. Two historical micro-benchmark paragraphs now read dissonantly against the fresh tables and were left verbatim: the import/export analysis ("import ~18x export" — the new tables show ~3.4x) and the Mutation "previous record" note (1.91 ms single-row vs the note's older framing). Both are separate un-re-measured benches; annotate or drop as you prefer.
3. The 2026-08-05 ClickBench subsection is kept as an independent historical run; I annotated it where Ungrouped vectorized aggregate errors on a varlena filter column: unsupported byval length: -1 #423 resolved.

🤖 Generated with Claude Code

https://claude.ai/code/session_017N82wDmsawqSWoWkmxtHmW

[jdatcmd]

Request changes — one blocking gate, then this is good to merge. The substance is accurate and the honesty about my own #452 regression is exactly right.

Blocking: the CI red is real, and it is docs_style

The "suites (PG 17)" / "suites (PG 18)" failures are not a timeout or infra flake this time. docs_style is a registered suite, and this refresh introduces STE violations that fail it. I verified by attribution, not assumption:

• ste_check.py on main's benchmarks.md: ok (clean).
• ste_check.py on this branch's benchmarks.md: FAIL — 17 long, 6 prose double-hyphen.

So the refresh introduced them; they are not pre-existing. A docs-only PR can legitimately turn the suite job red, and here it does. Representative hits:

• 41 words: "It is a large win on a wide SELECT * under a highly selective filter (q24 from 6098 ms to 3395 ms ...)"
• 38 words: "This is why the ratio against heap falls as the number of rows rises, from 64 times ..."
• 38 words: "Do not compare absolute milliseconds.** This run is on a larger machine ..."
• 31 words: "It helps a wide bulk load of many numeric columns by up to about 14 percent and regresses ..."
• 27 words: "Detoast once (#587). A toasted text value is now detoasted a single time per scan ..."
• The 6 prose double-hyphens include the -- in the #452 blockquote ("a wide SELECT * under a highly selective filter --", "q11 ... q32, each about 1.2 to 2x --") and the q21 line ("q21 no longer errors --").

Fix: split the 17 sentences under 25 words and restructure the 6 -- into separate sentences (the same STE discipline the rest of docs/ already meets). python3 test/ste_check.py docs/benchmarks.md prints the full list; green there turns the suite job green.

Substance: accurate, and I checked the part that is mine

The #452 phase-2 trade is reported correctly. Phase 2 evaluates the executor qual per 1024-row vector even when nothing is skippable, so it costs on less-selective scans and pays off only under a selective filter over a wide row. The A/B (gating on vs off) is the right instrument, the eight attributed regressions (q11/q12/q14/q15/q25/q27/q31/q32, ~1.2–2x, recovering with gating off while q24 loses its win) match the mechanism, and 33→25 with #595 tracking the selectivity gate is the correct disposition. Calling it "a deliberate trade, not a regression to hide" is the right call — thank you for not burying it.

The three judgment calls you flagged

1. Derived ClickBench aggregates (totals + geomeans). Keep them — a derived aggregate beats a blank. One ask: label them as derived (e.g. a one-line "totals and geomeans computed from the per-query rows; the harness does not emit aggregates") so a later reader does not mistake them for harness output, and re-verify the geomean once against the 43 rows. I did not independently re-sum all 43 here, so I am not vouching for the arithmetic, only for the decision to publish them.
2. The two dissonant historical paragraphs (import/export ~18x vs new ~3.4x; the Mutation "previous record" note). Annotate, do not drop — one clause each pinning them to their run and machine so the numbers do not read as comparable to the 2026-08-12 figures.
3. Keeping the 2026-08-05 subsection as annotated history. Correct — a second independent run is worth keeping, and annotating where #423 resolved q21 is the right touch.

Nothing else blocks. Fix the STE gate and I will approve.

[jdatcmd]

The STE gate is the only thing between this and merge. The "suites (PG 17/18)" red is not a flake or a timeout: docs_style is a registered suite, and the refresh introduced STE violations, so the suite job fails. I attributed it rather than guessed:

• ste_check.py on main's benchmarks.md -> ok
• ste_check.py on this branch -> FAIL: 17 long, 6 prose double-hyphen

So the refresh created them. python3 test/ste_check.py docs/benchmarks.md prints the full list; the -- hits include the #452 blockquote and the q21 line. Split the 17 sentences under 25 words and restructure the 6 double-hyphens into separate sentences, and the suite goes green.

The substance is already right and I have signed off on it, including the honest #452 phase-2 trade and #595 as the tracked fix. Push the STE fix and I will re-review and approve.

[ChronicallyJD]

Fixed the STE gate (c9ca601). Split the 17 over-25-word sentences at clause boundaries and restructured the 6 prose double-hyphens (the #452 blockquote and the q21 note) into separate sentences. python3 test/ste_check.py docs/benchmarks.md now reports ok.

No numbers moved. I verified independently of the sentence edits: the multiset of every numeric/ratio/byte/#id/qNN token is byte-identical before and after — the only token-level difference is one 12, becoming 12. where a sentence split at the "8 and 12 GB" machine spec. The tables, the 25/13/5 win count, the #452 A/B figures, and the #423-resolved note are all exactly as reviewed. Suites should go green now; ready for re-review.

[jdatcmd]

STE fix verified: ste_check.py on the updated benchmarks.md now reports ok (was 17 long + 6 prose double-hyphen), and the CI suites are green — docs_style passes. The substance survived the rewrite intact: q24 6098->3395, win 33->25, #595 tracked, the totals/geomeans, and the #452 trade blockquote split into short sentences with nothing dropped (the runtime-probe note is sharper for it). My changes-requested is resolved. Approving.