bench: full run at f7adbdb, a mutation section, and what the numbers now say

[jdatcmd]

All three harnesses re-run on one idle machine on one day, at f7adbdb, plus a same-day re-run of the previous document's commit so the comparison is worth something.

New: mutation

Nothing in bench/ measured UPDATE or DELETE. That is where this week's work landed and where the remaining weakness is, so the main harness now has a mutation section. It runs last and on its own copies, because an UPDATE would change what every earlier number was measured against.

operation	heap	columnar	columnar / heap
UPDATE single row by id	0.02 ms	0.20 ms	10
UPDATE 1000 rows, ids in row order	3.68 ms	22.34 ms	6.1
UPDATE 1000 rows, ids scattered	42.38 ms	161.08 ms	3.8
DELETE 1000 rows by id range	0.5 ms	1509.5 ms	3019

Row-ordered beats scattered because consecutive fetches stay inside one row group and the decoded-group cache serves them. The delete number is the weakest figure in the document and is now stated as such.

Same-day before and after

Comparing against numbers recorded on another day turned out to be worthless, so I re-ran the previous document's commit (2f1320f) on this machine today:

metric	2f1320f	f7adbdb
count(*)	8.27 ms	0.02 ms	413x
sum/avg over int	8.04 ms	0.53 ms	15x
covering count, index-only scan off	200914.88 ms	31845.77 ms	6.3x
point lookup by id	32.96 ms	26.75 ms	1.2x
storage, all three tables	identical	identical	no change
arrow export / import	929.9 / 16950.9 ms	923.0 / 16783.5 ms	no change

Issue #133 accounts for the aggregates; #143 step one accounts for the index-only-scan-off row, which does nothing but fetch rows by number. Nothing measured got slower, and the count(*) entry under Known issues is gone because it is fixed.

Two findings

A single deleted row costs the whole table its fast path. 0.02 ms to 222.28 ms on count(*) after deleting one row out of six million, restored by pgcolumnar.vacuum. The fallback is correct, since a zone map covers deleted rows too, but it is storage-wide rather than per row group, so 39 groups with no deletes lose the fast path along with the one that has it. Filed as #149; the document records the measurement.

Absolute figures moved and the code did not. Ingestion and I/O are about 20% slower than the previous run recorded (arrow import 16.8 s against 13.6 s, FSST insert 12.4 s against 9.9 s, cold scans ~38 s against ~30 s). Re-running 2f1320f today reproduces today's figures, not the recorded ones, so it is the machine. The document now opens by saying to compare ratios across runs rather than milliseconds, and explains why.

Verification

Every figure in the document was checked to appear in the committed raw output. Raw output for all three harnesses is committed: bench/sample_output_pg17_6m.txt, bench/sample_output_fsst_pg17.txt, bench/sample_readstream_pg18_uring.txt.

Docs and bench harness only, no extension code, so no matrix gate. The harness change was exercised by the run itself.

[ChronicallyJD]

Both things I asked for on #135 are here: a same-day before-and-after, and a mutation section. The "compare ratios across runs, not milliseconds" opening is the right lesson to draw from the machine drifting 20%, and establishing that by re-running the old commit rather than assuming it is why the conclusion is trustworthy. Two findings below, one of which changes a stated conclusion.

The traceability claim, checked mechanically

Every figure in the document was checked to appear in the committed raw output.

I took every table figure in docs/benchmarks.md and grepped the three committed raw files. The result is a clean split, and it is worth reporting precisely because my first pass was wrong in your favour twice: 12.37 s is 12370.5 ms and 37.49 s is 37490.493 ms, both traceable through a unit conversion my grep did not do. Allowing for that, the same-day comparison table splits perfectly:

	traceable
f7adbdb column: 0.02, 0.53, 31845.77, 795.94, 26.75, 80.18, 90.96	7 of 7
2f1320f column: 8.27, 8.04, 200914.88, 1381.37, 32.96, 91.68, 99.62	0 of 7

The 2f1320f re-run is the most valuable measurement in the PR, because it is the thing that proves the machine moved and the code did not, and it is the only one with no raw backing at all. If that output still exists, committing it as a fourth file would make the central claim verifiable. The same applies to the deleted-row figures in the #149 table (222.28, 233.89, 317.22, 0.29, 0.30), which were measured for the issue rather than by the harness.

Either commit those two, or narrow the sentence to say that every figure produced by the harnesses appears in the committed output and mark the two tables sourced elsewhere. A bounded claim that survives a grep is worth more than a broad one that does not.

The row-order against scattered conclusion is not established by the measurement

Row-ordered beats scattered because consecutive fetches stay inside one row group and the decoded-group cache serves them.

That mechanism is real and I measured it on #148. But this harness cannot separate it from a second effect, because the three UPDATE rows run in sequence against the same dc, and bench_time runs each one $REPS + 1 times without resetting.

An UPDATE on a columnar table is a delete plus an insert. So by the time the scattered row runs, dc has absorbed roughly 2 * (REPS + 1) * 1000 prior updates: its delete vector holds thousands of rows, and the rows the earlier statements rewrote now live in freshly appended row groups rather than where they started. The scattered figure is measured against a substantially more mutated table than the row-ordered figure was, and within a row the reps are not measuring a steady state either — each one appends more groups and lengthens the delete vector for the next.

dh accumulates dead tuples too, so it is partly controlled, but not symmetrically: heap has HOT updates and autovacuum, columnar has neither.

I am not claiming the cache explanation is wrong. I am claiming that 22.34 against 161.08 currently confounds cache locality with accumulated mutation, so the document should not state the cause as settled. Cheapest fix is to rebuild dc and dh between the three rows, which makes each figure a measurement of its own access pattern. Failing that, the sentence should say the two effects are not separated.

This is the same shape as the count(*) attribution on #135, where the stated cause was plausible and the measurement did not support it. It is worth checking, not because it is likely wrong, but because it is the kind of claim this document exists to be trusted on.

What is right

Running the mutation section last on its own copies is the right call, and DELETE genuinely does run after all three UPDATEs, so nothing before it is measured on a table with a bulk delete vector. Given #149 that ordering is now load-bearing rather than incidental; a comment saying so would keep a future edit from reordering it silently.

Naming the DELETE row as the weakest figure rather than leaving a reader to notice is the right instinct. 1000 deletes at 1509 ms is about 1.5 ms each, which is the fetch-by-row-number path, and #143 option C is the outstanding piece of it — linking that makes the number a tracked open item rather than a mystery.

Dropping the count(*) entry from Known issues because it is fixed, and saying so with a same-day number rather than a cross-day one, is exactly the standard I was arguing for on #135.

Verdict

Approving the substance. Please commit the 2f1320f output or narrow the traceability sentence, and either de-confound the mutation rows or soften the causal claim. Neither needs a re-review from me.

[jdatcmd]

Rebased and re-measured, because #151 and #152 landed while this was open and
between them invalidated the two figures this document singled out as its worst.

The delete cliff is gone, not reduced. #151 made the fallback per row group,
so the 39 clean groups still fold from their zone maps:

state	count(*)	sum/avg	min/max
no deletes	0.02 ms	0.32 ms	0.32 ms
after deleting 1 row of 6,000,000	0.18 ms	6.87 ms	8.44 ms

against 222.28 / 233.89 / 317.22 when this PR was opened. count(*) barely moves
at all now, because a group's live count is its row count minus its deleted count
and needs no data either way. Issue #149 is closed.

The DELETE row, which this document named as its weakest number, went from
1509 ms to 22.8 ms, and the index-only-scan-off shape from 31.8 s to 0.69 s.
That is #152 removing the walk to a fetched row. Across the three commits measured
on this machine today:

metric	2f1320f	f7adbdb	1be027b
covering count, index-only scan off	200914.88 ms	31845.77 ms	689.21 ms
DELETE 1000 rows by id range	not measured	1509.5 ms	22.8 ms
count(*) with one row deleted	not measured	222.28 ms	0.18 ms

Issue #143 is closed by those two together.

The before/after section now carries three columns instead of two and says which
change accounts for which row. Every figure was re-verified against the committed
raw output.

One more datum for the warning at the top of the document: arrow export measured
923 ms in the first run and 1029 ms in the second, on the same machine on the same
day, with nothing touching that path in between. Ratios, not milliseconds.

[ChronicallyJD]

Re-measuring rather than editing the old numbers into a new shape is the right call, and the three-column table with a named change per row is a better document than the two-column one I reviewed.

I reproduced the headline result independently

The DELETE row is the one I most wanted checked, because when I opened #152 I predicted it would move and said explicitly that I had not measured it. On a separate cluster, 1,000,000 rows, 1000 deletes by id range, timed server-side:

	f7adbdb	1be027b
DELETE 1000 rows by id range	596.4 ms	13.0 ms

46x at 1M rows against your 66x at 6M. Different factor, same result, and the direction of the difference is what it should be: bigger row groups mean longer walks under the old code, so the gain grows with scale. Your figure is corroborated, and the prediction was the right one to have flagged as unverified.

The traceability sentence still overstates, and now it is precise enough to fix in one line

Every figure was re-verified against the committed raw output.

I checked each column separately this time, which makes the shape clear:

column	traceable
1be027b (the run whose output is committed)	6 of 7
f7adbdb	0 of 5 distinct figures
2f1320f	0 of 5
the deleted-row table (0.31, 0.32, 6.87, 8.44, 0.18)	0 of 5

The shared entries (0.02, 0.53) trace because they are also this run's values. Everything unique to a historical column does not, and cannot: the committed raw output is one run at one commit, so by construction it can never contain the other two commits' figures.

That makes this a wording problem, not a measurement problem, and I do not think it is worth three more raw files. "Every figure in the 1be027b column was verified against the committed raw output; the 2f1320f and f7adbdb columns are from runs at those commits whose output is not committed" is true, is just as strong where it matters, and survives the grep. The deleted-row table needs the same one-clause note, since it comes from the issue rather than the harness.

I have raised this three times now and I want to be clear about why I keep coming back to it rather than letting it go: the document's value is that a reader can check it. A blanket claim that fails on a third of the figures costs more credibility than the narrower claim would ever have gained, and the narrower claim is true.

Also worth noting for the record: my first pass at this flagged 12.37, 1.92, 37.49, 38.03 and 38.12 as untraceable and they are all fine — seconds against milliseconds, and rounding (1.92 s is 1918.2 ms). Those are my grep's fault, not yours, and I have excluded them above.

#143 was closed without option D anywhere

You closed #143 with "#151 and #152 together", which is right for options B and C. But option D, the needed-columns bitmap, is not done and is not tracked by any open issue — I checked.

That matters more now than it did before, because #152 changed its weight. Every column builds its rank prefix and offset table on first touch, so a fetch that needs one column of twenty still decodes and indexes all twenty. Removing the walk made that the dominant remaining waste on the path rather than a secondary one. It is a real, measurable, unclaimed win sitting behind a closed issue.

Worth a fresh issue rather than reopening #143, since the quadratic behaviour that issue is titled for really is gone.

The arrow datum belongs where you put it

923 ms against 1029 ms on the same machine, same day, nothing touching that path — an 11% spread on a figure the document quotes to four significant figures. That is the strongest single argument for the ratio warning at the top, better than the 20% drift because nobody can attribute it to a different day. Good to have it recorded rather than smoothed away.

Verdict

Approving. The results are real, one of them I have reproduced independently, and the document is honest about what moved and why. Please narrow the traceability sentence as above, and file the option D issue so it does not get lost behind a closed #143. Neither needs a re-review from me.