The ClickBench report counts wins and losses from the string it already printed
for the table, not from the times:
r1=$(ratio "$c" "$h") # "%.2f"
if [ "$(awk -v r="$r1" 'BEGIN { print (r < 1) ? 1 : 0 }')" = 1 ]; then
wins=$((wins + 1)); else losses=$((losses + 1)); fi
ratio formats to two decimals. So a query where columnar is faster by less
than half a percent prints 1.00, fails r < 1, and is counted a loss.
That contradicts the legend printed directly above the same table:
-- hot times, milliseconds. 'x' is columnar over heap; above 1.00 means we lose.
1.00 is not above 1.00. The table and the sentence under it disagree, and
the sentence is the part that gets quoted.
What it did to the 2026-08-09 run
The report said:
columnar beats heap on 33 queries and loses on 10, at defaults.
From the raw times in the same run, three of those ten losses are queries where
columnar was faster:
| query |
heap |
columnar |
printed |
counted |
| q13 |
1588.095 |
1582.859 |
1.00 |
loss |
| q34 |
7247.047 |
7221.889 |
1.00 |
loss |
| q35 |
7278.204 |
7277.247 |
1.00 |
loss |
Counted on the unrounded times it is 36 and 7, not 33 and 10.
The fix is not simply to compare unrounded
That corrects the sign but over-claims in the other direction. q35 is 0.01
percent apart, and the two arms' own warm tries that run were 2.9 percent apart.
A difference smaller than the instrument's own scatter is not a result in either
direction.
The run already measures its scatter and throws it away: it keeps the best of
the warm tries and discards the rest. Keeping the range gives a per-query band
with no constant to argue about. Three-way on the 2026-08-09 data:
wins=33 ties=4 losses=6 ties: q13 q19 q34 q35
q19 is the one this catches that a two-way count would not: 1.03 against a warm
scatter of 4.29 percent on the heap arm, so that run cannot separate them.
The six real losses are q21, q22, q23, q24, q28, q29.
With CB_TRIES=2 there is one warm try per arm, the band is 0, and the verdict
degenerates to a strict comparison. That is the right degeneration: no repeat
means no evidence about repeatability.
Scope
ratio itself is fine and stays; it is a formatter and it is not the bug. The
decision moves to bench/cb_guards.sh beside the other guards, so
test/bench_guards.sh covers it in the matrix, which is what #465 set that file
up to do.
The ClickBench report counts wins and losses from the string it already printed
for the table, not from the times:
ratioformats to two decimals. So a query where columnar is faster by lessthan half a percent prints
1.00, failsr < 1, and is counted a loss.That contradicts the legend printed directly above the same table:
1.00is not above1.00. The table and the sentence under it disagree, andthe sentence is the part that gets quoted.
What it did to the 2026-08-09 run
The report said:
From the raw times in the same run, three of those ten losses are queries where
columnar was faster:
Counted on the unrounded times it is 36 and 7, not 33 and 10.
The fix is not simply to compare unrounded
That corrects the sign but over-claims in the other direction. q35 is 0.01
percent apart, and the two arms' own warm tries that run were 2.9 percent apart.
A difference smaller than the instrument's own scatter is not a result in either
direction.
The run already measures its scatter and throws it away: it keeps the best of
the warm tries and discards the rest. Keeping the range gives a per-query band
with no constant to argue about. Three-way on the 2026-08-09 data:
q19 is the one this catches that a two-way count would not: 1.03 against a warm
scatter of 4.29 percent on the heap arm, so that run cannot separate them.
The six real losses are q21, q22, q23, q24, q28, q29.
With
CB_TRIES=2there is one warm try per arm, the band is 0, and the verdictdegenerates to a strict comparison. That is the right degeneration: no repeat
means no evidence about repeatability.
Scope
ratioitself is fine and stays; it is a formatter and it is not the bug. Thedecision moves to
bench/cb_guards.shbeside the other guards, sotest/bench_guards.shcovers it in the matrix, which is what #465 set that fileup to do.