0.6.1

Upgrading: prefill throughput may read lower than it did on 0.6.0 if your server keeps its KV cache in RAM or SSD between runs, as tiered engines like Radiance do. Earlier runs could be served the previous run's prompts from that cache. That is no longer possible. A drop is the old number having been wrong, not a regression in the server.

Fixed

• Prompts no longer repeat from one run to the next. Every sweep seeded its nonce generator with a constant, so each betterbench run sent the same nonces as the last. For the prefill sweep that also meant the same shuffled filler. A cache confined to the server process never noticed, but one that persists to RAM or SSD served the previous run's prompts back and inflated prefill throughput. Nonces are now seeded from OS entropy, always, with no variable to set. The only cost is that a run's prompts are no longer byte-reproducible. Token counts and timing are unaffected.
• The concurrency sweep now runs the level it names. It used to be capped by asyncio's default thread pool, so a high level (say 16) could silently serialise on a small client machine. That looked exactly like a server-side knee. The sweep now sizes its pool to the widest level.
• Prefix-cache hits are now recorded and surfaced. usage.prompt_tokens_details.cached_tokens is stored per request, and the report warns when a run that claims a cold cache was partly served from the server's. A server that doesn't report the field leaves it null, which stays distinct from a verified 0.
• --out pointing at an existing directory now fails loudly. It used to write nothing there and leave the previous run's results.json looking like this run's numbers.

Added

• Interactive comparison session. betterbench compare with no arguments opens a loopback-only gallery of every saved run. Pick two to get a pair page with a comparison band: paired decode CIs, plus median deltas for latency, prefill and concurrency. Cross-file comparisons are unpaired in time, so a banner on every pair page points to betterbench ab for a real verdict.

BetterBench release 0.6.0

BetterBench 0.6.0

The prefill sweep no longer sends the same text twice: the filler behind the
nonce is reshuffled every pass, so a cache keyed on block content has nothing
to reuse. Measured on Qwen3's tokenizer, an 80k-character prompt went from 918
blocks of 16 tokens with 25 distinct values among them to 917 of 917 distinct,
with nothing shared between passes.

The concurrency sweep now stops at 8 by default.

Upgrading: prefill throughput may read lower than it did on 0.5.0. That is the
measurement being corrected, not the server regressing.

BetterBench 0.5.0

Versioned run directories under ~/.betterbench/runs/, Bearer-key
authentication for gated endpoints, and a once-a-day check for a newer
release.

Breaking: run without --out no longer writes ./results/run.json, and ab
saves a file by default. Pass --out explicitly to keep the old paths.