Skip to content

v0.1.1

@SvangenStudios SvangenStudios tagged this 02 Aug 10:05
Corrects the decode-share measurement published in v0.1.0 and re-measures the
8192/2048 A/B token-weighted. One published conclusion is overturned.

  decode share, 8192      7.1%  ->  1.7%  (median of 3, verified 1.5%)
  decode share, 2048      7.3%  ->  5.0%  (median of 6, verified 5.1%)
  prefill cost of 2048   -3.9%  -> -7.7%
  agent prefix reuse    97-99%  ->  95.7 / 98.4 / 98.1%

"Chunk size is a jitter lever, not a fairness lever" is wrong. 2048 delivers
2.9x the decode share of 8192 during a long prefill. Decode receives one
speculative step per prefill chunk yielding ~k accepted tokens regardless of
chunk size, so decode throughput scales with the NUMBER of chunks.

The original instrument counted SSE events as tokens (~2.1-2.5 accepted tokens
per chunk here), had no prefill window, and took a cold reference. It is now
token-weighted with an explicit window, uses each stream as its own baseline,
cross-checks every stream against the server's usage.completion_tokens,
propagates worker-thread errors, and salts all measurement prompts. 27
regression tests in CI.

v0.1.0 remains as published. Raw output for all eleven runs is committed with
provenance in results/raw/ab-256k-2026-08-02.log.
Assets 2
Loading