Skip to content

v1.108.309 — A mean hides the tail, and a default invents a comparison

Choose a tag to compare

@jgravelle jgravelle released this 29 Aug 19:08
· 32 commits to main since this release

Two instruments reported a confident number about a distribution they could not see. Both were found by reading an external audit against our own tree — Revenium, 2026-08: the top 1% of 14,680 agent runs carried 46% of spend, the top 5% carried 77%, and one unattended session ran 4,819 calls over four days.

Fixed — analyze_perf differenced latency against a baseline that never measured it

_diff_baseline read float(b.get("p50_ms", 0.0)). The only baseline that ships, benchmarks/token_baselines/v1.108.163.json, carries tokens_saved for its three tools and no latency keys at all — so the zero stood in for a measurement nobody took, and the subtraction was published as p50_delta_ms, a name asserting a comparison happened. Measured against that file, a tool at p95 900 ms reported p95_delta_ms: 900.0: read by any human as a 900 ms regression against a release that never timed it.

The test could not see it because the fixture was richer than the artifact. test_baseline_diff_with_synthetic_baseline builds a baseline carrying calls, p50_ms and p95_ms — keys the real file has for no tool — so the fabricated path was structurally invisible to the test written about that exact code. The replacement reads every baseline off disk.

An absent measurement now returns null plus a not_comparable entry naming which side could not answer, and baseline_meta.tools_not_fully_comparable carries the count. Calls and tokens keep their meaningful zero on the current side: a tool nobody called really did save nothing and really was called zero times. Latency has no such zero, and inventing one is the same defect from the other end.

Added — heaviest_by_total_ms: where the time actually went

slowest_by_p95 ranks how slow one call is, and it was the only ranking. Where the time went is count x latency, and the two orderings disagree whenever a fast tool is called often: 4,000 calls at p95 900 ms consume 100x three calls at 12,000 ms, and the report put the second first.

totals carries the grand total; each row carries total_ms, share and count. A share over a zero total refuses rather than dividing. A ring-capped tool's share is a lower bound and the capped tools are named — the 512-call ring bites hardest on the busiest tool, which is the one this ranking exists to surface.

Added — retrieval inflation reports where the excess sits

analyze_regret's ratio is a mean, so one need burning 400 calls inside a corpus of 1,000 comes out at 1.4x — and the digest one-liner quoted exactly that number. concentration reports top_need_share and a head_share over the worst tenth of information needs, computed over excess_calls rather than calls, since every need costs one call by definition. The digest now distinguishes two ledgers the ratio cannot: an identical 1.4x that is one runaway query versus four ordinary ones.

Changed — one producer for the per-tool latency shape

token_tracker.latency_bucket is now the only place that shape is built. analyze_perf held the second copy and the two agreed digit for digit, which is what makes a later divergence invisible; its local _percentile is deleted rather than kept as a wrapper. The bucket gains total_ms and a measured p95_is_max — the percentile index collapses to the last element for every n <= 20, so two published fields carried one sample with nothing saying so.

Tests: 8,848 passed, 13 skipped, 0 failed. CI green on lint plus all 8 matrix jobs.