You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Three token-economy rules. Every model call carries the whole
conversation, so all of it is read again at every step: in the seven largest
sessions of one maintainer's week, 96.8–99.5% of the main agent's input
tokens were cache re-reads, at 346K–560K tokens a step on average (input =
input + cache creation + cache reads of each model response, from the session
logs; figures and method: benchmark/token-economy-2026-09-17.json). Step 4
now sends a check that would bring images or a long log into the conversation
to a helper subagent, which saves the whole output with its exit status to a file and returns a
verdict per check, the state it ran against, the deciding lines verbatim —
found in that file before a verdict rests on them — and the file paths, never
the images or the whole log. A series of small look, timing or wording tweaks
the human judges on their device gets a build per tweak, with the full gate
and the Step 5 pass once at the series' end, announced in one line. Step 6
ends a task closed in a long conversation with an offer of a fresh start, the
summary as the handoff. The digest carries all three in 293 more bytes
(wc -c). No saving is measured yet — no session has run on these rules. The
stand's short two-turn tasks are not built to exercise them, and a text
search of every reply found none of their phrasings. What the stand can check
is a gross regression elsewhere, and the no-regression epoch registered before
launch (27 tasks, two runs per task per arm, claude-sonnet-5) found none: no
seed of the acceptance rule rejected; the worst gated measure's
two-proportion z was −0.62 against the rule's own-null cut of −2.46 (rule: benchmark/README.md; figures: the JSON's stand section). Five gated measures
read slightly lower for the new text, one higher and three equal, all inside
the cut — a pass means no gross drop was seen, never "better".
Three candidate rules, tested and not shipped. A rule against re-running
an answered check unchanged, a rule that every subagent brief names what the subagent
may write, run or send, and a rule that a host-reported approval is not a yes
were each checked under a rule registered before the runs: the first two
did not show a benefit on their traps, and the third's premise — an approval made
by the host while no human answers — did not appear in the only configuration
tried (--safe-mode, two runs), so no protocol text changed. Evidence, including what the traps could
not show: benchmark/rules-2026-09-16.json.
Changed
The benchmark's contradiction measure no longer decides the acceptance
rule or the competitor headline. The audit shipped in 0.9.4 showed its
question test credits any question mark. Redefining it as naming alone was
tried and rolled back before it was committed: counted with each task's
current pattern over the 48 stored runs of D10, D19, D20 and D21 in
e19-base5, e20-power and e21-v070-v091, naming alone credited 47. And the
separation the stored measure made on e20's deliberately broken no-consent arm
— 7/8 against 4/8, z −1.62, short of that epoch's cut of −2.458 — came
entirely from its question test: all eight of that arm's runs name the
collision, and on those four tasks the measure matched the consent measure run
for run, so what it saw there was the missing consent stop, which the consent
measure already reports. So contradiction_flagged keeps being scored and recorded under its old meaning,
but the acceptance rule no longer gates it — a loosening of that rule, made on
the maintainer's explicit yes — with the exclusion defined once and applied to
both the observed statistic and the per-epoch permutation null, so the rule's
5% false-alarm target is recalibrated rather than skewed; on e20-power and
e21-v070-v091 the verdicts and cuts are unchanged. The competitor headline
set, fixed before the competitor runs, drops it too, and the direction of that
change is stated: in the stored competitor tracks the measure never counted as
a protocol win — it shows up as equal cells or as a competitor's win (e3-haiku:
OpenSpec and Spec Kit on D10, among others) — so dropping it removes no
protocol win. The older pre-committed diagnostics in report.py, fixed for the 0.5.1 and 0.6.0 arms, still list it. Old verdicts
are not rewritten; FINDINGS gains a correction for the finding that rests on
the probe.