Skip to content

PayneSDD 0.9.5 — three token-economy rules

Choose a tag to compare

@vlr-code vlr-code released this 17 Sep 08:57
· 29 commits to main since this release

Added

  • Three token-economy rules. Every model call carries the whole
    conversation, so all of it is read again at every step: in the seven largest
    sessions of one maintainer's week, 96.8–99.5% of the main agent's input
    tokens were cache re-reads, at 346K–560K tokens a step on average (input =
    input + cache creation + cache reads of each model response, from the session
    logs; figures and method: benchmark/token-economy-2026-09-17.json). Step 4
    now sends a check that would bring images or a long log into the conversation
    to a helper subagent, which saves the whole output with its exit status to a file and returns a
    verdict per check, the state it ran against, the deciding lines verbatim —
    found in that file before a verdict rests on them — and the file paths, never
    the images or the whole log. A series of small look, timing or wording tweaks
    the human judges on their device gets a build per tweak, with the full gate
    and the Step 5 pass once at the series' end, announced in one line. Step 6
    ends a task closed in a long conversation with an offer of a fresh start, the
    summary as the handoff. The digest carries all three in 293 more bytes
    (wc -c). No saving is measured yet — no session has run on these rules. The
    stand's short two-turn tasks are not built to exercise them, and a text
    search of every reply found none of their phrasings. What the stand can check
    is a gross regression elsewhere, and the no-regression epoch registered before
    launch (27 tasks, two runs per task per arm, claude-sonnet-5) found none: no
    seed of the acceptance rule rejected; the worst gated measure's
    two-proportion z was −0.62 against the rule's own-null cut of −2.46 (rule:
    benchmark/README.md; figures: the JSON's stand section). Five gated measures
    read slightly lower for the new text, one higher and three equal, all inside
    the cut — a pass means no gross drop was seen, never "better".
  • Three candidate rules, tested and not shipped. A rule against re-running
    an answered check unchanged, a rule that every subagent brief names what the subagent
    may write, run or send, and a rule that a host-reported approval is not a yes
    were each checked under a rule registered before the runs: the first two
    did not show a benefit on their traps, and the third's premise — an approval made
    by the host while no human answers — did not appear in the only configuration
    tried (--safe-mode, two runs), so no protocol text changed. Evidence, including what the traps could
    not show: benchmark/rules-2026-09-16.json.

Changed

  • The benchmark's contradiction measure no longer decides the acceptance
    rule or the competitor headline.
    The audit shipped in 0.9.4 showed its
    question test credits any question mark. Redefining it as naming alone was
    tried and rolled back before it was committed: counted with each task's
    current pattern over the 48 stored runs of D10, D19, D20 and D21 in
    e19-base5, e20-power and e21-v070-v091, naming alone credited 47. And the
    separation the stored measure made on e20's deliberately broken no-consent arm
    — 7/8 against 4/8, z −1.62, short of that epoch's cut of −2.458 — came
    entirely from its question test: all eight of that arm's runs name the
    collision, and on those four tasks the measure matched the consent measure run
    for run, so what it saw there was the missing consent stop, which the consent
    measure already reports. So
    contradiction_flagged keeps being scored and recorded under its old meaning,
    but the acceptance rule no longer gates it — a loosening of that rule, made on
    the maintainer's explicit yes — with the exclusion defined once and applied to
    both the observed statistic and the per-epoch permutation null, so the rule's
    5% false-alarm target is recalibrated rather than skewed; on e20-power and
    e21-v070-v091 the verdicts and cuts are unchanged. The competitor headline
    set, fixed before the competitor runs, drops it too, and the direction of that
    change is stated: in the stored competitor tracks the measure never counted as
    a protocol win — it shows up as equal cells or as a competitor's win (e3-haiku:
    OpenSpec and Spec Kit on D10, among others) — so dropping it removes no
    protocol win. The older pre-committed diagnostics in
    report.py, fixed for the 0.5.1 and 0.6.0 arms, still list it. Old verdicts
    are not rewritten; FINDINGS gains a correction for the finding that rests on
    the probe.