Skip to content

v1.22.0

Choose a tag to compare

@martinholovsky martinholovsky released this 05 Aug 15:02
· 5 commits to main since this release
5b0bfd9

[1.22.0] - 2026-08-05

The activation release — and gates that prove they can fail. Two findings, both
from real sessions rather than from re-reading docs. First: a ~25-turn session doing
upstream-contribution work invoked zero sota-* skills, because only the
frontmatter description auto-loads and the old trigger verbs all assumed you own the
codebase. The router's content was never in question; it was never read. Second: the
repo's own CI printed "ok" fifteen times without anything ever proving those checks
could still fail — the thing this library requires of everyone else.

Front door checked: check-negative-controls · pull request · publishing · negative control

Nothing in this release is measured; no lift is claimed. desc-routing reads
+0.00 (saturated) and cannot distinguish two descriptions; the AUDIT arm remains
+0.00 across seven instruments.

Activation, not content. A real ~25-turn session doing upstream contribution work
invoked zero sota-* skills. The router body — which already contained rules that
would have caught the worst error — was never read, so its quality was irrelevant. Only
the frontmatter description auto-loads
(Agent Skills docs:
"until a Skill is triggered, only its name and description occupy context"), which makes
the description the entire trigger classifier and everything downstream dead weight until
it fires.

Added

  • Router description now covers code you do not own — reviewing a pull request or
    diff, responding to code review, evaluating someone else's patch, preparing an upstream
    contribution, including mid-session once you are already reading source, a diff or CI
    config
    . The old verbs all assumed you own the codebase. Paid for by cutting the
    31-domain enumeration (−445 chars), which duplicated the routing table in the body;
    the description is a matcher, and verbs like diff/upstream/review match harder
    than 31 nouns. Net 951/1024, recovering 73 chars of headroom against invariant 4.
  • Principle 7 — restate from the artifact, never from your own summary. Principle 0
    listed what does not validate (training data, plausibility, "the rules file says so")
    and never named the commonest one: your own earlier prose in this session.
  • Principle 8 — publishing under someone else's name raises the bar, with the full
    procedure as sota-docs-workflow rules/03 §8. The library assumed findings go to
    the person who asked; a PR comment, issue or commit message posted upstream is public,
    attributed and permanent. Nothing covered that. (llm-engineering rules/04's approval
    gates are about agents you build; rules/03 §3's "ask, don't assert" is a tone rule.)
  • Falsification precondition on principle 0 — state what result would falsify a claim
    before measuring; if none could, read the deciding code path instead of benchmarking
    symptoms.

Fixed

  • The shipped hook buried its own routing rule. install.sh's HOOK_CMD put routing
    in a subordinate clause after two numbered rules. In the session above, rules (1) and
    (2) were obeyed every turn and the routing clause was dropped every turn — same text,
    same repetition, opposite outcome. Routing is now numbered rule (3) of equal weight,
    and it is recoverable: "If you have already read code this session without routing,
    route now."
  • A reworded hook silently stopped receiving updates. HOOK_SIG was
    "sota standing rules:" — the first three words of the message — so a user who
    reworded the opening un-managed their own hook, and --update would then add a second
    hook
    rather than refresh it. Verified against a real install with the exact jq
    install.sh runs. The marker is now sota-* skills, a phrase present in every
    version ever shipped and in hand-edited variants; tested to match those and to ignore an
    unrelated hook.
  • Three different hook texts existed across README.md, install.sh and installed
    configs. The README now shows exactly what install.sh writes, and documents that
    hand-edits must keep the sota-* skills marker.

Changed

  • skills/sota/SKILL.md reflowed (library map, cross-cutting rules, AUDIT workflow) from
    ~72 to ~98 columns to buy the lines for principles 7 and 8 — content unchanged, 491/500.

Nothing here is measured; no lift is claimed. The repo's one adjacent instrument, the
desc-routing eval, reads +0.00 (saturated) and cannot distinguish these descriptions.

Proving the gate can fail. v1.21.1 shipped two known gaps in our own CI and wrote
them down as ROADMAP #8 and #9. Both are now closed, and the second one justified
itself within a minute of first running.

Added

  • Invariant 15 — the router's library map lists every rules file, both
    directions.
    Invariant 7 proves every skill is in the map; invariant 10 proves
    every rules file is indexed by its own SKILL.md. Neither reads the map's
    contents, which is how sota-code-security/rules/11 sat unlisted in
    skills/sota/SKILL.md from v1.19.8 to v1.21.0 with all fourteen checks green.
    The check compares the NN numbers the map enumerates per skill against
    skills/<skill>/rules/NN-*.md and reports a file missing from the map and a map
    entry naming a file that does not exist. The number is anchored to a list position
    so a title containing digits cannot be misread — 02 NIST 800-53/800-171 yields
    02, never 80 or 53. Watched to fail first, per the script's own header:
    once by recreating the real defect, once on its inverse, then restored.
  • scripts/check-negative-controls.sh — a negative control for our own gates,
    running in CI as its own job. check-invariants.sh passing proves the tree is
    clean; it has never proved the checks still work, and those two states print
    identically. The harness injects a known-bad per invariant into a disposable git
    worktree and requires the intended check to be the one that complains — a
    non-zero exit for any other reason is a FALSE PASS, not a catch. It runs a
    positive control first (clean copy must pass, else abort rather than report), and
    copies in the working-tree gate and byte-compares it, because a worktree at HEAD
    would otherwise test the committed gate rather than the one being edited.
    Covers invariants 1, 2, 6, 10, 15; states plainly that the diff-, history- and
    release-shaped checks are not covered.

Why the second one earned its keep immediately

On its first run the harness reported a FALSE PASS against its own probe 15.
git clean does not remove staged files, so a fixture added by probe 10 leaked
into the next mutation, which then failed on the file-count check instead of the
check it targeted. A harness that accepted any non-zero exit would have printed
5/5 caught and been wrong about one of them
— precisely the rules/12 §2.1
"instrument that cannot fail" mode, caught in our own instrument by the one
assertion added to catch it. Fixed with git reset --hard (which clears the index)
plus a re-copy and re-compare of the gate, since the reset also reverts it.

Changed

  • Invariant count 14 → 15 across AGENTS.md (table + prose),
    CONTRIBUTING.md (numbered list + a new "proving the gate can still fail"
    section) and docs/MAINTENANCE.md. docs/ROADMAP.md line 426 says "14 checks"
    about verify-setup.sh, a different script, and was deliberately left alone.
  • AGENTS.md's "two gaps in the gates, known and unfixed" paragraph — added one
    release ago — was false as of this change and now records both as closed.
  • ROADMAP #8/#9 struck through with what shipped; docs/CONVENTIONS-LEDGER.md
    moves both candidates from "gateable but not gated" (2 → 0) to gated, and
    records that the doctrine-only candidate is no longer doctrine-only.

Nothing here is measured; no lift is claimed.