Skip to content

v2.2.0 — descriptions that say what they are not for, and a way to measure it

Choose a tag to compare

@cmj0121 cmj0121 released this 16 Sep 22:46
· 45 commits to main since this release
ce0aa0f

Fifteen merges since v2.1.0, fourteen of them pull requests. Nothing here changes what a
plugin does at the surface. What changed is how reliably the right one fires — and whether
that claim can be checked rather than asserted.

Routing

  • Every description now says what its skill is not for (#9, #22). Six were rewritten
    after a blind router measured them misfiring; the rest gained a boundary clause. That is
    why the average description grew from 173 to 204 characters — the trade that bought fewer
    misfires.
  • The description's fourth copy is now checked (#11). The same string is catalogue copy in
    /plugin, marketplace.json, the plugin README.md blockquote and the skill's own
    frontmatter; the blockquote is line-wrapped, so an edit by exact string match used to reach
    three of them and leave the fourth stale.

Measurement

  • make eval drives claude plugin eval over 13 committed cases (#20) — the misses
    from a 3-sample blind-router baseline that scored 367/380. It stays outside make test
    on purpose: each case is a full Claude child session on your own credential, so it wants a
    network and about $5.50, and CI here has no secret to give it.
  • All 19 trigger-eval sets hardened or relabelled (#18, #21, #23, #27). Two hardened
    negatives were themselves caught by the router run and resolved in opposite directions —
    one a mislabel, one a description finding that was withdrawn rather than kept as a query
    engineered to pass.
  • What the two runners actually measure is now recorded (#26). The blind router is a
    forced choice among 19 descriptions; make eval observes whether the Skill tool was
    invoked at all. So a blind-router miss is a hypothesis and a make eval miss is an
    observation — and a skill should not fire when the assistant can simply comply in the
    current turn.

Skill bodies

  • Every phase states what it does when its precondition fails (#19), as one shared
    paragraph per skill rather than 76 separate stop conditions.
  • references/ is in use for the first time (#12), by agent-smith's Phase 5 — Autonomous
    mode only, so a Partner-mode dispatch never pays for its scoring table.
  • agent-smith Phase 2 and 2.5 have stated conditions instead of (if needed), with a
    tie-break for when both hold.

Release mechanics

  • The version-bump rule is settled and written down (#17): a bump tracks what a plugin
    does. Catalogue copy alone moves the top-level version, and a plugin moves at most once
    per release window.

Seventeen plugins move to 2.1.0. briefing and shortcut stay at 2.0.0, having no
phases to change.

Full changelog: v2.1.0...v2.2.0