Skip to content

v2.2.0 — Measured Registry Hygiene

Choose a tag to compare

@charlesaspe charlesaspe released this 29 Jul 10:08
2b8b363

Everything in this release traces to a measurement. An 864-call tool-selection
study (31B + 4B tiers, random and adversarial distractor arms) found no
count degradation at 33 tools
— the failures that did exist were textual.
So this release fixes text, hardens the measuring instrument, and installs
gates so the registry stays healthy as it grows. The planned Tool-RAG build
was cancelled by its own feasibility data.

Highlights

  • get_security_policiesget_cisco_policies: the registry's worst
    name collision (0.794 embedding similarity with get_firewall_policies,
    both descriptions claiming Cisco ACLs) split on definitions-vs-verdicts.
    Re-measured at 0.547.
  • Discriminability gate: new test fails any NEW tool pair above 0.70
    similarity; the five known overlaps are explicit, documented debt.
  • Eval: all 33 tools asserted (25 → 40 questions), including
    honest-negative questions — the assistant must say "none" when the network
    has none.
  • The eval cannot lie: load-time catalogue validation (typo'd assertions
    used to pass forever), requires: preconditions that SKIP (never FAIL)
    when a feature is absent from the loaded run, and a baseline run-id guard
    that stops cross-network diffs from reporting phantom regressions.
  • llm/ollama.pyllm/openai_compat.py: one provider per wire
    format, not per vendor. OLLAMA_* env vars unchanged (back-compat).
  • Version-drift guard: the declared version can no longer silently lag
    the newest tag (it shipped stale twice).

Breaking change

The MCP tool get_security_policies is now get_cisco_policies. Clients
discovering tools via tools/list adapt automatically; hardcoded names must
update. Everything else is backwards compatible.

Full suite: 1440 passed. Live eval: 36 pass / 0 fail / 4 honest
skips, four identical consecutive runs.