v2.2.0 — Measured Registry Hygiene
Everything in this release traces to a measurement. An 864-call tool-selection
study (31B + 4B tiers, random and adversarial distractor arms) found no
count degradation at 33 tools — the failures that did exist were textual.
So this release fixes text, hardens the measuring instrument, and installs
gates so the registry stays healthy as it grows. The planned Tool-RAG build
was cancelled by its own feasibility data.
Highlights
get_security_policies→get_cisco_policies: the registry's worst
name collision (0.794 embedding similarity withget_firewall_policies,
both descriptions claiming Cisco ACLs) split on definitions-vs-verdicts.
Re-measured at 0.547.- Discriminability gate: new test fails any NEW tool pair above 0.70
similarity; the five known overlaps are explicit, documented debt. - Eval: all 33 tools asserted (25 → 40 questions), including
honest-negative questions — the assistant must say "none" when the network
has none. - The eval cannot lie: load-time catalogue validation (typo'd assertions
used to pass forever),requires:preconditions that SKIP (never FAIL)
when a feature is absent from the loaded run, and a baseline run-id guard
that stops cross-network diffs from reporting phantom regressions. llm/ollama.py→llm/openai_compat.py: one provider per wire
format, not per vendor.OLLAMA_*env vars unchanged (back-compat).- Version-drift guard: the declared version can no longer silently lag
the newest tag (it shipped stale twice).
Breaking change
The MCP tool get_security_policies is now get_cisco_policies. Clients
discovering tools via tools/list adapt automatically; hardcoded names must
update. Everything else is backwards compatible.
Full suite: 1440 passed. Live eval: 36 pass / 0 fail / 4 honest
skips, four identical consecutive runs.