v0.9.0

[0.9.0] - 2026-09-05

Iris explains its verdicts. Every rule result now says what kind of claim it makes, what it saw, where in the raw text it found what it found, and how wrong it tends to be — the published precision and recall carried as an interval on the result, not left on a page. Every evaluation carries a verdict that names which layer decided it, coverage by evaluation question that says what was not judged and why, and provenance that makes the verdict replayable. The server explains itself before an agent lists a tool, returns structured responses and structured errors with the steps that clear them, and serves what it can judge — with its gaps — as iris://capabilities and as the public capability map. The judge is a clear option with one workflow, stated once. And a deleted trace no longer leaves its evaluations' text behind. Two behaviour changes, each a bold sentence below; nothing about which outputs pass has changed.

Changed

• Behaviour change: deleting a trace erases the text of every evaluation linked to it. eval_results.trace_id is ON DELETE SET NULL, so delete_trace and the retention sweep used to leave every linked evaluation behind with output_text verbatim — including whatever no_pii had flagged — orphaned and readable by every query. Both paths now erase the output text, the expected text, the suggestions and the rule messages in the same transaction, before the foreign key can orphan the rows, and stamp erased_at; the verdict, the scores, the criticality and the evidence offsets stay, so history and drift analytics keep working over an erased row. Locked by tests/unit/storage/erasure.test.ts.
• Behaviour change: the retention sweep runs on a timer, not only at boot. A server that ran for a month never swept again, so "older than 30 days is deleted" held only on the day it started. retention.sweepIntervalHours (default 24; 0 keeps only the boot sweep) re-runs the same sweep on a timer that never holds the process open. One function serves both paths (src/retention.ts).

Added

• storage.redact: "critical_spans" (default none) stores each evaluation's output with the spans a critical detector flagged replaced by [REDACTED:<pattern>], so a tool that detects leaks need not keep the leak it found; the evidence offsets still index the text the caller saw, and the option says so.

• Quarantined gating rules are named on the verdict. A custom rule this version could not validate stays on disk and never runs; when its severity is high or critical it is a gate the operator believes is standing and is not, so every evaluation now carries it as coverage.dormant (the tool and the HTTP ingest path alike), list_rules lists it under quarantined[], GET /api/v1/rules/custom returns it beside rules, and a replace: true deploy records what it retired on its rule.deploy audit row (details.replaces).

• The capability map is public, with its gaps. capability-map.json states, cell by cell, what Iris can judge: ten evaluation questions (safe · grounded · complete · on-task · task completed · acted well · cost · better or worse · where and why · trusted) against six subjects (a single output · with input · a trajectory · multi-run · a population · the evaluator itself), each cell has, partial, gap or n/a with one sentence and the evidence behind it — a rule, a tool, a resource, a route, a proof row or a judge template. The truthbase carries it (capabilityMap), docs/capabilities.md renders from it, the site serves it at /capabilities, and llms.txt carries the counts. tests/capability-map-contract.test.ts locks it to the release: every evidence name resolves to something registered, every registered rule, evaluating tool, judge template and resource appears in an answered cell, the sixty ids are the ten questions by the six subjects, no cell names a private path, and for every cell's needs the real engine skips the cell's rules when that input is absent — a has never reads as "judged this" on a call that lacked what the cell needs. A gap is stated as a gap in Iris, never as a claim about anyone else.

• The judge is a clear option with one workflow, stated once. The five enable steps live in src/judge-enablement.json; the runtime imports them (the IRIS_JUDGE_NOT_ENABLED error's recovery, iris://capabilities' howToEnable, the server instructions), the truthbase carries them (llmJudgeTemplates.enable), the two skill files render them as a slot, and tests/judge-enablement-surfaces.test.ts asserts the README and docs/llm-as-judge.md carry the rendered block verbatim — so the step that users get wrong (a key exported in a shell is not passed to the process an MCP client spawns; it belongs in the client config's env block, then restart) is said the same way everywhere. --self-test prints a judge line for the shell it runs in — enabled with the provider name, or not enabled with the variables — before the env scrub, and says that the client passes only what its config lists. GET /api/v1/health reports judge: { enabled, provider } (provider name only, never a key) and mode (real or demo). tests/integration/judge-enablement.test.ts drives the tool, the resource, the health route and the self-test with the environment scrubbed and again with a dummy key and no network.

• The server explains itself before an agent lists a tool. The initialize response now carries server instructions built at boot from this server's runtime state — the rule count and bundles, the effective critical list after eval.criticalRules / eval.nonCriticalRules, the pass threshold, whether a judge key reached the process (and, if not, the one-line way to enable it), the resources, and how to read a verdict — under a 2,600-character ceiling and locked by a test that cross-checks every tool and resource it names against the registrations. Every tool declares an outputSchema and returns the same object as text and as structuredContent, through one serializer that parses the payload through the schema before sending it (a field the schema does not describe fails a test, not a user); responses link what they created as resource_link items (log_trace → the trace; the verdict tools → the evaluation and the trace; list_rules → the proof). Failures inside a tool return a structured envelope — {"error":{"code","message","recovery":[],"retryable",…}} with isError — from a catalogue of eleven codes, every one provoked over a real transport by tests/unit/tools/error-codes.test.ts (the provoked set must equal the catalogue), so IRIS_JUDGE_NOT_ENABLED carries the enable steps in recovery instead of naming a variable and hoping. Resources are registered the way the SDK wants them: iris://traces/{trace_id} and the new iris://evaluations/{id} as resource templates, plus iris://capabilities (what this server can judge, what each rule needs and its published accuracy, the judge state with howToEnable, the citation posture, the dashboard address, the limits, and the tools, resources and prompts registered — served identically at GET /api/v1/capabilities, provider name only, never a key) and iris://proof (the published accuracy per rule with ppvAt, the value of a fire at four prevalences); a missing trace or evaluation is the protocol's resource-not-found error instead of a 200 body with an error key. The nine tool descriptions now render from one template — five fixed headings in order, a Returns heading generated from the output schema, a 450-word cap enforced at registration — in place of nine essays of up to 1,400 words; the frame they each restated travels once, in the instructions. One prompt, evaluate-my-agent, walks a client through log → evaluate → read → explain. list_rules gains proof per built-in rule and quarantined[] (store entries this version could not validate; they do not fire).

• Every evaluation carries its verdict, its coverage and its provenance. evaluate_output, the dashboard's evaluate route and every stored evaluation read back through the dashboard now carry verdict (state pass · fail · unknown; passed; basis — which layer decided: policy_gate for a configured constraint, detector_veto for an effectively critical detection, score_below_threshold, clean, or no_rules when nothing could be judged; by, the rules that decided; risk: null until the risk composer lands), coverage (which of the seven evaluation questions were judged, which were not and the input that was missing, which had no rule in the selected bundles — plus the inputs the call carried) and provenance (the Iris version, a hash of the effective ruleset — name, version, kind, effective criticality, weight — a hash of the effective configuration, the thresholds, the proof corpus version, and when). trace_id is echoed on the response when the evaluation was linked. No verdict changes: verdict.passed equals passed on every call, derived from today's arithmetic (tests/unit/eval/verdict.test.ts). One serializer, src/eval/response.ts, builds the response for the tool and the dashboard route, so the two cannot drift. Migration 007-eval-provenance stores the provenance, the judge's spend (eval_cost_usd, eval_tokens — the tool description had said the spend was kept; the write path stored none of it) and an erased_at stamp for the coming erasure fix. verdict, coverage and critical_skipped are not columns: they are derived on read from the stored rule results and threshold, so rows written before this release read back the same way without a backfill, and a row with no provenance reads back with no verdict — absent, never fabricated. The migration ledger now records which Iris version wrote it, and an older Iris opened on a database migrated by a newer one refuses to start and says which version to upgrade to, instead of reading half a schema.

• Evidence is locatable, never an excerpt. Every fired detection reports evidence[] as offsets into the raw output ({ type: "span", source, start, end, label } — the leak detector can be redacted at the span it found, and the result never repeats the text); every trajectory rule names the call it judged by index ({ type: "toolCall", index, toolName, label }); every measurement carries value ({ stat, unit, value }) and a count evidence entry with the threshold it was held to and where that threshold came from (default · config · rule); a signal that yields no offset yet (the hallucination signals, an obfuscated injection match) is named with a count. Evidence lists are capped at 25 entries. No verdict moves: npm run proof -- --check holds the numbers, and tests/unit/eval/evidence-shape.test.ts asserts on real transcripts that the SSN spans slice to SSN-shaped text, the silent-grep failure names call 0, the loop names every repeated call and its threshold — and, over every positive case of the detection families in the proof corpus, that every span is inside the output, non-empty, and (for no_pii) matches the pattern its label names.

• Every rule result carries its receipt. evaluate_output (and every surface that reads a stored evaluation) now stamps each rule result with kind (the kind of claim: measurement · detection · inference · judgment · policy · verification), role (what the composer did with it — veto for an effectively critical rule, term for one that fed the weighted score), question, classes, ruleVersion, saw (which of the rule's declared inputs the call actually carried), skipClass when it skipped (not_applicable: never asked; defeated or config_invalid: asked and could not answer — a fail-closed gate treats those as unknown), and uncertainty: for a fired detection or inference the published positive predictive value with a 95% credible interval at the stated prior and the corpus provenance (version, release, same-model labelling); for a quiet one the residual miss rate; definition conformance for a measurement; policy for a configured constraint; unmeasured with the reason otherwise. Nothing about the verdict changes. src/eval/response-schema.ts is the one zod object that describes the response, and tests/unit/eval/response-shape.test.ts runs the real handler on a real transcript and asserts 15 of 15 built-ins carry the stamp.

• The published accuracy ships inside the package. npm run proof now writes a third output, src/eval/published-accuracy.ts — the same per-rule counts and intervals as proof/results.json, as a generated module the server imports (the npm package carries dist/ only, so nothing at runtime could read proof/). npm run proof -- --check diffs it in CI. src/eval/accuracy.ts turns a rule's published confusion matrix into what a fire is worth for a given deployment: the positive predictive value at a stated prevalence (the published precision is the value at corpus prevalence, about one half; at one percent the same rule's fire is worth far less), the residual miss rate when a rule did not fire, and a 95% credible interval on both by seeded Monte Carlo over the Beta posteriors of sensitivity and specificity (Jeffreys prior, two thousand draws, memoised). Every number travels with its corpus version, release and labelling (same-model until the blind label lands). No result changes yet: the next release stamps these on every rule result.

• Every built-in rule declares what it is. Each of the fifteen rules now carries kind (the kind of claim it makes: measurement · detection · inference · judgment · policy · verification), mechanism (formula · pattern · heuristic · model · external), needs (the inputs it reads — a rule skips, never passes, when one is absent), question (the evaluation question it answers, from the new registry src/eval/questions.ts), classes (the failure classes a failing result belongs to, from src/eval/failure-classes.ts) and version. Custom-rule types declare policy, their mechanism and their needs. list_rules.built_in[] and GET /api/v1/rules/builtin carry the six fields plus each rule's description; the truthbase carries the roster and the questions (evalRules.roster, evalRules.questions) so the public capability map can render from the registry. Additive: no result, score or verdict changes. Lock: tests/unit/eval/rule-metadata.test.ts — every built-in declares all six in registered vocabulary, the roster surfaces carry them, the generator and the committed truthbase equal the runtime registry.

Supply-chain transparency

• SBOMs: iris-npm-sbom.spdx.json + iris-docker-sbom.spdx.json (attached below). Both are SPDX 2.3 JSON, cover direct + transitive dependencies.
• SBOM signatures: each SBOM has a companion .sigstore.json (Sigstore bundle — signature + cert + Rekor entry) attached to this release. The release workflow runs this exact verification itself before it reports success. Verify with:

  cosign verify-blob \
    --bundle iris-npm-sbom.spdx.json.sigstore.json \
    --certificate-identity-regexp='https://github.com/iris-eval/mcp-server' \
    --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
    iris-npm-sbom.spdx.json
  

• npm provenance: published with --provenance (verifiable via npm audit signatures or on the package page).
• Docker signature: image signed with cosign keyless (Sigstore). Verify with:

  cosign verify ghcr.io/iris-eval/mcp-server:v0.9.0 \
    --certificate-identity-regexp='https://github.com/iris-eval/mcp-server' \
    --certificate-oidc-issuer='https://token.actions.githubusercontent.com'
  

• Build attestation: both the npm SBOM and Docker image manifest carry GitHub-signed build-provenance attestations. Inspect with gh attestation verify or cosign verify-attestation.

What's Changed

• feat(eval): every built-in rule declares kind, mechanism, needs, question, classes and version (arc 1, A1-1) by @irparent in #435
• feat(proof): the published accuracy ships inside the package, with PPV and miss-rate intervals at any prevalence (arc 1, A1-2) by @irparent in #436
• feat(eval): every rule result carries its receipt — kind, role, question, saw, skipClass, uncertainty (arc 1, A1-3) by @irparent in #437
• feat(eval): typed evidence on every built-in — spans by offset, calls by index, measurements with unit and threshold (arc 1, A1-4) by @irparent in #438
• feat(eval): verdict, coverage and provenance on every evaluation; migration 007; one serializer (arc 1, A1-5) by @irparent in #439
• feat(mcp): the agent-native layer and the judge enablement surfaces — instructions, structured responses and errors, resources, capabilities, one description template (arc 1, A1-6) by @irparent in #440
• feat(claims): the capability map, public with its gaps — loader, docs, /capabilities, llms summary, drift-lock (arc 1, A1-7) by @irparent in #441
• feat(storage): erasure on delete, the retention timer, storage.redact, dormant rules on the verdict (arc 1, A1-8) by @irparent in #442
• chore(release): v0.9.0 — Iris explains its verdicts by @irparent in #443

Full Changelog: v0.8.2...v0.9.0