v0.5.0

Breaking changes — check before upgrading

Added 2026-09-03. v0.5.0 shipped two deliberate behaviour changes that this page did not mention: the critical-rule veto (passed is now vetoed by a failing critical rule, and already-deployed high/critical custom rules become hard vetoes at next boot) and the dashboard no longer starting implicitly with --transport http. The text below is verbatim from CHANGELOG.md § 0.5.0.

Check before upgrading: two deliberate breaking changes are detailed below — the critical-rule veto (pipelines seeing passed: true on violating output will start seeing passed: false) and the dashboard no longer starting implicitly with --transport http (pass --dashboard for the ingest endpoint). Audit list_rules for high/critical severities that were labels rather than gates.

Changed

• BREAKING (deliberately): a detected PII leak, prompt injection, or blocklist hit now fails the eval. evaluate_output averaged its rules, so a genuine violation was routinely outvoted: an output containing a real SSN failed no_pii while the other safety rules passed, scored ~0.765 — above the 0.7 threshold — and reported passed: true. That is the one field every CI gate keys on, waving through the product's flagship failure scenario; a detection that reports an all-clear is worse than no detection. Rules can now be critical: a failing (non-skipped) critical rule forces passed: false regardless of the weighted score, and the response lists the culprits in a new critical_failures field. no_pii, no_injection_patterns, and no_blocklist_words are critical; no_stub_output and no_hallucination_markers deliberately are not (quality gradients and calibrated heuristics — they lower the score and stay visible in rule_results without vetoing). The numeric score is unchanged: score and passed answer different questions, and only passed was lying. If your pipeline currently sees passed: true on output containing PII, injection, or blocklisted phrases, it will start seeing passed: false. That is the fix, not a regression.
• deploy_rule severity now means something. A custom rule deployed with severity: "critical" could FAIL on a violating output and the eval still returned passed: true (score 0.895) — severity affected nothing but dashboard sorting. Deployed rules with severity high/critical are now hard-failing (same mechanism as the critical safety rules), on every registration path: MCP deploy_rule, the dashboard composer, and boot-time loading from custom-rules.json. low/medium stay weight-only. The deploy_rule description now states exactly what severity does. Upgrade warning: the OLD description said severity "does NOT affect scoring", so you may have chosen high/critical as a label with no behavioral intent. Every already-deployed high/critical rule in ~/.iris/custom-rules.json becomes a hard veto at next boot — audit list_rules before upgrading and redeploy as medium any rule where the severity was a label rather than a gate.
• The dashboard no longer starts implicitly with --transport http. Behaviour change — check this before upgrading. It started whenever the HTTP transport did, which contradicted the README (--dashboard, default false), put a second unauthenticated server on port 6920 beside a transport that had been deliberately bound to loopback, and — the failure a persona UAT actually hit — took the whole process down with exit(1) when 6920 was already busy, after the transport had bound successfully. A server nobody asked for must never be able to kill the one that was requested. The dashboard now starts only when explicitly enabled. If you relied on POST /api/v1/traces riding along with --transport http, you must now pass --dashboard (or set IRIS_DASHBOARD=true) — HTTP ingest is served by the dashboard, not by the MCP transport. Starting the HTTP transport without it logs that in full, so the change arrives as an instruction rather than as a connection refused.

---

Supply-chain transparency

• SBOMs: iris-npm-sbom.spdx.json + iris-docker-sbom.spdx.json (attached below). Both are SPDX 2.3 JSON, cover direct + transitive dependencies.
• SBOM signatures: each SBOM has a companion .cosign.bundle (Sigstore bundle — signature + cert + Rekor entry) attached to this release. Verify with:

  cosign verify-blob \
    --bundle iris-npm-sbom.spdx.json.cosign.bundle \
    --certificate-identity-regexp='https://github.com/iris-eval/mcp-server' \
    --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
    iris-npm-sbom.spdx.json
  

• npm provenance: published with --provenance (verifiable via npm audit signatures or on the package page).
• Docker signature: image signed with cosign keyless (Sigstore). Verify with:

  cosign verify ghcr.io/iris-eval/mcp-server:v0.5.0 \
    --certificate-identity-regexp='https://github.com/iris-eval/mcp-server' \
    --certificate-oidc-issuer='https://token.actions.githubusercontent.com'
  

• Build attestation: both the npm SBOM and Docker image manifest carry GitHub-signed build-provenance attestations. Inspect with gh attestation verify or cosign verify-attestation.

What's Changed

• fix(website): pricing page claimed limits, features and compliance that do not exist by @irparent in #338
• docs(roadmap): replace the Cloud-tier ladder with three evidence-led tracks by @irparent in #339
• docs: consolidate every doc onto the roadmap; retire the Cloud-tier ladder repo-wide by @irparent in #340
• fix(dashboard): every "vs prior period" delta has never worked by @irparent in #341
• feat: --demo first-run experience (seeded demo dashboard, hard-isolated from the real store) by @irparent in #342
• feat: --self-test — a cold diagnostic that proves the install works by @irparent in #343
• feat: POST /api/v1/traces — deterministic capture over HTTP by @irparent in #344
• feat(dashboard): land on the failure + elite-feel pass by @irparent in #345
• fix(dashboard): first-run empty state points at --demo now that it ships by @irparent in #347
• fix(dashboard): null costUsd/latencyMs white-screened the landing view by @irparent in #348
• fix(security): owner-only data files + close SSRF encoding bypasses by @irparent in #349
• ci: packages/init gets PR CI + Dependabot coverage (#335) by @irparent in #350
• fix(claims): platform-dependent test counts turned the truthbase gate red on main by @irparent in #356
• fix(dashboard): path disclosure, self-hosted fonts, and the Firefox flake root cause (#334) by @irparent in #357
• fix(tools): make four MCP tool descriptions true (#332) by @irparent in #358
• fix(eval): citation denominator, unreachable 422, and the stats population split (#333) by @irparent in #359
• fix(eval): repair the three safety-family rules against a labeled corpus by @irparent in #360
• fix(eval): hallucination detection never fired — rewrite as context-grounded signals by @irparent in #361
• fix(eval): no_pii must flag the canonical documentation SSN by @irparent in #362
• test(uat): promote the acceptance harness into the repo by @irparent in #363
• fix(dashboard): a mistyped API route returned 200 + the SPA's HTML by @irparent in #364
• fix(eval)!: a detected safety violation now fails the eval by @irparent in #365
• fix(transport): bind failures reject loudly; the dashboard starts only when asked by @irparent in #367
• fix(tools): unknown tool arguments are rejected, not silently dropped by @irparent in #368
• fix(security): user regex runs under a hard deadline in a sandbox worker by @irparent in #366
• docs(readme): quickstart reaches a first value moment in 60 seconds — FOUNDER REVIEW: new H1 by @irparent in #346
• release: v0.5.0 — the acceptance-test release by @irparent in #378

Full Changelog: v0.4.6...v0.5.0