v0.8.2

[0.8.2] - 2026-09-05

The truth patch. Nothing the evaluator does changes in this release; what the surfaces around it say does, and each correction ships with the lock that stops it recurring. Nineteen surfaces stated something untrue — a flag that does not exist, an endpoint that does not exist, a judge score called calibrated while its measurement is pending, four era stamps, a hosted tier that is not being built, latencies nobody measured, a package that is not published, a roadmap describing shipped work as planned, three different recipes for one fail-closed gate, compare pages with wrong counts, a playground that showed a rule that never ran as a pass — and the evaluator-of-evaluators runtime was weaker than its docs: the release workflow told readers to verify signatures it never verified itself, the signature bundles carried a suffix the OpenSSF Scorecard ignores, and the judge measurement went green when it could not run. Every one of those is corrected below, with five new scanner patterns, a docs-contract test, a release-workflow consistency test, one rendered source for the two skill files, and the truthbase carrying two new keys.

Fixed

• The evaluator-of-evaluators runtime does what its docs say. proof-judge.yml fails red when no provider key is configured instead of skipping and reporting green (dispatch-only, so no pull request is affected; the judge is user-keyed, and npm run proof:judge on a checkout with your own key is the other path). The release workflow's verify-release job now runs cosign verify-blob on both SBOM bundles under the workflow's signing identity — the notes had told readers to run it while no job ever did, and presence of a bundle is not a signature that verifies. The bundles are named *.sigstore.json instead of *.cosign.bundle: same Sigstore bundle bytes, but OpenSSF Scorecard's Signed-Releases check only counts suffixes it recognises, so every signed release had scored 0 there. SECURITY-EXPOSURE.md's hono row said 1.19.13 installed a month after the MCP SDK bump had put 2.1.0 in the tree and closed the advisory; the row is corrected and the security-exposure gate now checks every "installed" claim in the record against package-lock.json. SECURITY.md states the review posture (one maintainer; the required checks and CodeQL are the review; no second human reviewer today). Lock: tests/release-workflow-consistency.test.ts pins the bundle suffix at the sign, notes, upload and verify sites, the presence of the verify-blob step, and the fail-loud branch in both keyed workflows.
• The playground no longer shows a rule that did not run as a pass. The vendored rule library now mirrors the server's skipped / skipReason: a rule with nothing to judge (no input, no cost, no tool calls, an output too brief for topic analysis) renders as skipped with the reason, is excluded from the "x of y judged rules pass" tally and from the score, and never counts as a pass. The category picker's rule counts render from the vendored registry instead of hand-typed numbers (they said 4/3/4/2 against a 15-rule roster). tests/playground-parity.test.ts now asserts skip parity with the installed server instead of counting a server skip as a pass, with fixed cases that omit each context so every skip path is exercised.
• The two skill files are one rendered source. skills/iris-eval/SKILL.md (the npm package) and claude-plugin/skills/agent-eval/SKILL.md (the plugin marketplace) had been mirrored by hand with an "edit both together" comment and had drifted — three sections and a config row in one and not the other. Both now render from skills/iris-eval/SKILL.template.md through scripts/claims/render-llms.mjs (counts and template names from the truthbase; the front matter, one install-context paragraph and the example-link base are the only per-target facts), and npm run llms:check fails CI when either drifts.
• The docs contract. A new test (tests/docs-contract.test.ts) extracts what the code defines — the CLI flags src/index.ts parses, the /api/v1 routes the dashboard router registers, the iris:// resources the MCP server registers, the IRIS_* variables src/ reads, the rule roster — and asserts every mention on a prose surface (README, docs/, both skill files, server.json, smithery.yaml, the tool descriptions) names something that exists; the reverse holds for environment variables, so server.json now lists every variable the server reads (IRIS_TRANSPORT, IRIS_HOST, IRIS_ALLOWED_ORIGINS, IRIS_DASHBOARD_HOST, IRIS_OTEL_HEADERS, IRIS_OTEL_SERVICE_NAME, IRIS_OTEL_TIMEOUT_MS were missing). Each extractor guards its own regex with a floor, so a pattern that stops matching fails instead of passing vacuously.
• Surfaces that stated something untrue, corrected together with the lock that stops each from recurring. No evaluator behaviour changes. In the MCP tool descriptions an agent reads instead of the README: delete_trace named a --retention-days flag that does not exist (retention is retention.days in config.json); get_traces pointed at a dashboard event-stream endpoint that does not exist; evaluate_with_llm_judge called its score "calibrated" while its measurement is pending (the description now says so and links the proof page); four descriptions dated themselves "v0.4" and four described a "Cloud tier" that is not being built; three descriptions and the judge docs quoted latencies nobody measured ("~5-50ms", "<5ms", "~50ms", "1-10 seconds"). server.json now lists IRIS_PORT, IRIS_OTEL_ENDPOINT and IRIS_CITATION_DOMAINS, which the server reads. examples/README.md said the LangChain package ships; it is unpublished and the note now says so and points at HTTP ingest. docs/roadmap.md described shipped work (the proof page, the trajectory rules, HTTP ingest) as planned; every item is now marked shipped or not shipped. The README, docs/custom-rules.md, docs/api-reference.md and docs/architecture.md gave three different recipes for a fail-closed gate; they now give one (a non-empty critical_skipped is unknown, not clean) and present the default veto set as a default, not a constant. Six compare pages said "4 custom-rule types" against eight, one said "13 deterministic rules" against fifteen, and eight quoted "(<1 ms)"; the counts now render from the truthbase (evalRules.customRuleTypeCount is a new key, read from the CustomRuleType union) and the latency claims are gone. Two learn pages (Output Quality Score, Self-Calibrating Eval) now state precisely what Iris computes and does not. Locks: five new scanner patterns (custom-rule-type-count, latency-claim-without-measurement, retired-cloud-tier, era-stamp-in-tool-description, and "N deterministic rules" as a spelling of the rule count), the measurement-claim pattern extended to the tool descriptions, and a drift test anchoring the custom-rule-type count to the runtime switch.

Supply-chain transparency

• SBOMs: iris-npm-sbom.spdx.json + iris-docker-sbom.spdx.json (attached below). Both are SPDX 2.3 JSON, cover direct + transitive dependencies.
• SBOM signatures: each SBOM has a companion .sigstore.json (Sigstore bundle — signature + cert + Rekor entry) attached to this release. The release workflow runs this exact verification itself before it reports success. Verify with:

  cosign verify-blob \
    --bundle iris-npm-sbom.spdx.json.sigstore.json \
    --certificate-identity-regexp='https://github.com/iris-eval/mcp-server' \
    --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
    iris-npm-sbom.spdx.json
  

• npm provenance: published with --provenance (verifiable via npm audit signatures or on the package page).
• Docker signature: image signed with cosign keyless (Sigstore). Verify with:

  cosign verify ghcr.io/iris-eval/mcp-server:v0.8.2 \
    --certificate-identity-regexp='https://github.com/iris-eval/mcp-server' \
    --certificate-oidc-issuer='https://token.actions.githubusercontent.com'
  

• Build attestation: both the npm SBOM and Docker image manifest carry GitHub-signed build-provenance attestations. Inspect with gh attestation verify or cosign verify-attestation.

What's Changed

• docs(claims): the untrue surfaces, corrected with their locks (truth patch 2, part 1) by @irparent in #431
• fix(claims): one rendered source for the skill files, the docs contract, and a playground that skips (truth patch 2, part 2) by @irparent in #432
• ci(release): verify-release verifies the SBOM signatures; bundles named for Scorecard; proof-judge fails loudly without a key (truth patch 2, part 3) by @irparent in #433
• chore(release): v0.8.2 — the truth patch by @irparent in #434

Full Changelog: v0.8.1...v0.8.2