v0.8.1

[0.8.1] - 2026-09-04

The honest-surfaces patch. Nothing the evaluator does changes in this release; what the surfaces around it say does. A published versioning policy so a reader can predict what an upgrade will do; a reproducible blind-label sample with a review mode a stranger can fill in without reading the code; a nightly that fails when it cannot run instead of reporting a green that meant nothing; a registry publish that waits for npm as long as npm actually takes; and three surfaces that told an agent the wrong thing, corrected together with the lock that stops each from recurring.

Saying what a green means.

• A published versioning policy. VERSIONING.md states what each number promises, so a reader can predict what an upgrade does before running it: while the version starts with 0., the minor is the compatibility lever, and anything an existing caller can observe changing is written here as a bold Behaviour change: sentence. It also names, in public, the two things we got wrong — 0.5.1 shipped two behaviour changes as a patch, and three minors went out inside two days when they were one body of work — and the four conditions that have to hold before 1.0 is declared. CONTRIBUTING.md points at it from the pull-request process.

• A reproducible blind-label sample, so the accuracy numbers can get a human behind them. The corpus is LLM-authored and LLM-labelled, disclosed everywhere a number appears, and that is the biggest caveat on all of them: a model can agree with itself and still be wrong about what a rule means. node proof/blind-sample.mjs draws a fixed-seed, stratified 40-case sample into proof/blind-sample.json — ids only, no labels, safe to hand to an annotator — --check proves the committed manifest is what the seed produces, and --score scores a returned answer sheet and prints every disagreement beside the corpus's own reasoning. The sample covers the seven judgment rules only; the eight arithmetic ones are verified by the test suite, because asking a person to eyeball a percentage of overlapping terms measures their arithmetic, not the rule.

• A review mode that does not require having read the code. The blind-label sample asked reviewers whether a case matched a rule's written definition, thresholds and config keys included. That question is answerable only by someone who has read the implementation, which disqualifies exactly the reviewer whose answer is worth having: a person who can say whether the outputs Iris stays quiet about are outputs anyone would have wanted to hear about. npm run proof:review <sheet> scores the plain-language question instead. It runs every answered case through the real rule registry and reports the disagreements as the product's, not the reviewer's: a miss is something they wanted flagged that Iris passed, a false alarm is something Iris flagged that they would have waved through. There is no answer key in that mode, and the tool says so in its own output.

• The nightly real-LLM smoke fails when it cannot run. With no provider key configured it used to skip and report SUCCESS, on the reasoning that a perpetual red trains reviewers to ignore a check. It has therefore reported success every night while calling no provider at all. That is the same defect this project spent 2026-09-04 removing from the product: a green that means "did not run" is indistinguishable from a green that means "passed". The job now fails and names the one setting that fixes it.

Release plumbing.

• The registry publish waits fifteen minutes for npm, not three. On v0.8.0 npm's read replicas took longer than three minutes to serve the freshly published version, so the registry job failed, verify-release was skipped, and a released version sat off the Official MCP Registry until the job was re-run by hand. The package publish had succeeded the whole time. The window is now fifteen minutes, the poll says how long it waited, and a timeout prints the one command that recovers it. A slow content-delivery network is not a release failure; a wrong verdict about one is.

Surfaces that told an agent the wrong thing. Found by the arc-zero inventory (2026-09-04); each is a fact about a public surface, fixed with the lock that stops it recurring.

• Both skill files told an agent that evaluate_output "defaults to completeness, so safety rules do NOT run unless you ask for them." True until 0.7.0, false since — and the same files said, thirty lines lower, that it defaults to all. An agent reading the table would either set eval_type: "safety" by hand on every call or believe PII and injection went unchecked. The cell now states the shipped default. It survived a release because nothing in the truthbase carried the default, so no scanner pattern could disagree with the prose: .claims.json now carries evalRules.defaultEvalType, read from DEFAULT_EVAL_TYPE in the engine and published in the schema, the hardcoded-claim scanner value-checks every "defaults to " claim against it, and a guard fails on the retired sentence wherever it reappears.
• The dashboard drew a skipped rule as a failed one. The /evals detail and the eval card on the trace page branched on passed alone, and a skipped rule ships with passed: false, score: 0 as placeholders — so "no cost was supplied" wore the same red cross as "an SSN was found", the exact conflation 0.7.0 removed from the API (categories[].passed: null) and the moment page already avoided. The card now tells the three states apart, announces them to screen readers, and shows SKIPPED instead of a score for a rule that did not judge. The dashboard's EvalRuleResult type mirrors the server's, so the fields were there to read.
• The discovery manifests disagreed with the code. server.json — the Official MCP Registry entry — listed three environment variables and none of the four that switch on the LLM judge and the citation verifier, the two tools a registry reader most needs to know how to enable; smithery.yaml described IRIS_PORT as "the HTTP transport or dashboard" port when the dashboard listens on IRIS_DASHBOARD_PORT. Both corrected; a test checks that every variable a manifest names is one the server reads, and that the judge and citation variables are listed.

Supply-chain transparency

• SBOMs: iris-npm-sbom.spdx.json + iris-docker-sbom.spdx.json (attached below). Both are SPDX 2.3 JSON, cover direct + transitive dependencies.
• SBOM signatures: each SBOM has a companion .cosign.bundle (Sigstore bundle — signature + cert + Rekor entry) attached to this release. Verify with:

  cosign verify-blob \
    --bundle iris-npm-sbom.spdx.json.cosign.bundle \
    --certificate-identity-regexp='https://github.com/iris-eval/mcp-server' \
    --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
    iris-npm-sbom.spdx.json
  

• npm provenance: published with --provenance (verifiable via npm audit signatures or on the package page).
• Docker signature: image signed with cosign keyless (Sigstore). Verify with:

  cosign verify ghcr.io/iris-eval/mcp-server:v0.8.1 \
    --certificate-identity-regexp='https://github.com/iris-eval/mcp-server' \
    --certificate-oidc-issuer='https://token.actions.githubusercontent.com'
  

• Build attestation: both the npm SBOM and Docker image manifest carry GitHub-signed build-provenance attestations. Inspect with gh attestation verify or cosign verify-attestation.

What's Changed

• fix(release): a slow npm CDN must not fail the release by @irparent in #424
• chore: versioning policy, blind-label sample, and a nightly that can fail by @irparent in #425
• chore(proof): separate the annotator's verdict from their opinion of the rule by @irparent in #426
• feat(proof): a review mode that does not require having read the code by @irparent in #428
• fix: three surfaces told an agent the wrong thing (skill default, dashboard skipped-as-failed, manifest env vars) by @irparent in #429
• chore(release): v0.8.1 — the honest-surfaces patch by @irparent in #430

Full Changelog: v0.8.0...v0.8.1