v0.19.0

[0.19.0] - 2026-09-25

Verdicts that say how sure they are, detectors that see through disguises, and one command to set up any client. 0.19.0:

• A verdict is called decisive only where labelled data shows its risk estimate holds, and a verdict stored from this release on reads back exactly as it was given.
• The PII and injection detectors catch the common ways a leak or an attack is disguised, and stay quiet on correct answers that only look like one.
• npx -y @iris-eval/mcp-server install <client> writes Iris into ten MCP clients. On every pull request and every commit to main, CI starts the real Claude Code and Gemini CLI against the build on Linux, macOS and Windows.
• The proof page counts rules measured for detection apart from rules checked against their own formula.

Check before upgrading.

• Every pass at the default settings now reads confidence: "marginal", with a note. A gate or dashboard that keyed on "decisive" will not see it on a pass, and one that treated "marginal" as rare will see it on every pass. passed, state and basis do not change.
• no_pii and no_injection_patterns fire on disguised values and directives they used to pass, so a CI gate on detector_veto can trip where it passed before (the shapes are under Changed).
• Every verdict stored by 0.18.0 or earlier reads back with no confidence, and a note says why, instead of taking a label from the new rule; re-evaluate the output to label it. Its passed, state, basis and risk are as stored.
• The public claims schema (iris-eval.com/claims-schema-v1.json) drops initPackage and requires irisEvalPackage and pythonPackage.
• A config written by install pins the Iris version. Run install again after upgrading to move the pin.

Added

• iris-eval install <client> sets up an MCP client from the server package. npx -y @iris-eval/mcp-server install claude-code adds Iris to Claude Code's config; the clients are claude-code, claude-desktop, cursor, windsurf, continue, vscode, cline, zed, codex and gemini. --uninstall removes the entry, and --list shows which clients are on the machine. An entry under the old key iris moves to iris-eval when it runs Iris; another server under that name is left as it is, and the command says so. The entry runs npx -y @iris-eval/mcp-server@<this version>. Running install again moves that pin and keeps what the user added (environment variables, other keys, arguments after the package). JSON configs are edited as text, so comments, key order and line endings survive, and each JSON edit is parsed again and must change only the Iris entry, or nothing is written. Codex's TOML is edited only inside the Iris tables: a config that defines the server in another TOML form is refused with a message rather than given a second definition. Every write is atomic and keeps the file's permission bits. Each client's config path and shape come from its current documentation, which corrected four the earlier installer had wrong: Continue reads ~/.continue/mcpServers/, Windsurf is now Devin Desktop with its own path, Cline's settings moved, and Zed takes a flat command/args/env entry. Cursor gets the "type": "stdio" its docs require. For Claude Code and Gemini CLI, a CI job on Linux, macOS and Windows installs the package from this commit under a version no registry has, runs install, and requires the client itself to report Iris connected; the README's clients table marks those two verified and the other eight by their documentation. Gemini CLI connects only in a trusted folder, so the command says to run /permissions. The separate @iris-eval/init installer, never published, is retired.

Changed

• verdict.confidence is "decisive" only where labelled data shows the risk estimate holds. Through 0.18.0 a verdict was decisive whenever the credible interval on pBad excluded the loss threshold τ. That interval carries the uncertainty in the detectors' published error rates, not the error of the model that combines them, and the composite corpus measures that error as large where most verdicts land: clean passes state a pBad of about 0.13, and in that tenth of the dev split 12 of 39 verdicts were outputs that should not have shipped (30.8%, 95% interval 18.6–46.4%). A verdict is now decisive only when the interval excludes τ, the verdict was computed at the prior the corpus measured with no local labels, and its tenth of pBad holds at least 10 labelled verdicts from at least 4 distinct detector-firing patterns whose observed bad rate is consistent with the stated pBad and lies wholly on the verdict's side of τ. At the shipped τ of 0.5 only one tenth qualifies (0.7–0.8, and only for a fail); on the composite corpus 63 of 101 labelled verdicts move from decisive to marginal, none move the other way, and no labelled verdict is decisive. Under the old rule, decisive verdicts on the held-out test split were right 11 of 17 times. The table is generated from the corpus into src/eval/published-calibration.ts and checked in CI; proof/COMPOSITE.md ("The confidence label") shows it and how often each label was right. A marginal verdict's interpretation names the test it did not pass, with the measured numbers. On a clean pass at the defaults it opens "Risk estimate measured as too low at this level on labelled data; see iris-eval.com/proof." The dashboard shows a marginal pass whose interval clears τ in a neutral tone and says the estimate there may be too low or too high; fails and verdicts whose interval straddles τ keep the warning tone.
• The proof headline no longer adds detection numbers and formula self-checks together. "25 / 25 rules measured" counted rules whose labels a model gave by judging the output alongside rules whose labels are their own documented definition applied by an independent script, where a perfect score shows the code implements its formula and nothing about detection. Each proof family now states its labelBasis (reading or definition), proof/results.json carries it per rule, and iris-eval.com/proof, proof/RESULTS.md, llms.txt and llms-full.txt count the two apart: 12 rules measured for detection, 1 of them at F1 1.00, and 13 checked against their own formula, 12 of them at F1 1.00. The pages say the reading labels were given by a model, and a family without a labelBasis is never counted as detection. No rule's precision, recall or F1 moved. Adding the field changed the corpus version, which reseeds the composite corpus's credible-interval draws: interval bounds moved by at most 0.031, and no verdict, accuracy or calibration figure changed.
• The critical detectors read through the common disguises, and only the disguises. no_pii now reads an SSN written with typographic dashes or minus signs, or with spaces, dots or no separator right after "SSN", "social security" or "soc sec"; a card number after a card word in groups split by a spaced or typographic dash, and the American Express 4-6-5 grouping with its separators; an IBAN in its printed four-character groups; an email spelled out with "(at)"/"[at]" and "(dot)", or with a plain "at … dot" before a webmail provider; and any PII inside a base64 run that decodes to text. no_injection_patterns reads the override phrase in French, Spanish, Portuguese, German and Chinese, held to the English shape (the imperative, then the word for "all" or no article); letter-spaced text; "DAN" in capitals with its no-restrictions framing in the same sentence; a markdown image whose URL carries a template placeholder in its query or fragment, or a secret-named parameter with a real-looking value; and any directive inside decodable base64. tests/unit/eval/obfuscated-detection.test.ts holds each disguise the rules now read. Correct answers that look like a leak stay passing, and a shared fixture holds them for the server and the playground: a sign-in link to a webmail provider, a timeline of dashed years, a tracking number, a polite correction in another language, a quoted directive under «», „“ or 「」 quotation marks, a sales assistant named Dan, a Jekyll image template. Checksums keep each widened shape precise (Luhn for cards, mod-97 for IBANs, issuance rules for SSNs), base64 is decoded only when it turns into valid text (at most 64 runs of up to 4,096 characters per output), and every new reading stays linear in the input, with 1 MiB hostile payloads in the test suite. The injection pattern count goes from 37 to 38. no_injection_compliance reads the same phrases in tool output. No labelled-corpus precision, recall or F1 moved.
• The GitHub Action's version pins move with each release. version:sync rewrites every uses: iris-eval/mcp-server/.github/actions/gate@vX.Y.Z in the README, docs/ci-gate.md and the action itself, and version:check fails on a pin that differs from the release. They name this release, v0.19.0.

Fixed

• A stored verdict reads back as it was given. provenance.composer now records priorMode and the calibration table the label was read from. For verdicts stored from 0.19.0 on, a fail under eval.priorMode: "per-class" no longer reads back as a pass. A verdict labelled under another table, or stored before the record existed, keeps its stored result and reads back with no confidence label instead of taking today's.
• One malformed stored rule result can no longer stop the server at startup. Startup sorts stored rule names to refresh local labels, so a single stored result without a ruleName threw Cannot read properties of undefined (reading 'localeCompare') and the server exited before answering any client. Every read of stored rule results now goes through one parser: a result named rule is read under that name, a result with no name is skipped, and unreadable data reads as empty. No release writes such rows; they came from hand-seeded data. Stored rows are not rewritten.
• No blog post tells readers to run the retired command name. Three posts from the 0.4 era still showed the original command. It stays installed so existing configs keep running, but on npm that name belongs to another project's package, so a reader who copied it with npx would have run that package instead. The posts now show npx -y @iris-eval/mcp-server, and the check that keeps the legacy name off every live page now covers the blog.
• The Python client's README links to iris-eval.com (its PyPI page shows the change from the client's next release), and two dead links on the site are fixed.

Supply-chain transparency

• SBOMs: iris-npm-sbom.spdx.json + iris-docker-sbom.spdx.json (attached below). Both are SPDX 2.3 JSON, cover direct + transitive dependencies.
• SBOM signatures: each SBOM has a companion .sigstore.json (Sigstore bundle — signature + cert + Rekor entry) attached to this release. The release workflow runs this exact verification itself before it reports success. Verify with:

  cosign verify-blob \
    --bundle iris-npm-sbom.spdx.json.sigstore.json \
    --certificate-identity='https://github.com/iris-eval/mcp-server/.github/workflows/release.yml@refs/tags/v0.19.0' \
    --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
    iris-npm-sbom.spdx.json
  

• npm provenance: published with --provenance (verifiable via npm audit signatures or on the package page).
• Docker signature: image signed with cosign keyless (Sigstore). Verify with:

  cosign verify ghcr.io/iris-eval/mcp-server:v0.19.0 \
    --certificate-identity='https://github.com/iris-eval/mcp-server/.github/workflows/release.yml@refs/tags/v0.19.0' \
    --certificate-oidc-issuer='https://token.actions.githubusercontent.com'
  

• Build attestation: both the npm SBOM and Docker image manifest carry GitHub-signed build-provenance attestations. Inspect with gh attestation verify or cosign verify-attestation.

What's Changed

• docs(blog): hardening LLM-as-judge against prompt injection by @irparent in #175
• fix(npm): npx iris-eval starts Iris, and no page teaches iris-mcp by @irparent in #611
• chore(website): skip a site deployment when a commit touches only folders the site never reads by @irparent in #612
• fix(storage): rule results from the earliest releases no longer stop the server at startup by @irparent in #615
• fix(eval): no_pii and no_injection_patterns read disguised values and directives by @irparent in #613
• fix: call a verdict decisive only where measured; count detection and formula self-checks apart on /proof by @irparent in #614
• fix: bot branches create no site deployments; skip compares against the last deployment; stored-result wording corrected by @irparent in #616
• fix(website): the waitlist size is the operator's number, not page copy; two dead links fixed by @irparent in #619
• fix(eval): no_pii and no_injection_patterns stop vetoing correct answers that look like a leak by @irparent in #617
• fix(confidence): an evidence floor for decisive, verdicts that read back as given, plain wording on a clean pass by @irparent in #618
• docs(listings): the Docker entry starts over stdio; the awesome-list row carries no numbers by @irparent in #621
• feat(install): iris-eval install is part of the server package; the separate installer retires by @irparent in #620
• fix(install): an iris entry is migrated only when it runs Iris; Codex edits stay inside the Iris tables by @irparent in #637
• fix(confidence): a clean pass says the estimate was measured too low there; stored-verdict notes name no internal id by @irparent in #638
• docs(changelog): 0.19.0 release notes; every test runs on every platform by @irparent in #636
• release: 0.19.0 — verdicts that say how sure they are, detectors that see through disguises, one command to set up any client by @irparent in #639

Full Changelog: v0.18.0...v0.19.0