v0.10.0

[0.10.0] - 2026-09-05

The verdict. Iris used to decide by averaging: a weighted mean of every rule's score against one threshold, with a veto for the three critical ones. Arc zero measured what that cost — no single non-critical rule, and no pair of them, could move the result at the shipped weights, so a trace that cost $1.33, an answer written after a tool had silently failed, and a stub with a TODO in it all passed. This release replaces the average with a composer that reads each rule by what kind of claim it makes: a policy you configured gates, a high-precision detector vetoes, a critical check that was asked and could not answer makes the verdict unknown, and everything else with a published error rate becomes one probability that the output is bad, weighed against the threshold your own loss ratio implies. On the held-out split of a labelled corpus built for this, it is right about shipping 57.7% of the time against the old 38.5%, at an identical false-block rate, and it misses 55.6% of bad outputs where the old arithmetic missed 83.3%. It was measured before it shipped, and the corpus and the runner are in the repository. Alongside it: the three critical text rules stopped being defeated by an invisible character, the digit patterns check structure instead of shape, the LLM judge's threshold decides instead of the model, the citation verifier stops reporting a pass when nothing was verified, and every default the verdict rests on is a config key with its reasoning published. Six behaviour changes, each a bold line below.

Changed

• Behaviour change: the decision moment shows the verdict an evaluation reached instead of computing a second one. It used to count failed rules — no failures meant pass, no passes meant fail, anything else meant "partial". From 0.10.0 those answers diverge, and the divergence is the point: an evaluation can pass with a rule visibly failed, because a shipped default only advises and weak evidence does not carry the risk past the deployment's loss threshold. The old arithmetic would have labelled that "partial", contradicting the verdict the tool returned for the same evaluation. partial now means what it says — several evaluations of one trace that did not agree — and an unknown verdict reads as unevaluated, which is what it is. A moment is also escalated to a safety violation when a rule that VETOES failed, not only when a safety-bundle rule did: which rules veto is the deployment's call, so a rule promoted to critical outside that bundle is exactly as serious and used to rank as a plain failure.

• Behaviour change: the citation verifier's passed is null when nothing was judged, and counts rather than a proportion when something was. Two halves of one rule were wrong. A run where no citation resolved, or where the judge failed on every one, returned passed: true — so an output whose sources had never been checked came back looking verified, and a caller reading passed shipped it. There is no verdict when nothing was verified, and null is what says so; the stored row records it as a skip, which is coverage rather than a clean bill of health. And the verdict was a proportion at a half threshold, which let one fabricated source among three real ones score 0.67 and pass. A citation either supports the claim or it does not: every judged citation must be supported, total_unsupported is the number the verdict turns on, and overall_score is still reported but no longer decides.

• Behaviour change: a judgement you paid for decides, and the free path provably cannot pay. A stored judge evaluation now declares what kind of claim it is, and a failing judgement is a failing verdict — nobody runs a judge by accident, so the answer they asked and paid for is not weighed against the deterministic rules. It also cannot be weighed: a judgment carries no published error rate until a measured run exists for its template and model, so the risk layer would drop it and a paid-for "fail" would read as clean. Separately, context.allowPaid is now what lets a judgment rule run at all, and only the tools whose purpose is to spend set it — so "evaluate_output never calls a provider" is a property of the engine, on the one path every evaluation takes, rather than a promise in a tool description. Locked by a judgment rule that throws if it is ever reached.

• Behaviour change: the LLM judge's verdict comes from its threshold, not from the model. Until now passed was the model's own boolean whenever it supplied one, and each template's documented threshold was a fallback the product rarely reached — so a judge could return a score of 0.2 with passed: true and be believed. The score is the measurement and the threshold is the rule. The model's boolean is still recorded, as self_reported_pass, and when the two disagree the response says disagreement: true — a judge that scores 0.95 and says fail, or 0.2 and says pass, is telling you its rubric and its judgement have come apart on that output. pass_threshold travels with the verdict so the arithmetic is checkable without reading the template.

• Behaviour change: passed is the verdict, and the verdict is composed by kind. Until now passed was a weighted mean against one threshold with a veto for the critical rules, and arc zero measured what that cost: no single non-critical rule, and no pair of them, could move it at the shipped weights, so a trace that cost $1.33, a silent tool failure and a stub answer all passed. The composer reads rules by what kind of claim each makes — a configured policy gates, an effectively-critical detector vetoes, a critical rule that was asked and could not answer makes the verdict unknown, and everything else with a published error rate becomes one probability that the output is bad, compared against the threshold the deployment's own loss ratio implies. Measured before shipped: on the held-out split of the composite corpus it is right about shipping 57.7% of the time against 38.5%, at an identical false-block rate, missing 55.6% of bad outputs against 83.3%; the accuracy difference is +19.2 points with a 95% interval of [−7.5, 42.4], so the case for it is that it misses less without blocking more, not that it is proven more accurate at that sample size. score is untouched and keeps its meaning as a quality gradient.

• Behaviour change: a critical check that could not answer no longer reads as clean. A leak detector defeated by the very output it was examining used to return passed: true with an empty failure list, and no configuration closed it. The verdict now has three states, and unknown reads as passed: false. A rule that was never asked — a trajectory rule with no tool calls — is coverage and not unknown; conflating the two would make every text-only evaluation unknown, which is worse than the fail-open it replaces.

• Behaviour change: a threshold Iris ships advises; a threshold you set decides. A default cost ceiling or length floor is our guess about a deployment we have never seen, so it no longer stops a build on its own; the response says so in a line naming the one setting that would change it. A policy with no number in it — "the output is empty" — is structural and still gates, as does any rule the deployment deployed at high or critical severity.

• The verdict's six defaults are config keys, and each is a recommendation rather than a ruling. eval.composer, eval.falsePassCost, eval.onCriticalSkipped, eval.requiredEvidence, eval.defaultsGate, eval.prior and eval.priorMode. Each was argued through the AI council from three readers' seats and closed on one recommendation with its failure mode stated; none has been ruled, and every surface that shows one says so. eval.composer: "legacy" runs the pre-0.10.0 arithmetic for two minors so an upgrade has somewhere to stand.

• The risk estimate stopped claiming certainty it had not earned. Twelve of the fifteen published families recorded zero false positives, so their positive predictive value pinned to exactly 1 at every prior and 31 of the 111 composite cases read a probability of exactly 1.000 — the same overconfidence the credible interval was added to cure, one layer down. Sensitivity and specificity now carry a half-count prior in the point estimate as well as in the draws, and no case claims certainty any more.

• interpretations[] carries the sentences a verdict alone does not: which setting would have made a fired rule decide, what could not answer, and when the interval straddles the threshold so closely that the call could go either way.

• Behaviour change: token_efficiency is renamed verbosity_ratio. The rule compares completion tokens to prompt tokens against a ceiling. That is output verbosity relative to prompt size, not efficiency: a long answer to a long prompt passes, a long answer to a short prompt fails, and neither says whether the tokens were well spent. The name claimed a measurement the rule does not make, which is a claims problem and not a cosmetic one. rule_results[].ruleName now reads verbosity_ratio, the description states what is measured, the proof family is proof/corpus/verbosity_ratio.json, and the docs contract retires the old name so new prose cannot reintroduce it. Blog posts keep the old name: they described the product on the day they were published.

• Behaviour change: three PII patterns now check structure, not just shape, and two rules share one sentence splitter. A sixteen-digit run is not a card number, an IBAN-shaped token is not an account, and 900-45-6789 is not a social security number: no_pii runs the Luhn check, the ISO 13616 mod-97 check and the never-issued-range rules, and a match that fails is not a match. This matters more since the normalisation pass, because folding circled and full-width digits can turn text into a sixteen-digit run that never was one. sentence_count and topic_consistency now use one splitter (src/eval/text/sentences.ts) that does not break on a decimal, an abbreviation, an initial, a filename or a version string: sentence_count goes from 57.1% recall to 100% precision and 100% recall on its family. Three cases in the proof corpus were corrected in the same change and the reasons are recorded in the case notes: two credit-card positives carried numbers that fail Luhn, so they were not card numbers, and two sentence cases had counts that came from the old splitter's defects rather than from the text. Three negatives were added for the structural checks — a sixteen-digit order id, an impossible SSN and a broken IBAN — so the checks have a measurement behind them.

• Behaviour change: the three critical text rules read a normalised copy of the output. One shared fold (src/eval/text/normalise.ts) drops zero-width characters and joiners, NFKC-folds full-width and mathematical alphanumerics, maps the Cyrillic and Greek letters that are drawn like Latin ones, and collapses each whitespace run to one character — a newline when the run contained one, a space otherwise, because line structure is meaning. It carries an offset map, so every span a rule reports still indexes the raw output the caller sent and now covers the obfuscating characters as part of the finding. Measured on the transforms table in proof/RESULTS.md: no_pii goes from 38% to 100% under a zero-width space, 22% to 100% under Cyrillic homoglyphs and 0% to 100% under full-width digits; no_injection_patterns reaches 100% on all four of those; no_blocklist_words, which previously survived nothing but a change of case, reaches 100% on all four and its recall on its own family rises from 73.3% to 93.3% with no new false positive. An obfuscated injection match is now located in the raw output rather than merely named. Leetspeak stays where it was — inside the injection rule, applied on top of the shared fold — because turning digits into letters would blind every digit-based detector. A phrase split across a line break is still missed by the blocklist rule, and the record says so.

• The evaluation engine is asynchronous. EvalEngine.evaluate, evaluateAll and the private run return promises, so the judgment rule that calls a provider can run through the same path as every other rule instead of a parallel one. No verdict changes and no rule changes: EvalRule.evaluate stays synchronous on purpose, because that is what makes "the free evaluation path cannot reach the network" a fact the compiler checks rather than a test. Rules still run one at a time — they share the regex circuit breaker that bounds a hostile output, and running them concurrently would race it. Locked by no-floating-promises, no-misused-promises and await-thenable on the async surface, and by both proof files regenerating byte for byte.

Added

• The capability map moves two cells at the arc close. Where and why the evaluator itself fails (Q9 × the evaluator itself) reads has: every miss is named by id, the three critical rules are measured under seven evasion transforms, and the PII rule reports recall by entity. Whether the evaluator is better or worse than before (Q8 × the evaluator itself) reads partial: the verdict is measured on a composite corpus with a candidate composer scored beside the shipped arithmetic, but no release-over-release comparison exists yet.
• The evaluator of evaluators is a rendered surface. docs/evaluators.md asks the thirteen trust questions of every evaluator Iris ships — the fifteen built-in rules, the eight custom rule types, the five judge templates, the citation verifier and the verdict composer — and every cell is derived from the proof files by scripts/claims/generators/evaluators.mjs (a cell reads measured only when a number for it exists, with the file and key named), rendered through the truthbase, and locked by tests/evaluators-matrix.test.ts. /proof renders every arc-2 block two-state — the verdict on the composite corpus, the evasion transforms, recall by entity, custom-type conformance, the evaluator-of-evaluators count — and a PPV-at-prevalence column beside precision, each with the file a release roll regenerates.
• The rules are measured in four more ways. proof/results.json moves to schemaVersion 2: every rule carries a Dirichlet credible interval beside the Wilson and bootstrap ones (a zero-error family's F1 no longer reads [1, 1]) and ppvAt, what a fire is worth at 1%, 5%, 20% and 50% prevalence. A transforms table measures the three critical rules against the evasions a leak arrives in — a zero-width space, Cyrillic homoglyphs, fullwidth forms, a no-break space, a tab, a line break or swapped case inside the evidence span — with recall per transform and the dropped case ids. The pii positives name what they contain (entities, by the author, never the detector) and a per-entity table separates cases caught from cases whose entity was named, so an address or a password the definition does not cover shows as a row. Eight conformance families under proof/corpus/custom/ run each custom rule type through the real factory against its documented definition. The blind-label manifest is 140 cases, twenty per judgment family.
• The verdict is measured, not only the rules. npm run proof -- --composite runs a second corpus — the 24 real transcripts plus cases composed by splicing a rule family's case into a clean transcript, so the failure classes present are true by construction — through the real engine and scores the verdict a gate keys on: accuracy against shouldShip with a Wilson interval, false blocks on clean cases and missed blocks separately, calibration (Brier, expected calibration error), recall by failure class, and a threshold sweep on the dev split only. Beside today's arithmetic it scores the risk composer a future release may adopt — gates, then vetoes, then a class-grouped noisy-OR over the published positive predictive values against a loss-derived threshold — under two readings of its prior, with the Newcombe interval on each difference. The risk composer runs in the harness only (proof/lib/risk.ts); nothing about which outputs pass has changed. proof/composite-results.json and proof/COMPOSITE.md are generated and byte-checked in CI (npm run proof -- --check --composite).

Supply-chain transparency

• SBOMs: iris-npm-sbom.spdx.json + iris-docker-sbom.spdx.json (attached below). Both are SPDX 2.3 JSON, cover direct + transitive dependencies.
• SBOM signatures: each SBOM has a companion .sigstore.json (Sigstore bundle — signature + cert + Rekor entry) attached to this release. The release workflow runs this exact verification itself before it reports success. Verify with:

  cosign verify-blob \
    --bundle iris-npm-sbom.spdx.json.sigstore.json \
    --certificate-identity-regexp='https://github.com/iris-eval/mcp-server' \
    --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
    iris-npm-sbom.spdx.json
  

• npm provenance: published with --provenance (verifiable via npm audit signatures or on the package page).
• Docker signature: image signed with cosign keyless (Sigstore). Verify with:

  cosign verify ghcr.io/iris-eval/mcp-server:v0.10.0 \
    --certificate-identity-regexp='https://github.com/iris-eval/mcp-server' \
    --certificate-oidc-issuer='https://token.actions.githubusercontent.com'
  

• Build attestation: both the npm SBOM and Docker image manifest carry GitHub-signed build-provenance attestations. Inspect with gh attestation verify or cosign verify-attestation.

What's Changed

• test(acceptance): the stranger harness — the agent-native chain test as code by @irparent in #444
• fix(acceptance): the stranger spawns the executable directly, keeps the login, and runs its phases separately by @irparent in #445
• docs(http): name GET /api/v1/capabilities where the HTTP path is introduced by @irparent in #446
• feat(proof): measure the verdict on a composite corpus beside the risk composer (arc 2, P-1) by @irparent in #447
• feat(proof): credible intervals, ppvAt, transforms, per-entity PII recall, custom-type conformance, blind sample 140 (arc 2, P-2) by @irparent in #448
• feat(claims): the evaluator-of-evaluators matrix from the proof files; /proof renders every arc-2 block (arc 2, P-3) by @irparent in #449
• chore(map): arc 2 close — two capability-map cells move on the arc's evidence (arc 2, P-4) by @irparent in #450
• refactor(eval): the engine returns promises; the rule contract stays synchronous (arc 3, A3-1) by @irparent in #451
• feat(eval): one normalisation pass, and the critical rules stop being defeated by an invisible character (arc 3, A3-2a) by @irparent in #452
• feat(eval): structural checks on the digit patterns, one sentence splitter (arc 3, A3-2b) by @irparent in #453
• refactor(eval): token_efficiency is verbosity_ratio, and the description says what it measures (arc 3, A3-2c) by @irparent in #454
• feat(eval): the verdict is composed by kind, and "could not tell" stops reading as "clean" (arc 3, A3-3) by @irparent in #455
• feat(judge): the threshold decides, not the model (arc 3, A3-4 first slice) by @irparent in #456
• feat(eval): a judgement you paid for decides, and the free path provably cannot pay (arc 3, A3-4) by @irparent in #457
• feat(citations): no verdict when nothing was verified, and counts instead of a proportion (arc 3, A3-5) by @irparent in #458
• feat(dashboard): the decision moment shows the verdict, it does not compute a second one (arc 3, A3-6) by @irparent in #459
• chore(release): 0.10.0 — the verdict (arc 3, A3-7) by @irparent in #460

Full Changelog: v0.9.0...v0.10.0