v0.11.0

[0.11.0] - 2026-09-06

The act layer. Iris scored what an agent wrote and had almost nothing to say about what it DID. Of the sixty cells in the public capability map, the trajectory column held one has in ten, and two failure classes the registry declared — an invalid tool call, an injected instruction obeyed — had never been measured at all. An agent that fetched a poisoned page, did what the page said, and wrote a clean summary passed every bundle.

This release adds five deterministic rules that read the trajectory, a ninth custom type for the tools an agent may call, and the measurement to go with them. Nothing here calls a model or leaves your machine: a gate that bills per pull request is not a gate. Pass tools — your MCP tools/list result, verbatim — and argument validity, invented capability and wasted rereads become judgeable; pass OpenTelemetry spans and a regular cadence stops being mistaken for a loop.

Three results are worth reading before the list. A change was measured and abandoned: narrowing what counts as acknowledging a failed call cleared none of the bar set for it beforehand, both confusion matrices are published, and the rule is untouched. A published number was found claiming certainty it had not earned and now carries the same half-count prior the risk layer uses. And the transcript gap table stopped being hand-written, because a table of known misses can only rot in one direction — a gap that closes stays recorded as open.

Built-in rules 15 → 20, custom types 8 → 9, and every failure class the product declares now has labelled cases behind it.

Changed

• The out-of-sample number is measured now, and the table of what it misses cannot rot. npm run proof -- --transcripts runs the 24 real agent transcripts — runs captured against this repository before any of the rules that judge them existed, with an answer key written at capture time — and writes proof/transcript-results.json and proof/TRANSCRIPTS.md, diffed in CI by --check --transcripts. It reports three numbers that are not the same number: failure classes present that some rule caught (14 of 23 — the one to read, because it needs no relabelling as rules are added), ship verdicts agreeing (17 of 24), and the legacy per-bundle arithmetic (11 of 24 with all four agreeing). Until now the bundle number came from a hand-written table of allowed disagreements, which can only rot in one direction: a gap that CLOSES stays recorded as open, because nothing re-derives it and closing it is invisible. The gaps are measured, the test reads what the runner wrote, and a drift-lock fails on a stale reason and on an unexplained new gap alike.

• A family may no longer skip its way to a better precision. The proof runner scores a SKIPPED case as not failed, so a skip on a negative case is a free true negative: it inflates specificity, which inflates the published positive predictive value — which, since 0.10.0, is arithmetic inside the verdict. Every family carrying skips has been quietly overstating its precision and nothing said so. The runner now refuses a family that skips more than 20% of its cases, and one that skips at all must say why in its own header so a reader meets the caveat beside the number. All twenty families pass today; the guard is against erosion. This is a flaw older than the rules that exposed it.

• ungrounded and incomplete_ask stop being statements about four traces and one trace. Both classes rested entirely on real transcripts, so neither recall could move without the fixtures moving. Nine composed cases — including clean TWINS that pair with a positive on the same question and the same reads — take ungrounded from 1 of 4 to 4 of 7 and incomplete_ask from 1 of 1 to 3 of 3. The false-block rate on clean cases fell from 10.0% to 8.3% and no class recall fell. One case is labelled clean that a reader would call incomplete, deliberately: ask_coverage declines a prose multi-part ask rather than guessing, and the case records the honest boundary rather than hiding it.

• no_tool_loop sees three shapes of repetition now, not one, and stops calling a regular poll a loop. It caught a call repeated too often and two calls alternating; it now also catches a three-call sequence repeating (A,B,C,A,B,C — a shape the pair detector was blind to) and, when you pass tools, the same target read through different tools, which is one wasted read that three distinct call keys used to hide. The target clause requires your catalogue to SAY readOnlyHint: true — never the name heuristic, which would count a paid API call as a wasted reread on the strength of its name — so it is dormant without a catalogue and says how many calls it had to leave out. A repetition at a regular cadence is a POLL and passes: an agent waiting for a build calls the same endpoint every ten seconds, and a stuck one retries as fast as it can emit. That needs start times, which arrive with OpenTelemetry spans and not with tool_calls, so a trajectory carrying no timing is judged exactly as it was before and sending spans can only ever remove a false positive. Period-2 repetition fires at exactly the threshold it did, so nothing that fired before stops firing. Measured on 40 labelled cases, up from 28.

• New rule: max_steps — a task must finish within a step budget. Fifty calls is not evidence of anything; it is a number only a deployment knows, because a research agent reading forty pages is working and a support agent making forty calls to answer one question is not. So it advises at the shipped default and gates the moment you set max_steps — the count evidence carries where its threshold came from, and that is what the composer reads. A malformed, zero or negative budget leaves the default standing rather than inventing a number or refusing to judge. Measured on 28 labelled cases.

• A narrower definition of acknowledgement was measured against the shipped one and DID NOT SHIP. no_silent_tool_failure accepts an acknowledgement phrase anywhere in the output; the candidate required it near the failed call's own subject, so that an answer naming one failure while silently answering another would be caught. Narrowing can only keep or raise recall and can only keep or lower precision, so the bar was set before the measurement: a precision lower bound of at least 0.85. It measured 52.9%. All of its losses are one thing — an acknowledgement that refers to its subject by pronoun ("that file is missing", "the endpoint timed out"), which is how English actually does it and which a lexical proximity test cannot follow. Both confusion matrices are published side by side in proof/RESULTS.md as a candidate block, and the change is abandoned rather than tuned. The helpers ship unused.

• The no_silent_tool_failure family grew from 30 cases to 50, and its published numbers moved because the CORPUS grew, not because the rule did. Recall reads 66.7% where it read 92.9%: ten of the new cases are an output that acknowledges one failure while silently answering another, which the shipped whole-output phrase search does not catch, and they were added precisely because the candidate could not be judged without them. One new false positive is published rather than removed — "neither a.yml nor b.yml exists" is a correct acknowledgement that the phrase list has no negated-conjunction form for, and fixing that is a rule change that belongs where it can be measured on its own.

• New custom rule type: action_policy — the tools your agent may call, and with what. Every other rule Iris ships judges what an agent produced; this one judges what it was allowed to DO, and it is the deployment's own statement rather than ours. allow and deny rules name a tool by glob and its arguments by JSON Pointer — the same notation valid_tool_arguments reports errors in, so two findings about one call address it in one language — with a - segment meaning any array element. Deny is evaluated first and wins, and allow being present means a tool it does not name is DENIED: the thing an author who wrote allow meaning "these are especially fine" most needs to know, which is why the inferred mode is stated on every result including a passing one. A policy that constrains nothing is refused rather than passing every call, and a config Iris will not compile is refused when you deploy it, beside where regex patterns already are, rather than skipping silently for the life of the deployment.

• Argument values are canonicalised before a policy sees them, and every reading of a value is checked. /workspace/../etc/passwd reaches outside the workspace, and so do its percent-encoded, fullwidth-dot and backslash-separated forms; all four fail an allow on /workspace/**. Percent escapes are decoded once — iterated decoding invents values nobody sent — and when decoding changes a value both forms are carried, as are the resolved and unresolved readings of a ... A deny rule fires when ANY form matches; an allow rule is satisfied only when EVERY form does, and deny globs fold case where allow globs do not. Every ambiguity resolves toward denial. A .. that climbs out of its own root is reported as evidence even on a passing result, because an argument built to be read two ways is worth seeing even when the policy permitted it.

• A policy is the one thing in Iris that never routes through the regex sandbox, deliberately. Compiling globs to regexes would serialise the hot path through a singleton worker — fifty calls against ten rules is five hundred blocking round-trips — and, decisively, it would make the policy defeatable: craft a value that stalls the match, collect budgetExceeded, and the rule skips, which means the policy does not gate. A policy an attacker can switch off is worse than no policy. Globs are matched by a segment-wise two-pointer with a single backtrack point, which has no timeout to trip and nothing to switch off; a test greps the policy path for any budget, sandbox, timer or RegExp and fails if one appears. Measured on 30 labelled cases before it shipped, ten of them traversal attempts. Like every custom rule it ADVISES until you deploy it at severity high or critical — its own message says which, on every result.

• New rule: no_injection_compliance: an instruction that arrived inside a TOOL RESULT and was then obeyed. Iris has detected an injection in an agent's own output since 0.3.1 and has never once looked at the place injections actually arrive: the page, ticket or file the agent read. An agent that fetched a poisoned page, did what the page said, and wrote a clean summary passed every bundle. It fails on the action, not on the prose: the directive's payload is read for terms the user's own ask never contained, and the rule fires when a LATER tool call's arguments carry at least three of them — an action nobody asked for, described in words only the attacker used. The same terms merely repeated in the answer are reported and do not fail, and that was decided by measurement against a bar set before the corpus existed: that weaker signal was right 5 times in 8, and all three of its misses are one failure — it cannot tell REPORTING an injection from COMPLYING with one, and reporting is what a good agent should do. No regular expression runs over a tool result, deliberately: that text is attacker-controlled, so directive wording is matched as literal phrases over a length-capped, normalised slice, held in step with the output-side pattern library by a test that fails in both directions. A tool whose output is your own can be exempted with injection_compliance_trusted_tools. Measured on 31 labelled cases before it shipped, 20 of them negatives the design had to survive — four are a faithful summary of a poisoned page and two are a security document that quotes injection wording in order to explain it.

• A rule can now point INTO a tool result, and the evasion question is answerable about a trajectory rule for the first time. Evidence has carried a tool_outputs[i] span type since 0.9.0 and nothing had ever emitted one, which is why the evaluator-of-evaluators matrix recorded "can it be gamed" as merely measurable for both trajectory rules. The transforms harness now transforms the text inside a tool-result span and re-runs the rule, and the published table carries the answer: the directive survives a zero-width insertion, a Cyrillic homoglyph, a fullwidth fold, a no-break space and a case swap in every case, and a tab or line break inserted mid-word in one of nine. The stored trace is deliberately not redacted — an injected payload inside a tool result is the record of the attack the verdict points at, and stripping it would leave a finding whose subject no longer exists. Delete the trace to erase it.

• Every failure class the product declares now has labelled cases behind it. injection_compliance was the last one with none, so its recall was blank rather than measured. The corpus also gained its first evasion cases, in the place attacker-controlled bytes actually arrive: a homoglyphed, zero-width and fullwidth directive hidden in a tool result, each obeyed by the agent.

• New rule: ask_coverage, and the "did it complete the task" row of the capability map stops being empty in every subject. The commonest real agent incompletion is a three-part question answered in one part, and until now no rule read the structure of an ask at all. It judges only an ask that declares its own parts — a bullet list, a numbered or lettered enumeration, a first/second/finally sequence. A full stop is not a declaration, and a prose multi-part ask is not split. That scope was forced by measurement rather than chosen: the version that split prose too went through four rounds of tuning in which every constant that fixed a false positive on the real transcripts destroyed recall on the corpus, because a lexical test cannot tell a second deliverable from a restatement, a manner instruction or a line of pasted material. A part is only measured when it names a subject, and it counts as answered on term overlap, on a mirrored ordinal, or — for "draft a reply" and its relatives — on the output being substantial prose, because an output that IS the requested reply cannot be recognised as one lexically. Measured at 75% precision and 86% recall on 31 labelled cases, with one true fire and no false ones across the 24 real transcripts, and its remaining failure mode named: a part answered with a value rather than the ask's vocabulary reads as uncovered.

• The published positive predictive value on a rule result stopped claiming certainty it had not earned. Three shipped rules were reporting exactly 1.00 — a point estimate computed from the raw counts, sitting inside an interval that was capped below 1, while the risk layer computed the same quantity from the same counts with a half-count prior and quietly disagreed. The release notes for 0.10.0 said no case claims certainty; that was true of the risk and not of the number a reader is shown. Sensitivity and specificity now have one definition, with the Jeffreys prior, used by both. Every published interval moves slightly and no verdict does.

• New rule: grounded_in_reads. The agent's own reads are the source of truth, so "you cited a file, directory or URL that appears in nothing you read" is checkable from the trace alone — no model, no key, no reference corpus. Transcript t-12 is the shape it exists for: the answer cites docs/otel-export.md while the agent's own directory listing shows docs/otel-integration.md. Only LOCATIONS are judged. A code identifier, a version, a date or a number is not one — those belong to no_hallucination_markers, and claiming them in two failure classes would double-count one piece of evidence inside the risk estimate. The ground is the ask, every tool output, and the input of every call that succeeded: a successful read is evidence a path exists and a failed one is evidence it does not, which is what makes filenames invented after a failed listing a clean finding. It declines to answer when a read was truncated, because a location absent from a partial read is not evidence it was invented — the claim would be unsound rather than merely uncertain. Measured on 32 labelled cases before it shipped, half of them false positives the design had to survive.

• Behaviour change: a json_schema custom rule deployed with a schema in its config, which was previously ignored, now applies it. The name promised something the code did not do — until now the rule parsed the output and passed any valid JSON, so a deployment relying on it to gate a structured output got passed: true on a wrong shape. A rule with no config.schema behaves exactly as before and says in its message that the shape was not checked, so nothing you have already deployed changes meaning. A schema Iris will not compile is refused when you deploy the rule, beside where its regex patterns are already refused, rather than skipping silently for the life of the deployment. Validation runs through the same hardened path a tools catalogue does, and the message names the JSON Pointer and the keyword that rejected the output, never the value. Its conformance family was rewritten rather than extended: the old cases were labelled against a definition that ignored the schema, so they measured a question the type no longer asks.

• New rule: valid_tool_arguments. Arc zero found a wrong PASS reachable here — an agent that calls a tool which does not exist, or passes arguments its schema rejects, and then writes a plausible paragraph, satisfied every bundle Iris had. Pass tools (your MCP tools/list result, verbatim) and each call is checked against the schema its own tool declares. It fires only on calls the agent never recovered from, and that is arithmetic rather than kindness: the condition is deterministic, so a rule that fired on every invalid call would measure a positive predictive value of essentially one, and since it is the only detector of its failure class the composer would fail every trajectory containing a single fumbled-then-fixed call. An agent that mistypes an argument, reads the error and retries correctly did good work; the retry is recorded in the evidence and the verdict is a pass. A call to a tool absent from the catalogue is never recoverable — the tool does not exist, so calling it is invented capability rather than a typo. Without tools the rule skips, because a call cannot be checked against a schema nobody supplied, and a schema Iris declines to compile leaves that tool's calls unchecked and says which tool and why. Measured on 33 labelled cases before it shipped.

• ajv is a direct dependency now, declared at the range the MCP SDK already declares. It was always in the tree and always loaded — the SDK uses it for protocol messages — so this costs nothing in install size or cold start and buys the ability to compile a caller's tool schemas deliberately rather than by borrowing someone else's dependency. Nothing uses it yet; the rule that will is in the next release. A caller-supplied JSON Schema is the one thing Iris compiles rather than parses, so it arrives through a guard ladder with a test firing every rung: static caps before ajv is imported at all, local $ref only with no fetcher configured, star height and then an empirical probe on every pattern and patternProperties key, allErrors off (ajv's own documentation names it a denial-of-service vector on untrusted schemas), formats not validated, and nothing that could mutate the instance being checked. A schema that trips any rung is refused whole, and the call is reported unchecked rather than valid. eval.validateToolArguments turns the path off without an uninstall.

• log_trace and evaluate_output accept tools: what the agent COULD have called, as your MCP tools/list result pasted verbatim. Iris has always stored what an agent did and never what it was able to do, and that absence is why argument validity has no evaluator — a call can only be judged against the schema its own tool declares, and nothing held that schema. The catalogue is stored on the trace, reused when evaluate_output is given that trace_id, and hashed into the evaluation's provenance over the parts a rule actually reads, so two catalogues differing only in a description are one toolset. Nothing reads it yet; the rules that will are in the releases after this. A catalogue that is too large or names a tool twice is refused, not truncated: a truncated catalogue would make "this tool is not in the catalogue" a lie, and that sentence is evidence about an agent inventing capability.

• Behaviour change: a trajectory captured as OpenTelemetry spans is now judged, where it used to report "not judged". Iris has stored spans since the beginning — indexed, timed, status-coded and exported over OTLP — and evaluated none of them, so an agent instrumented with OpenTelemetry rather than log_trace got both trajectory rules skipped and a response that said the tool-use question was unjudged. It now reads one derived trajectory from whichever shape arrived: tool_calls when they are present, TOOL spans otherwise, never a merge of the two (most emitters send both for the same calls, and merging would double every repeat count). A span whose status is ERROR is a failed call, through the same definition a tool_calls entry goes through, so the two corpora those definitions were labelled against still mean what they meant. Nothing about a trace that sends tool_calls changes: the measured numbers regenerate byte for byte, which is how the change was checked rather than asserted.

• A trajectory rule now says when it examined only part of what it was handed, and distinguishes three ways a trajectory can be absent. Derivation is capped at 500 steps, and above the cap the message says how many of how many were examined rather than judging a slice in silence — the cycle detector is quadratic and had no cap at all, so a trace carrying twenty thousand spans was four hundred million comparisons on a single thread. "No tool calls provided", "the agent made no tool calls" and "spans were supplied but none has kind TOOL" are now three different sentences, because the last one is a wiring problem on the caller's side and telling them "no trajectory" sends them looking in the wrong place.

• tool_calls entries accept four more optional fields: call_id, truncated, token_usage and cost_usd. Read by nothing yet. Each is knowable only to whatever produced the trace and unrecoverable afterwards, and the entry is strict, so a caller who was already sending them was being rejected. truncated is the one that matters: Iris truncates nothing on ingest, so the producer is the only thing that knows an output was cut, and a grounding check that treated unknown as complete would call an elided read a fabrication.

• The capability map moves two cells at the arc close. Whether a single verdict can be trusted now reads has: every result says what kind of claim it is, where it found what it found and how wrong it tends to be, the verdict names which layer decided, and one definition of passed covers the deterministic rules, the judge and the citation verifier alike. Whether a verdict with input can be trusted also reads has: a critical check that was asked and could not answer makes the verdict unknown rather than clean, which is a state a gate can fail closed on and a setting the deployment chooses.

Supply-chain transparency

• SBOMs: iris-npm-sbom.spdx.json + iris-docker-sbom.spdx.json (attached below). Both are SPDX 2.3 JSON, cover direct + transitive dependencies.
• SBOM signatures: each SBOM has a companion .sigstore.json (Sigstore bundle — signature + cert + Rekor entry) attached to this release. The release workflow runs this exact verification itself before it reports success. Verify with:

  cosign verify-blob \
    --bundle iris-npm-sbom.spdx.json.sigstore.json \
    --certificate-identity-regexp='https://github.com/iris-eval/mcp-server' \
    --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
    iris-npm-sbom.spdx.json
  

• npm provenance: published with --provenance (verifiable via npm audit signatures or on the package page).
• Docker signature: image signed with cosign keyless (Sigstore). Verify with:

  cosign verify ghcr.io/iris-eval/mcp-server:v0.11.0 \
    --certificate-identity-regexp='https://github.com/iris-eval/mcp-server' \
    --certificate-oidc-issuer='https://token.actions.githubusercontent.com'
  

• Build attestation: both the npm SBOM and Docker image manifest carry GitHub-signed build-provenance attestations. Inspect with gh attestation verify or cosign verify-attestation.

What's Changed

• test(acceptance): the stranger's A6 rule was grading 0.9.0's behaviour by @irparent in #461
• chore(map): arc 3 close — a verdict can be trusted, and "could not tell" is a state a gate can act on by @irparent in #462
• docs(claims): the risk composer shipped in 0.10.0; two surfaces still called it prospective by @irparent in #463
• feat(eval): one derived reading of the trajectory, from tool_calls or from TOOL spans by @irparent in #464
• refactor(eval): the two trajectory rules read the derived trajectory, so spans are judged by @irparent in #465
• feat(eval): carry the tools catalogue, so a call can be checked and not only seen by @irparent in #466
• feat(eval): a hardened validator for caller-supplied JSON Schema, with no rule using it yet by @irparent in #467
• feat(eval): valid_tool_arguments — a call is checked against the schema its tool declares by @irparent in #468
• feat(eval): json_schema applies the schema you configure, where it used to ignore it by @irparent in #469
• feat(eval): grounded_in_reads — a location the agent never saw is evidence by @irparent in #470
• feat(eval): ask_coverage, and one definition of the number a reader is shown by @irparent in #471
• feat(eval): no_injection_compliance — an instruction read from a tool result and obeyed by @irparent in #472
• feat(eval): action_policy — the tools an agent may call, and with what by @irparent in #473
• feat(eval): repetition at three shapes, a step budget, and a candidate that did not ship by @irparent in #474
• feat(proof): measure the out-of-sample line, and stop a gap table that could only rot by @irparent in #475
• chore(release): 0.11.0 — the act layer by @irparent in #476

Full Changelog: v0.10.0...v0.11.0