v0.18.0

[0.18.0] - 2026-09-24

Honest numbers, a lighter footprint, nothing done silently. 0.18.0 finishes the work of the 0.17.0 security review:

• The published accuracy numbers no longer count the real transcripts they are checked against, and the page says plainly what that check is.
• Regression alarms fire about ten times less often when nothing has changed.
• The tool list an agent loads in every session is 58% smaller.
• A verdict says when the rules under it changed while the server ran, and deleting a trace leaves a record the agent can read.
• The website runs only the scripts it issued, and its playground judges with the server's full safety libraries.

Every change carries a test. Every published number that moved is listed under Fixed; no verdict on the composite corpus changed.

Check before upgrading.

• answers_the_ask no longer blocks at the shipped thresholds. A gate that relied on it at the defaults stops blocking: set a keyword_overlap or topic_consistency threshold of your own to keep it gating.
• The health endpoints no longer return trace_count. Read total from the authenticated GET /api/v1/traces?limit=1 instead.
• Stdio refuses a request over security.requestSizeLimit (default 1 MB), as HTTP already did. Send less, or raise the limit.
• On a loopback bind, a request without a Host header gets 403. Browsers, MCP clients, curl and the container health check all send one.
• Verdicts can carry rules_changed after a custom rule is deployed, deleted, enabled or disabled while the server runs. It never changes passed, score or verdict.
• no_pii no longer counts an email address that appears in input. A gate on detector_veto stops tripping on answers that repeat an address the agent was given.
• Several published accuracy numbers moved (see Fixed); the largest is no_silent_tool_failure recall, 68.0% → 63.6%.

Added

• Deleting a trace leaves a record, and the agent can read the record. delete_trace used to remove a trace with no audit entry, so an agent could delete the evidence against its own run and nothing would show that it had. A deletion now appends a trace.delete entry (time, user, trace id) to the same audit log that already records every rule deploy, delete, toggle and update. That log is now an MCP resource, iris://audit (the newest 100 entries, newest first), as well as the dashboard's Audit page, which now shows and filters trace deletions. The agent keeps the power to delete; what it loses is the ability to delete silently.
• A verdict says when the rules under it changed while the server ran. An agent that can deploy, delete, enable or disable a custom rule can shape the rules it is then judged by, and the verdict that followed read exactly like one under the operator's rules. evaluate_output, log_trace with evaluate: true, evaluate_runs, POST /api/v1/traces and OTLP ingest now carry rules_changed: { count, last_change_at, since, audit: "iris://audit" } whenever the deployed rule set changed since the server started. It never changes passed, score or verdict, and it is absent otherwise; the rules the demo seeds at startup count as its starting set. iris://audit and the trace.delete entry delete_trace writes now use the same audit file as the dashboard's Audit page, so a demo never reads or writes the real log.

Changed

• The tool list costs the agent being evaluated 58% less context. Every session of an agent that runs Iris pays for tools/list before it does any work, and that list was 104,567 bytes (about 26,000 tokens) for 12 tools. It is now 44,285 bytes (about 11,000 tokens). Each tool description is capped at 1 KB and says what the tool does, when not to use it, what it returns, its errors and its sibling tools; the long form of each tool (the statistics behind a comparison, the fetch guards, every parameter's full explanation and what every output field means) is served once, on request, as toolGuide in iris://capabilities and GET /api/v1/capabilities. Each tool's outputSchema in the list now names its top-level fields and their types only; every response is still validated against the full schema before it is sent, and the full shape of an evaluation stays published at https://iris-eval.com/response-schema-v1.json. No tool's arguments, behaviour or response changed.
• iris-eval.com runs only the scripts it issued. The site's Content Security Policy allowed inline scripts, so any bug that let text reach a page as HTML could also have run it as script. Every page now carries a fresh per-request nonce on the scripts it renders, with 'strict-dynamic' for the code they load, and adds object-src 'none', base-uri 'self' and form-action 'self'; the API and static files get a policy that runs no script. Pages are now rendered per request instead of prerendered, because a nonce has to be new on every response. The footer's comparison links now list every comparison page, from the same registry as the pages.
• The playground judges with the server's full safety libraries and says which release it runs. It ran a reduced subset of the server's PII, credential and injection patterns, without the obfuscation fold, and labelled itself v0.8.0 nine releases later. It now carries every pattern, both injection tiers and the fold, a test fails when any of them differs from the server's, and the version label comes from the package version rather than a typed string. Its rate limit no longer switches off when the shared store is unreachable or unconfigured: the same 30-per-minute limit then applies per server instance, and a limited response carries Retry-After.
• The files written for language models list what the site and the server actually have. llms.txt listed eight of the fourteen comparison pages, and llms-full.txt described nine of the twelve tools the way they worked several releases ago, gave a Docker command without the dashboard port or the API key the image requires, and said Node.js 20 was enough after 0.17.0 moved the floor to 22.13. Both now render their comparison list from the registry the comparison pages are built from, their tool list from the summaries the server sends on tools/list with each tool's long form (what it does, when another call is better, the errors it returns) from the same guide the server serves in iris://capabilities, and the Node floor from engines, and a test fails when either drifts. Every install snippet in llms.txt, llms-full.txt and /.well-known/mcp.json now reads npx -y @iris-eval/mcp-server@<current release> (and the Docker image :v<current release>), so a copied config keeps running the version it was copied for; the README's client configs gain the same -y, which a client that starts the server needs because it has no terminal to answer npx's install prompt.
• answers_the_ask advises at the shipped thresholds, and names refusals and copies of the ask. At the defaults it failed 6 of 10 correct paraphrased answers, because they reused few of the ask's words, and it passed every refusal (too short to measure) and every copy of the ask (all of its words). It now gates only once you set a keyword_overlap or topic_consistency threshold of your own. A bare refusal and the ask handed back each fire it directly, before the brevity skip. A decline that goes on to answer is judged on its content. A gate that relied on this rule blocking at the defaults stops blocking: set a relevance threshold to keep it.
• An inline custom rule can fail a verdict. A custom_rules entry takes the same severity a deployed rule has: high or critical gate, and absent, low or medium advise. Before this, an inline rule could never fail a verdict, whatever it found. The note for an advising custom rule now says to set a severity, instead of blaming "a threshold Iris ships".
• no_pii no longer vetoes an email address the agent was given. A support answer that repeats the returns@ address from the policy text in its input used to fail exactly like a fabricated answer did, so the critical veto could not tell a right answer from a wrong one. An address that appears verbatim (case-insensitively) in input is not counted, and the pass message says so. Email only, deliberately: an SSN or a card number repeated back from a ticket still fails. A gate on detector_veto stops tripping on those answers.
• The health endpoints no longer report trace_count. GET /api/v1/health and the transport's GET /health answer without a key, and the count told any caller how much data the server held. They still run the same storage check, so checks.storage and the 200/503 status mean what they did. A script that read trace_count from health reads total from the authenticated GET /api/v1/traces?limit=1 instead.
• Stdio enforces the same request size limit as HTTP. security.requestSizeLimit (default 1mb) bounded HTTP bodies only, so a 2 MB output was refused over HTTP with 413 and evaluated over stdio. A stdio request over the limit is now answered with JSON-RPC error -32600 naming its size and the setting, and never reaches a tool; the session stays open for the next request. An agent on stdio that sent more than 1 MB in one call gets an error where it got a verdict: send less, or raise the limit. A requestSizeLimit that is not a size ("lots") is now refused at startup by name, where HTTP used to treat it as no limit at all.

Fixed

• On a loopback bind, a request with no Host header is refused. The DNS-rebinding guard on the dashboard API and the HTTP transport checked Host against the loopback allowlist only when the header was present, so an HTTP/1.0 or hand-built request that left it out, or sent it empty, was served. Such a request now gets the same 403 Forbidden: invalid Host header a foreign Host gets. Browsers, MCP clients, curl and the container health check all send Host, so none of them is affected. A client that deliberately omits Host against a loopback server must send it. Binds beyond loopback are unchanged.
• The website says what Iris does, in the numbers Iris measures. The homepage showed "<1ms" eval latency while /proof measured a median near 9 ms, and "0 lines of code to integrate"; several pages said every agent uses Iris automatically, and pricing pointed to a Discord server. The latency stat now reads the median from the published proof, the integration stat says one config block connects, the copy says what happens (every MCP client lists Iris's tools on connect, and Iris scores every run it receives), support points to GitHub Issues and Discussions, and the hero's test count is the full total.
• The published per-rule numbers no longer include the 24 real transcripts, so the transcript line is genuinely held out — and several published numbers moved. 85 labelled cases in 13 rule families were copies of, variants of, or modelled on those transcripts, and the per-rule rates the verdict is composed from were partly estimated on them; the /proof page then called the transcript line "the only out-of-sample one". The cases are gone (each family header lists them), 24 newly authored cases refill the seven families that fell under the 24-case floor, and four composed cases built on transcript-derived family cases now use authored ones. Moved: no_silent_tool_failure recall 68.0% → 63.6% (17/25 → 14/22), F1 0.810 → 0.778; ask_coverage precision 75.0% → 73.3%, recall 85.7% → 84.6%, F1 0.800 → 0.786; grounded_in_reads n 32 → 30; no_tool_loop n 42 → 37; sentence_count n 30 → 24; expected_coverage n 29 → 24; keyword_overlap n 30 → 24; topic_consistency n 31 → 24; min_output_length n 29 → 24; non_empty_output n 29 → 24; no_blocklist_words n 32 → 26; cost_under_threshold n 26 → 24; verbosity_ratio n 25 → 24 — with their intervals (every one in proof/RESULTS.md). No verdict on the composite corpus changed; the composer's probabilities and calibration moved slightly (test-split Brier 0.253 → 0.255 for the risk composer). The transcript line is now described as what it is: 24 scripted runs of one agent, 18 of them bad by design (a 75% base rate), held out of the accuracy numbers but not blind to rule design, and not a sample of production traffic. transcript-results.json gains totals.stagedToFail, and a test fails if a transcript-derived case returns to any family.
• Regression alarms fire far less often when nothing has changed. The drift watcher set each rule's alarm line on its own, so an agent running 25 steady rules raised about 14 false regression_alarms per 1,000 evaluations, most of them soon after the baseline settled. The false-alarm budget is now per agent — about one per 500 of its evaluations however many rules it runs — and the line allows for the baseline being an estimate. On the same simulation the rate is 1.5 per 1,000, and the median first false alarm moved from evaluation 104 to 473. A baseline now also needs a standard error of at most 2.5 points before its rule is watched (about 260 evaluations at a 20% fail rate), and a real 10-point rise is flagged later than before: a median of about 210 evaluations for a 20%-rate rule on a 25-rule agent. Each alarm reports how many streams its line was set for; docs/webhooks.md explains the method.

Supply-chain transparency

• SBOMs: iris-npm-sbom.spdx.json + iris-docker-sbom.spdx.json (attached below). Both are SPDX 2.3 JSON, cover direct + transitive dependencies.
• SBOM signatures: each SBOM has a companion .sigstore.json (Sigstore bundle — signature + cert + Rekor entry) attached to this release. The release workflow runs this exact verification itself before it reports success. Verify with:

  cosign verify-blob \
    --bundle iris-npm-sbom.spdx.json.sigstore.json \
    --certificate-identity='https://github.com/iris-eval/mcp-server/.github/workflows/release.yml@refs/tags/v0.18.0' \
    --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
    iris-npm-sbom.spdx.json
  

• npm provenance: published with --provenance (verifiable via npm audit signatures or on the package page).
• Docker signature: image signed with cosign keyless (Sigstore). Verify with:

  cosign verify ghcr.io/iris-eval/mcp-server:v0.18.0 \
    --certificate-identity='https://github.com/iris-eval/mcp-server/.github/workflows/release.yml@refs/tags/v0.18.0' \
    --certificate-oidc-issuer='https://token.actions.githubusercontent.com'
  

• Build attestation: both the npm SBOM and Docker image manifest carry GitHub-signed build-provenance attestations. Inspect with gh attestation verify or cosign verify-attestation.

What's Changed

• fix(no_pii): an email address the agent was given is not a leak by @irparent in #601
• fix(verdict): answers_the_ask advises at defaults and names refusals; inline rules take a severity by @irparent in #603
• fix(delete_trace): a deletion leaves an audit record the agent can read by @irparent in #604
• perf(tools): cut the tools/list payload from 104,567 to 44,285 bytes by @irparent in #606
• docs: comments, test names and changelog read for outside contributors by @irparent in #608
• fix: loopback Host guard, one request size limit, health without counts, verdicts name rule changes by @irparent in #605
• fix(website): llms files from their sources, the playground on the full safety libraries, a nonce CSP by @irparent in #607
• fix(proof, drift): hold the real transcripts out of every per-rule number; budget regression alarms per agent by @irparent in #609
• fix(website): public copy matches the facts by @irparent in #602
• release: 0.18.0 — honest numbers, a lighter footprint, nothing done silently by @irparent in #610

Full Changelog: v0.17.0...v0.18.0