v0.17.0

[0.17.0] - 2026-09-23

Hostile input, honest verdicts, a harder release path. An adversarial review of 0.16.0 attacked Iris the way a determined user or a compromised dependency would. 0.17.0 closes what it found:

• Outputs that stalled the server for seconds to minutes.
• A crafted number that crashed an evaluation.
• One accent mark that walked past both critical detectors.
• Credentials with no vendor prefix that passed as clean.
• A run comparison that called a 42-point regression "not enough evidence".
• A trace batch that half-committed.
• A release pipeline in which about 860 build dependencies sat beside the npm publishing identity.

Every fix carries a regression test. The published accuracy numbers moved only where a detector got better: no_pii recall rose from 75.6% to 88.9% with the same false positives.

Check before upgrading.

• Node 22.13 or later. Node 20 reached end of life in April and is no longer supported (see Removed).
• no_pii fires on two more shapes: a password inside a scheme://user:password@host URL, and a secret-named key assigned a 12+ character letters-and-digits value. A CI gate on detector_veto can trip where it passed before, and those are real leaks.
• compare_runs answers worse: true on regressions it used to call undecided, because the top line is now one-sided like the per-rule rows. A comparison against a run with no evaluations answers comparable: false.
• POST /v1/traces stores at most 2,000 traces per request. The rest come back in partialSuccess.rejectedSpans. A span repeated within a batch answers 400 with nothing stored.
• A rule that throws becomes a skip naming the error, so the verdict reads unknown instead of the whole evaluation failing.
• The Claude Code plugin, the Cursor plugin and the CI gate action run the version they shipped with. Pass the gate action's version input to run another.

Security

• No output can hold the server hostage any more. An adversarial review timed every built-in rule on long repetitive inputs and found patterns that backtracked from every position of a run. 80,000 digits took 16 seconds to evaluate, 900 KB of DAN took over three minutes, and 5,000 spaces at the end of an ask took more than 15 seconds. Each blocked the one event loop, and with it every other client. A static audit of all 335 patterns in src/ located them. Each one is now linear, with its old and new forms fuzzed against each other on 20,000 random inputs to prove they match the same text, and a regression suite times the worst shapes at 200,000 characters. The published accuracy numbers did not move: proof --check reproduces all three reports byte for byte.

• A rule that throws no longer takes the evaluation down with it. It becomes a skip whose message names the error, so the verdict reads unknown, never passed. The throw it replaced came from a 100,000-digit "number" reaching a runtime-built pattern. Runtime-built patterns now refuse to interpolate tokens longer than 64 characters.

• An accent no longer hides an injection or an SSN. One combining mark (i + U+0301) or a precomposed í inside "ignore", or an acute inside an SSN, passed both critical detectors. The shared text fold now strips the combining accents used on Latin script, whether composed or not. Marks that carry meaning in other scripts (Devanagari vowel signs, Arabic harakat) are untouched. A custom blocklist phrase now takes the same fold as the text it is matched against.

• A release can only ship reviewed code, and the job that holds the npm publishing identity runs nothing but npm.

  • Before, the one job that could publish to npm also ran npm ci twice and the whole build, so any of about 860 dependencies could have used that identity. The build now runs without it and hands a packed tarball to a separate publish job, which publishes it with --ignore-scripts.
  • The release refuses a tag whose commit is not on main, and refuses a manual run on anything but a v* tag.
  • Release tags can no longer be moved or deleted, so a workflow pinned to gate@v0.16.0 always runs what shipped as 0.16.0.
  • The documented signature checks now name the signer exactly: --certificate-identity=…/.github/workflows/release.yml@refs/tags/vX.Y.Z. The old repository-wide pattern accepted a signature from any workflow on any branch.

• The Claude Code plugin, the Cursor plugin and the CI gate action run the version they shipped with. They used to run whatever npm called latest, so a new publish reached every installed plugin and every consumer's CI at once. Each is now pinned to its release, version:sync rolls the pin, and version:check fails when a pin falls behind. The gate action's version input still overrides it.

• no_pii catches credentials that carry no vendor prefix. Every credential pattern keyed on a prefix a vendor published (sk-, ghp_, AKIA), so a bare AWS secret access key, a database URL with its password in it, and a PASSWORD= line all passed as clean, with a "decisive" verdict. Two patterns now read the shape instead:

  • Credential in URL: a password inside scheme://user:password@host.
  • Secret Assignment: a value of 12+ characters with letters and digits, assigned to a name that says it is secret (*_SECRET_KEY, *_TOKEN, *PASSWORD, client_secret, …).

  Documentation stand-ins (${VAR}, <password>, user:password@, your-…, changeme, masked values) are ignored. On the labelled corpus, no_pii recall rises from 75.6% to 88.9% with the same five false positives. The held-out and real-transcript composite accuracy is unchanged.

• The note printed when a failed rule did not decide the verdict gave the wrong advice. It said "Lower eval.falsePassCost to block on weaker evidence". The loss threshold is 1 / (1 + falsePassCost), so lowering it blocks less. It now says Raise and names the formula.

• POST /v1/traces stores a batch whole or not at all, and at most 2,000 traces per request. One 1 MB request of 11,500 one-span traces held the server for 11 seconds, and a span repeated mid-batch returned a 500 after the earlier traces had committed. A batch is now one transaction. A repeated span answers 400 with nothing stored. Traces past the limit come back as partialSuccess.rejectedSpans, the OTLP way to say "resend these".

Fixed

• compare_runs calls a paired regression a regression. The top-line verdict tested two-sided while every per-rule row tested one-sided. So 12/12 → 7/12 on paired cases (five regressed, none recovered) answered worse: false and "not enough evidence": the two-sided exact McNemar cannot go below p = 0.0625 on five discordant pairs. Meanwhile the per-rule row for the same data read worse at q = 0.031. The top line now asks the same one-sided question at α = 0.05, and the summary quotes that p. A CI gate keyed on worse will now trip on regressions it used to let through. An undecided paired comparison no longer advises pairing runs that are already paired. A comparison against a run with no evaluations now answers comparable: false and names the empty run.

Removed

• Node 20 is no longer supported. The minimum is now Node 22.13. Node 20 reached end of life on 2026-04-30 and stopped receiving security fixes then. The floor is 22.13 rather than 22.0 for a reason you can check: 22.13.0 is the first release that ships node:sqlite, so it is the first version on which the storage driver seam added in 0.15.0 actually has a second driver to fall back to when the native addon will not load. On Node 20, and on 22.0 through 22.12, a missing prebuild is still a dead start. engines now says >=22.13.0 in the server and in @iris-eval/init, and the CI matrix tests Node 22 and Node 24. For scale: 22.13.0 shipped 2025-01-06 and the current 22.x line is 22.23.2, so this excludes no maintained runtime.

Changed

• CI stopped hiding its own results, and runs on macOS. The test matrix is fail-fast: false, so one red cell no longer cancels its siblings — a reading that kept Dependabot #308 open for six weeks on the belief that better-sqlite3 13 crashed on three Node versions when it crashes on one. Every runs-on: line in this repository was ubuntu-latest, so the native SQLite addon had never been exercised on Apple Silicon; test (22, macOS) now runs the whole suite there. If you pin required checks for a fork, test (20) is gone and test (24) and test (22, macOS) are new.

Supply-chain transparency

• SBOMs: iris-npm-sbom.spdx.json + iris-docker-sbom.spdx.json (attached below). Both are SPDX 2.3 JSON, cover direct + transitive dependencies.
• SBOM signatures: each SBOM has a companion .sigstore.json (Sigstore bundle — signature + cert + Rekor entry) attached to this release. The release workflow runs this exact verification itself before it reports success. Verify with:

  cosign verify-blob \
    --bundle iris-npm-sbom.spdx.json.sigstore.json \
    --certificate-identity='https://github.com/iris-eval/mcp-server/.github/workflows/release.yml@refs/tags/v0.17.0' \
    --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
    iris-npm-sbom.spdx.json
  

• npm provenance: published with --provenance (verifiable via npm audit signatures or on the package page).
• Docker signature: image signed with cosign keyless (Sigstore). Verify with:

  cosign verify ghcr.io/iris-eval/mcp-server:v0.17.0 \
    --certificate-identity='https://github.com/iris-eval/mcp-server/.github/workflows/release.yml@refs/tags/v0.17.0' \
    --certificate-oidc-issuer='https://token.actions.githubusercontent.com'
  

• Build attestation: both the npm SBOM and Docker image manifest carry GitHub-signed build-provenance attestations. Inspect with gh attestation verify or cosign verify-attestation.

What's Changed

• docs: the three defects the 0.16.0 stranger found (action_policy undocumented, deny's shape, the two doors' field spellings) by @irparent in #590
• ci: the test matrix stops cancelling itself, and runs on macOS for the first time by @irparent in #591
• chore: drop Node 20 — an end-of-life runtime, and the only line with no second SQLite driver by @irparent in #592
• ci: the real-LLM smoke runs on demand, not nightly by @irparent in #593
• fix(security): linear-time rules, a guarded engine, an accent fold, and all-or-nothing OTLP batches by @irparent in #594
• ci(release): only reviewed code ships, and the job with the publishing identity runs nothing but npm by @irparent in #595
• fix(no_pii): catch credentials with no vendor prefix; the falsePassCost note says Raise by @irparent in #596
• chore: the public repo names no private planning surfaces by @irparent in #597
• fix(website): the waitlist neither confirms who is on it nor keeps the IP hash; the export defuses formulas by @irparent in #598
• fix(compare_runs): the top line asks the same one-sided question as the per-rule rows by @irparent in #599
• release: 0.17.0 — hostile input, honest verdicts, a harder release path by @irparent in #600

Full Changelog: v0.16.0...v0.17.0