v0.15.0

[0.15.0] - 2026-09-21

Reach, honestly. Iris reaches further in 0.15.0 — the engine and a client are importable, traces arrive over OTLP, rules can be plugged in, a dataset can gate a release, Node's built-in SQLite carries the store when the native addon cannot — and every reach is stated at exactly its verified width: every client Iris names has a row that says whether it was verified or claimed, every cell on a comparison page names the page it was read from and the date, and the proof's two oldest evasions are measured moving rather than declared fixed.

Check before upgrading.

• The configuration is strict: an unknown or misspelled key refuses startup and names it (with the nearest known key). A config that loaded on 0.14.0 with a typo will not load on 0.15.0 until the typo is fixed — the refusal says which key.
• GET /health on both ports answers one shape: { status, version, uptime_seconds, driver, checks: { storage, rules_store, migrations }, trace_count, storage, judge, mode }, and returns 503 when any check fails. A monitor that parsed the old body must read status and checks.
• When the native SQLite module cannot load, Iris now falls back to Node's built-in SQLite on Node 22.13+ with one warning on stderr, instead of failing to start. Set IRIS_SQLITE_DRIVER=native to keep the old failure, or node to choose the built-in on purpose. --self-test and /health name the driver in use.
• The MCP rate limiter can key by API key (security.rateLimit.mcpKeyBy: 'apiKey'); the default stays ip.
• A tripped ingest --fail-on receipt carries spans — the tripping rule's label and offsets, never the text. A parser that rejects unknown receipt fields must accept it.
• Migration 012 adds the datasets and dataset_cases tables on first start; it applies once, beside 001–011, and is recorded like every other.
• The compare pages moved to one route rendered from data; every /compare/<vendor> URL is unchanged, six are new, and their vendor-specific extra rows are gone.

Added

• The deploy-gate walk-through is rows, and the ingest receipt names the span of what tripped. docs/ci-gate.md walks the four steps the person who gates deploys takes — the refusal without a key, eval.requiredEvidence, an action_policy at severity high, ingest --fail-on detector_veto --dataset release-gate — and each is graded on its outcome twice: on every CI run (tests/integration/deploy-gate-walkthrough.test.ts) and against the published package by the stranger harness (--phase gate, rows H-G1 to H-G4). A tripped receipt now carries spans — the tripping rules' span evidence as rule, label, source and offsets, never the text.
• The compare pages are data, and every cell about another product has a source and a date. website/src/lib/compare/<vendor>.json carries the vendor's side of the same twelve features on every page, each cell with the vendor's own page as its source, a verbatim quote and the date it was read; the Iris side lives once in website/src/lib/compare/iris.ts with its counts read from the truthbase. One dynamic page renders all of them; six comparisons are new (Promptfoo, Galileo, Opik, Weave, Judgment Labs, Latitude) beside the eight that existed, whose cells were re-read against their sources. Winner marks are stated on the page as Iris's own call, not a measurement. tests/compare-contract.test.ts locks the files to the schema, the sources and the dates. The compare index carries the one sentence that separates Iris from tools that test MCP servers.
• The release narrative is rendered from the changelog. website/scripts/render-changelog.mjs writes website/src/lib/changelog.generated.json from CHANGELOG.md — the current release whole (lead, paragraphs, every entry) and one line per earlier release — and /releases on the site renders it; the nav banner links there. npm run changelog:check runs in CI beside the other rendered surfaces, and tests/release-narrative.test.ts locks the render to the truthbase's current version and headline.
• docs/contributing-a-rule.md — the path a built-in rule takes from an idea to a shipped, measured, documented rule: eleven steps, each with the file it touches and the test that refuses the PR if it is skipped.
• Every MCP client Iris names has a row — verified or claimed, never just listed. clients.json carries one row per client the installer knows: the status word (verified — a test in the repository drives the client's real integration surface on every CI run; claimed — the installer writes the configuration shape the client documents and that writer is tested on the shape, nobody on the Iris side has watched it connect), the evidence paths, the source read and the date. The truthbase carries the rows (clients), the site renders them at /clients, and the README, the install section and llms.txt render their client lists from the rows; tests/clients-contract.test.ts locks the file to the installer's profiles and to the prose. Today one row is verified (Claude Code, through the capture plugin's hooks) and nine are claimed.
• The engine and a client are importable from the package. package.json gains an exports map: @iris-eval/mcp-server/engine is the evaluation engine with the shipped rules, the composer and the published-accuracy readers — no server, no database, no model — and @iris-eval/mcp-server/client is a typed client for the HTTP route (createClient({ baseUrl, apiKey }).logTrace(body) posts the body every door accepts, sends the Bearer key, returns the same evaluation evaluate_output returns, and throws IrisClientError with the server's own sentence on a refusal). A ./dist/* passthrough keeps existing deep imports working for one minor. The CI build job packs the tree, installs the tarball into a never-used directory and imports both subpaths, evaluating one stub through the engine; the prepack guard refuses a pack without them. The 0.14.0 stranger wrote a script that spawned the CLI to score three files — this is the supported form of that script.
• An unknown config key refuses startup and names it. config.json is validated against a strict schema before it is merged: a key Iris does not read — a typo such as eval.critcalRules, a key from another tool — or a value of the wrong type stops the server with one sentence naming the full key, the key it most likely meant, or the type it wanted. Until now such a key merged in silently and did nothing, so a file could say a deploy gate existed while none did. The two keys Iris writes itself (eval.configuredThresholdKeys, eval.priorConfigured) are refused as reserved.
• One health contract on both ports, and the image checks itself. GET /health on the MCP transport and GET /api/v1/health on the dashboard are built by one function and answer one shape: status, version, uptime_seconds, driver (better-sqlite3), and checks for storage, the deployed-rules file and the migrations (applied against known); status is ok only when every check is, otherwise degraded with 503. The transport's /health used to answer { status, server, timestamp }. The dashboard's health now sits ahead of every rate limiter, as the API reference has said since 0.5.0. The Docker image gains a HEALTHCHECK on that route and the OCI labels (org.opencontainers.image.*) the GHCR package page reads; CI runs the shipped image until it reports healthy and asserts the labels.
• Keys rotate without a gap, and can live in files. IRIS_API_KEY_FILE (or security.apiKeyFile) reads the primary key from a file — the secret-file pattern Docker and Kubernetes mount — and security.apiKeys holds any number of further keys, each with an id, exactly one of keyFile or keyHash (the sha256 hex of the key, so config.json holds no secret), and an optional expiresAt after which the key stops matching at that instant. Every configured key authenticates on the Bearer path and on the dashboard sign-in; the compare is one constant-time walk over the whole ring. Rotation is add, restart, move the clients, remove, restart. An unreadable key file, a malformed hash or a duplicate id refuses startup with a sentence; the startup log names the key ids and warns of any already past its date.
• The MCP rate limit can be counted per API key. security.rateLimit.mcpKeyBy: "apiKey" gives each authenticated key its own per-minute budget (a request no key authenticated falls back to its client address); the default stays per address, and the 429 sentence says which.
• Every evaluation is one structured log line. The server logs {"event":"evaluation", evaluation_id, eval_type, verdict, basis, passed, score, rules_evaluated, rules_skipped, critical_failures, prior_source, duration_ms} at info for every evaluation, whichever door asked (the tools, HTTP ingest, re-evaluation, runs) — never the text, never a key — so a log pipeline can count verdicts without parsing prose. The engine exposes the seam as setObserver.
• Datasets restrict a comparison and a gate. POST /api/v1/datasets promotes a run's case keys — or an explicit list, or cases carrying the answer the reader expects — into a named dataset (migration 012: datasets, dataset_cases); GET /api/v1/datasets[/:idOrLabel] reads them back. compare_runs and POST /api/v1/compare take dataset and pair only the cases in it, saying how many rows each run matched; ingest --fail-on … --dataset <id|label> fails the job only on those cases, and each receipt says whether the trace was in the gate. No new statistics: the same McNemar and Newcombe over a chosen set of cases.
• Five views answer the org reader's weekly questions. GET /api/v1/views/<name> over a closed list — failures_by_rule, cost_by_agent, flaky_cases, unjudged_questions, regression_alarms — each a parameterised read over the existing storage and statistics with a documented row shape; GET /api/v1/views lists them. An unknown name is 404, a parameter outside its range 400; no caller text reaches a query.
• Traces arrive by OTLP. POST /v1/traces on the dashboard port accepts an OTLP/HTTP JSON ExportTraceServiceRequest — the path every OTLP exporter already posts to — and each OTLP trace id becomes one Iris trace with its spans, read from the GenAI semantic conventions (service.name, gen_ai.input.messages / gen_ai.output.messages, gen_ai.usage.*, gen_ai.tool.*); tool spans feed the trajectory rules through the same toSteps. A payload with no GenAI attributes is stored with what it carries and the answer says what it lacked. Evaluation is off by default (otel.evaluateOnIngest); nothing that arrived by OTLP is re-exported.
• A plugin rule with a wrong hash is refused at boot. eval.plugins: [{ path, sha256 }] loads rules you wrote as code — an ES module whose default export is { name, kind, mechanism, version, needs, evaluate(ctx) } — after reading the file and comparing its sha256 to the pinned one; a mismatch, a missing file, a module without the contract or a name that clashes with a built-in refuses startup naming the path. A loaded plugin fires on every evaluation of its type like a deployed custom rule, its result is stamped like a built-in's with origin: "plugin", and list_rules names it under plugins. A plugin that throws or answers the wrong shape skips that evaluation as config_invalid naming itself. The server, the demo and the CLI share one loader.
• The task-completed question has answers on three of its surfaces. ask_coverage (v2) reads the trajectory: a part of the ask the answer never mentions counts as covered when a tool call the trace shows carries its terms — the same lexical test as the answer, so the trajectory can only add coverage (recall-only for the proof family, which regenerates). compare_traces takes question and reads a case's rate for one question — only evaluations that judged it count, an attempt passing when every rule answering it passed. A task_completed judge template (the sixth) scores whether an output's claims of completion are borne out by the ask and, when passed as source_material, the trajectory; user-keyed, its measurement pending. The capability map's task-completed row goes from four gaps to one, and that one names why.
• When the native SQLite module cannot load, Iris falls back to Node's built-in SQLite on Node 22.13+. A driver seam (src/storage/driver.ts) with five verbs sits between the store and SQLite; the adapter, the twelve migrations and the self-test are typed on it, not on better-sqlite3. IRIS_SQLITE_DRIVER=native|node chooses; unset is native with the fallback, which warns once. The built-in is opened with extension loading off and trusted_schema off; the PRAGMA set (busy timeout, WAL, foreign keys, secure delete) is the same on both; Node's own ExperimentalWarning about node:sqlite is printed, not silenced. --self-test and /health name the driver; CI runs the whole suite on the built-in on Node 22 and 24 beside the native runs.

Fixed

• A tab or a line break inserted inside a number, a key or a word no longer hides it from the pattern rules. The transforms table had measured the two evasions for a year: no_pii kept 53% of its catches under an inserted tab and 38% under an inserted line break, no_injection_patterns 44% and 37%, no_blocklist_words 7% and 0%. The shared normaliser now drops a lone tab or break that sits inside one token — between two digits, between a digit and the punctuation a number carries, before a lowercase letter that continues a word, or inside a capital run — for the four pattern rules that ask for it (no_pii, no_blocklist_words, no_injection_patterns, no_injection_compliance); a break before a capital, a digit or a space still ends a line, because line structure is meaning the injection rule reads. Rules that compare tokens between texts keep the old fold: applied to grounded_in_reads it cost that rule 20 points of precision, and the proof said so before it shipped. Every span still indexes the raw text.
• no_silent_tool_failure no longer fails an answer that owns two failures in one clause — "Neither a.yml nor b.yml exists" was the published false positive (silent-044, 0.11.0); a negated-conjunction form (neither … nor … exists|found|present|…) now counts as an acknowledgement, with three corpus cases added first (two negatives of the shape, one "neither option is ideal" that must stay positive). Measured: precision 94.1% → 100%, recall 66.7% → 68.0%, no other rule moved.
• ingest reads every line of an NDJSON file, not only the first. The reader took the first complete line as a trace, then buffered every later line together and parsed them as one document, so a file (or stdin) of three or more traces died on the second with a JSON syntax error — two lines happened to work because the lone second line parsed as the "document". Found by the dataset gate's test, the first to pipe three; a three-trace case now guards it on both --file and stdin.
• The architecture guide's data-flow step no longer states the pre-0.10.0 rule for passed. Step 4 said passed "requires the score to meet the configured threshold AND no critical rule to have failed" — four minors after the composer replaced it, and in the same file whose verdict section describes the composer correctly. It now says what decides passed: the verdict, composed by kind, with the score never consulted. The docs contract holds the retired sentence out and the current one in. (Found by the 0.14.0 distribution audit, which also corrected a "20 rules" comment in the playground's rule library.)
• Two processes opening one cold database at the same instant no longer race on the WAL switch. busy_timeout was set after PRAGMA journal_mode = WAL, so the second process could fail on that very first statement with database is locked and no wait (seen once on a loaded CI runner in the CLI ingest race test, 0.14.0's day). The five-second wait is now on the connection itself (new Database(path, { timeout })) and the pragma runs first; the migration race closed in 0.13.0 with BEGIN IMMEDIATE stands.

Changed

• Every compare quote says whether it was found verbatim on its page. The quotes behind the fourteen comparison pages were extracted through a summarising fetch; each was re-checked against a plain download of its source (tags stripped, whitespace folded) and carries quoteVerified — 125 of 165 found verbatim, 40 not (client-rendered pages or paraphrase), three cells that say the page does not answer carry none. A source link shows its quote only when verified and a cell says "quote unverified" otherwise; the link and the date stand either way, and the page says when the check ran. Patronus AI's tagline and category follow the vendor's own current words (eval models and simulation infrastructure), which its sourced cells already carried.
• The candidate measurement is retired from the proof page. 0.11.0 measured a narrower acknowledgement for no_silent_tool_failure (proximity to the failed call's subject) as a shadow beside the shipped rule, against a bar set beforehand — precision lower bound 0.85 — and it came in at 52.9%, so it did not ship and the block stayed published as a negative result. Its helpers (acknowledgesFailureNear, the proximity window, the subject anchors) and proof/lib/shadow.ts carried no shipped behaviour and are removed; the proof page, proof/results.json and the truthbase no longer carry the shadow block. The result stays in the CHANGELOG, and in docs/contributing-a-rule.md as the model for setting a bar before measuring.
• The support policy is the current minor plus the previous minor for 90 days. SECURITY.md said only the latest minor received security fixes; it now says the previous minor receives them for 90 days after the current minor's first release (0.13.x until 2026-12-19), and the release gate (scripts/check-security-minor.mjs) refuses a table that says otherwise.
• The stranger harness's H rows read the CLI's own receipt: an ingest --evaluate run counts however the agent wrapped the command (the 0.14.0 stranger, refused an env-prefixed command by its session, wrote a ten-line script that spawned npx … ingest --file … --evaluate and ran the script — three correct verdicts, graded as zero because the graders looked for the word ingest in the command's text), and the silent-tool-failure rule counts by its identifier or by its name in prose. Each re-derivation is proven with --regrade against the 0.14.0 run that exposed it.

Supply-chain transparency

• SBOMs: iris-npm-sbom.spdx.json + iris-docker-sbom.spdx.json (attached below). Both are SPDX 2.3 JSON, cover direct + transitive dependencies.
• SBOM signatures: each SBOM has a companion .sigstore.json (Sigstore bundle — signature + cert + Rekor entry) attached to this release. The release workflow runs this exact verification itself before it reports success. Verify with:

  cosign verify-blob \
    --bundle iris-npm-sbom.spdx.json.sigstore.json \
    --certificate-identity-regexp='https://github.com/iris-eval/mcp-server' \
    --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
    iris-npm-sbom.spdx.json
  

• npm provenance: published with --provenance (verifiable via npm audit signatures or on the package page).
• Docker signature: image signed with cosign keyless (Sigstore). Verify with:

  cosign verify ghcr.io/iris-eval/mcp-server:v0.15.0 \
    --certificate-identity-regexp='https://github.com/iris-eval/mcp-server' \
    --certificate-oidc-issuer='https://token.actions.githubusercontent.com'
  

• Build attestation: both the npm SBOM and Docker image manifest carry GitHub-signed build-provenance attestations. Inspect with gh attestation verify or cosign verify-attestation.

What's Changed

• test(stranger): the H rows read the CLI's own receipt, and the rule name in prose (0.14.0 re-derivations) by @irparent in #544
• fix(storage): the busy wait belongs to the connection — set it before the WAL switch by @irparent in #545
• docs: the architecture guide states the composer's rule for passed (distribution audit, 2026-09-20) by @irparent in #546
• feat(exports): the engine and a client are importable from the package by @irparent in #547
• feat(shape): strict config, one health contract on both ports, the image checks itself by @irparent in #548
• feat(keys): keys rotate without a gap and can live in files; per-key MCP rate limit; one log line per evaluation by @irparent in #549
• feat(datasets): datasets restrict a comparison and a gate by @irparent in #550
• feat(views): five views answer the org reader's weekly questions by @irparent in #551
• feat(otel): traces arrive by OTLP by @irparent in #552
• feat(plugins): a plugin rule with a wrong hash is refused at boot by @irparent in #553
• feat(eval): the task-completed question on three surfaces — ask_coverage reads the trajectory, compare_traces takes a question, a task_completed judge template by @irparent in #554
• feat(storage): the SQLite driver seam — when the native module cannot load, Iris falls back to Node's built-in SQLite by @irparent in #555
• feat(clients): every MCP client Iris names has a row — verified or claimed, with its evidence, source and date by @irparent in #556
• feat(docs): contributing a built-in rule end to end; the release narrative rendered from the changelog at /releases (part 1 of 2) by @irparent in #557
• feat(website): the compare pages from data — fourteen vendors, every cell sourced and dated; one sentence on what Iris is not (part 2 of 2) by @irparent in #558
• ci: Bump the github group across 1 directory with 3 updates by @dependabot[bot] in #315
• feat(gate): the deploy-gate walk-through as rows — the stranger's gate phase, the same four outcomes on every CI run, and the ingest receipt names the span of what tripped by @irparent in #559
• docker: Bump node from a0b9bf0 to ebfe2f9 by @dependabot[bot] in #299
• ci: Bump ossf/scorecard-action from 2.4.3 to 2.4.4 by @dependabot[bot] in #329
• ci: Bump anchore/sbom-action from 0.24.0 to 0.24.2 by @dependabot[bot] in #389
• deps: Bump the linting group across 1 directory with 4 updates by @dependabot[bot] in #301
• deps: Bump @types/node from 26.1.1 to 26.6.1 in the types group across 1 directory by @dependabot[bot] in #304
• deps: Bump @playwright/test from 1.61.1 to 1.63.0 by @dependabot[bot] in #310
• deps: Bump express-rate-limit from 8.6.2 to 8.7.0 by @dependabot[bot] in #486
• fix(eval): the two evasions and the false positive — an inserted tab or line break inside a token, and "neither … nor … exists"; corpus cases first, proof regenerated by @irparent in #560
• ci: Bump softprops/action-gh-release from 3.0.1 to 3.0.3 by @dependabot[bot] in #482
• ci: Bump the codeql group across 1 directory with 4 updates by @dependabot[bot] in #385
• deps: Bump the testing group across 1 directory with 2 updates by @dependabot[bot] in #387
• deps: Bump zod from 4.4.3 to 4.6.5 by @dependabot[bot] in #484
• deps: Bump tsx from 4.23.5 to 4.23.13 by @dependabot[bot] in #485
• deps: Bump ajv from 8.18.0 to 8.20.0 by @dependabot[bot] in #487
• chore(proof): retire the A4-12 candidate measurement — the shadow, its helpers, the proof-page block and the truthbase field; the result stays in the record by @irparent in #561
• ci: Bump the docker group across 1 directory with 4 updates by @dependabot[bot] in #538
• deps(website): Bump @radix-ui/react-toggle from 1.1.10 to 1.1.18 in /website by @dependabot[bot] in #309
• fix(website): every compare quote says whether it was found verbatim on its page; Patronus AI's tagline follows the vendor's own words by @irparent in #562
• chore(release): 0.15.0 — Reach, honestly by @irparent in #563

Full Changelog: v0.14.0...v0.15.0