v0.15.0

0.15.0 (2026-10-27)

⚠ BREAKING CHANGES

• plan: run a compare 4 times a side when runs is unset (#687)
• gateway: end every gateway list with direct(), with direct({ enabled: false }) for strict routing (#676)
• expect: fail the attempt on a caught failed expect (#684)
• spring clean: remove insights, single-source option docs and trim slow or duplicate tests (#667)
• history: fingerprint an eval's body, options, cases and revision (#664)
• updated en grammar to us grammar (e.g. JudgmentRecord, judgment_key, canceled)

Features

• doctor: check vertex() models online, with a Vertex example and recipe (#674) (bc75f37)
• gateway: accept the Anthropic API's dated Claude ids on Vertex AI (#672) (47958ee)
• gateway: add openrouter() routing gateway (#547) (0778bbc)
• gateway: end every gateway list with direct(), with direct({ enabled: false }) for strict routing (#676) (be3c673)
• gateway: run Antigravity, OpenCode, Goose and pi on Vertex AI (#673) (4d0e7c9)
• gateway: run fx on openrouter() (#680) (9c3b3b6)
• gateway: run fx on Vertex AI (#686) (238abff)
• gateway: run Grok Build on Vertex AI (#675) (31a8cd2)
• gateway: run pi on openrouter() (#679) (1830a9b)
• gateway: run pi on vercel() and send effort through gateways where the model reads it (#671) (52280a5)
• gateway: vertex(), with Claude Code, judges and analysis on Vertex AI (#505) (b0c3690)
• history: fingerprint an eval's body, options, cases and revision (#664) (fdb040b)
• plan: hint the compares and families that can't get a verdict (#685) (b478ba9)
• plan: run a compare 4 times a side when runs is unset (#687) (74df7c8)
• query: name a mixed cell below the flaky minimum mixed (#681) (d3a1e78)

Bug Fixes

• accept files saved with a UTF-8 BOM wherever reeval reads user JSON, frontmatter or .env (aefe0c6)
• attempt: keep assertion labels unique when a given label looks numbered (aefe0c6)
• attempt: record a judge error only when a judge caused the failure (aefe0c6)
• capture: accept only capture names the studio can serve (aefe0c6)
• cli: check the keys of group and eval judges before the run (aefe0c6)
• cli: switch out of cancel mode as live.json turns finishing (#548) (e84168a)
• diff: keep changed paths as written and count hunk lines that start with -- or ++ (aefe0c6)
• engine: exit a worker at once when the CLI goes while its file is loading (aefe0c6)
• engine: fail a file's attempts when it fails to load in the run pass (af665de)
• engine: let a filter part that names a file also match names and cells (6f763ad)
• engine: run up to concurrency eval files at once, not one per cpu (6f763ad)
• engine: skip .only evals inside a skipped group in the plan (aefe0c6)
• environment: refuse a password or token in skill and npm source URLs (af665de)
• evaluate: refuse evals and pooled families that would share an id (aefe0c6)
• expect: fail the attempt on a caught failed expect (#684) (68c3447)
• harness: record writes of unknown origin as write, not modify (6f763ad)
• harness: refuse MCP names and values opencode would substitute (6f763ad)
• history: claim a run before writeRun touches its artifact directory (#559) (4514416)
• history: compare pooled-family gates and fail on their regressions (#562) (d229233)
• history: count attempts with no tool calls in effort's per-class medians (aefe0c6)
• history: count gateway changes as drift, not noise (#560) (bd7a243)
• history: keep each failed attempt's artifacts with artifacts 'failures' (#561) (6c4ce39)
• history: keep minLift, provenance and pairwise answers on resume (#563) (00b74e8)
• history: key environments and env_skills on name and digest (aefe0c6)
• history: name a gate change in compare instead of calling it a regression (#565) (6239b12)
• history: settle run commits past rev-list's 20,000 with merge-base (aefe0c6)
• judge: grade a language runner's judge on value as the value itself (aefe0c6)
• judge: mask the judge's input before it's sent (aefe0c6)
• judge: number judgement keys by call order, not completion order (aefe0c6)
• pi: carry a turn on through the run pi continues after compacting (aefe0c6)
• pi: keep a turn pi recovered by auto-retry (aefe0c6)
• profiles: refuse a limit.harness id no eval runs on (6f763ad)
• profiles: turn compare off when limit.harness drops every compared harness (6f763ad)
• python: fail a call that passes NaN or Infinity instead of hanging reeval (aefe0c6)
• python: read REEVAL_VERSION in pep 440 spelling as reeval's version (aefe0c6)
• python: register each eval file's module in sys.modules while it loads (aefe0c6)
• redact: mask a second key of the same provider whole (aefe0c6)
• redact: mask a secret that spans several stream chunks in the trace and raw transcript (aefe0c6)
• redact: mask a token that follows a letter, digit or underscore (aefe0c6)
• redact: mask only JSON strings so records stay valid (#564) (6226b29)
• redact: mask text before it's shortened (aefe0c6)
• redact: mask xai-, vck_ and other distinctive credential prefixes (6f763ad)
• refuse durations above 596h, the longest timer Node can set (aefe0c6)
• sandbox: answer 400 to a gateway request with a port outside 1-65535 (aefe0c6)
• sandbox: hash every file a .dockerignore re-include sends (#624) (668bc0d)
• sandbox: note processes setup leaves running with a checks-only secret (aefe0c6)
• sandbox: substitute placeholders in websocket control frames (aefe0c6)
• secrets: don't broker http:// MCP header secrets, and serve plain HTTP in a CONNECT to a brokered host (af665de)
• secrets: record secret.template variables and name header secrets by host variable (af665de)
• secrets: warn on credential-like MCP headers written as plain strings (af665de)
• studio: serve nothing through a symlinked run directory (aefe0c6)
• studio: show a resumed run's kept attempts with their outcome in live progress (aefe0c6)

Code Refactoring

• spring clean: remove insights, single-source option docs and trim slow or duplicate tests (#667) (46a5000)