v3.9.0

What's new

Features

The evals harness now records bounded prompt-composition evidence and models three distinct observation states for each catalog check — present, absent, and impossible — instead of collapsing the latter two together. Previously, a run where the coverage signal simply couldn't be observed was treated the same as a run that observed no coverage at all, which masked infrastructure problems as real coverage gaps.

Systematic and OpenCode catalog evidence is now partitioned by workflow-block containment, and that partitioning fails closed on malformed or unmatched blocks. This guarantees that Systematic-rendered catalog content can never be miscounted as host-runtime coverage, even when a workflow-block marker is broken or missing.

A new permanent host-skill-coverage case performs host-catalog coverage on the pinned runtime as exact set inclusion against the raw bundled skill names. Extra host entries are tolerated, but missing entries fail the case and are explicitly named in the failure output, and the case manifest is drift-protected by deriving expected names from the live bundled catalog at test time — so the manifest can't silently go stale as skills are added or renamed.

Execution failures are also now split into two categories: diagnosed task failures (including host runtime timeouts) and undiagnosed infrastructure failures when no transform is observable at all, giving clearer signal about where a failing eval run actually broke.

This release also lands a fast differential test that feeds a shared corpus — including valid markers, unmatched markers, nested markers, duplicated Systematic catalogs outside the workflow block, mixed host/Systematic catalogs, and non-string entries — through both the typed TypeScript catalog-observation logic and the generated JS that actually executes in the spawned host, to keep the two implementations from drifting apart. A positive control ensures the test can't pass by having both implementations return the same "impossible" state. The probe artifact identifiers (probeId and probeDigest) were also bumped to v3 so persisted run artifacts map cleanly to the implementation revision that produced them.

This work covers eval observation and gating only; no production behavior under src/ changed in this release.

Also in this release

• build: Updated OpenCode to v1.18.17.
• deps: Updated typebox to v1.3.13, bfra-me/.github to v4.16.46, and fro-bot/agent to v0.99.0.

Compare

v3.8.0...v3.9.0