Repository navigation
The 0.6.0 trustworthy-eval release.
What's new
- The reworked eval instrument: per-run outcome classification (completed / killed_at_cap / crashed) with a completion floor (≥3 of 5 per arm per scenario, else a distinct data-quality RED); behavior is asserted over completed runs only — a killed or crashed run is never "did not ask".
- The baseline arm is alive:
node_modulesis hard-linked into the baseline worktree. The 0.4.1 arm was structurally dead since 0.5.0 (the plugin never loaded there), so this is the first release whose lift-vs-baseline is a real measurement. - New S4
delegatescenario: main composes the peer-required briefing itself — it measures the task-design guidance's lift, with the 0.4.1 arm (no guidance doc) as the one-shot control. - S1's child task is bounded (concrete file + concrete missing value) — fixes the unbounded-
find /timeout artifact. - Advisory LLM judge (Qwen3.8-27B): scores question/answer quality and the noise guard on gradable runs, reports only — never a gate. Per-run artifacts are retained under
$XDG_DATA_HOME/opencode/eval/(last 3 evals). - The second injected agent-facing doc,
docs/task-design.md: the peer-required task test, the shaping patterns, the anti-patterns, worked example pairs. - The tuning-loop norm: README "Evaluation" section + AGENTS.md rule — the eval is a hard local gate on protocol text, the baseline rolls to each release, and instrument changes are release-level events.
Release-run eval (one-shot, vs 0.4.1; the baseline ref rolls to v0.6.0 after this release)
PASS — 40 runs (4 scenarios × 5 runs × 2 arms):
| scenario | new (0.6.0) | baseline (0.4.1) |
|---|---|---|
| ask | 3/3 asked (3/5 completed, 2 cap) | 0/5 asked |
| answer | 2/5 answered | 2/5 answered |
| noise | 0/5 noised | 0/5 noised |
| delegate (new) | 3/3 asked (3/5 completed, 2 cap) | 0/4 asked (1 crash) |
Advisory judge: 35/40 gradable — question avg 0.86/14 scored, answer avg 0.89/9 scored, noise pass 10/10. The delegate row is the task-design guidance's lift measured directly: the new arm's main composes peer-required briefings (child asks in 3/3 completed runs); the 0.4.1 arm never does (0/4).
Also
- SDK range re-proof (per §13):
@opencode/plugin@2.0.14(range top) — 161/161 tests + typecheck in a scratch worktree. - Out of scope for 0.6.0 (the map's fog/out-of-scope): multi-model eval, structural delivery, Nix packaging, CI.