Skip to content

v0.6.0

Latest

Choose a tag to compare

@kido5217 kido5217 released this 22 Sep 19:00
· 2 commits to main since this release
v0.6.0
1f1f54a

The 0.6.0 trustworthy-eval release.

What's new

  • The reworked eval instrument: per-run outcome classification (completed / killed_at_cap / crashed) with a completion floor (≥3 of 5 per arm per scenario, else a distinct data-quality RED); behavior is asserted over completed runs only — a killed or crashed run is never "did not ask".
  • The baseline arm is alive: node_modules is hard-linked into the baseline worktree. The 0.4.1 arm was structurally dead since 0.5.0 (the plugin never loaded there), so this is the first release whose lift-vs-baseline is a real measurement.
  • New S4 delegate scenario: main composes the peer-required briefing itself — it measures the task-design guidance's lift, with the 0.4.1 arm (no guidance doc) as the one-shot control.
  • S1's child task is bounded (concrete file + concrete missing value) — fixes the unbounded-find / timeout artifact.
  • Advisory LLM judge (Qwen3.8-27B): scores question/answer quality and the noise guard on gradable runs, reports only — never a gate. Per-run artifacts are retained under $XDG_DATA_HOME/opencode/eval/ (last 3 evals).
  • The second injected agent-facing doc, docs/task-design.md: the peer-required task test, the shaping patterns, the anti-patterns, worked example pairs.
  • The tuning-loop norm: README "Evaluation" section + AGENTS.md rule — the eval is a hard local gate on protocol text, the baseline rolls to each release, and instrument changes are release-level events.

Release-run eval (one-shot, vs 0.4.1; the baseline ref rolls to v0.6.0 after this release)

PASS — 40 runs (4 scenarios × 5 runs × 2 arms):

scenario new (0.6.0) baseline (0.4.1)
ask 3/3 asked (3/5 completed, 2 cap) 0/5 asked
answer 2/5 answered 2/5 answered
noise 0/5 noised 0/5 noised
delegate (new) 3/3 asked (3/5 completed, 2 cap) 0/4 asked (1 crash)

Advisory judge: 35/40 gradable — question avg 0.86/14 scored, answer avg 0.89/9 scored, noise pass 10/10. The delegate row is the task-design guidance's lift measured directly: the new arm's main composes peer-required briefings (child asks in 3/3 completed runs); the 0.4.1 arm never does (0/4).

Also

  • SDK range re-proof (per §13): @opencode/plugin@2.0.14 (range top) — 161/161 tests + typecheck in a scratch worktree.
  • Out of scope for 0.6.0 (the map's fog/out-of-scope): multi-model eval, structural delivery, Nix packaging, CI.