v0.4.0 — DAgger vs a size-matched control, and the v0.3 self-review

ResolveLab runs tool-using LLM agents against a real HTTP service that injects failures, and grades each episode from server state and the commit log. v0.4 does two things: it takes the v0.3 headline apart, then runs a registered DAgger study against a size-matched behaviour-cloning control.

Headline

Locked test of 120 independent episodes across 10 held-out fault compositions; half use wording never seen in training. Qwen3-0.6B + LoRA, three training seeds per arm.

Arm	Mean success
Qwen3-0.6B base	0.0%
Qwen3-1.7B base (one demo / three demos)	23.3% / 15.0%
v0.3 published adapters	29.4%
Behaviour cloning	90.8%
BC + extra expert rows	90.8%
BC + verified DAgger rows	98.3%

• Primary comparison: DAgger minus the size-matched control is +7.5 pp, 95% CI [1.9, 13.3]. The pre-registered rule classifies this as supported, and safety-violation counts are equal.
• More data alone: extra expert data changed nothing.
• Mechanism: the gap comes from one compounding error. A mis-copied ETag triggers a self-inflicted 412 loop, which causes 26 of 33 control failures and none of the DAgger failures.

What's in this release

• The v0.3 self-review. The 30 test episodes were three records reused across ten compositions, and the matched base never reached a fault. A counterfactual that moves only the target shows the v0.3 adapters copying the in-context demo's layout: 54/54 later-page targets with the published demo, 0/54 with the demo's target moved to page one.
• resolvelab.casedesk:
  • an independent-record benchmark with declared factors (288 episodes);
  • an off-policy-safe teacher with replay-verified labels;
  • grader v2;
  • DAgger data with a size-matched control;
  • a seed × record bootstrap.
• Evidence:
  • Every input and adapter was hash-locked before the first test episode.
  • scripts/verify_casedesk_v2.py re-derives all 2,843 DAgger labels and replays all 1,920 test episodes over real HTTP without a GPU.
  • Traces and data ship in artifacts/v0.4/.
• Quick start: python examples/casedesk_quickstart.py runs one fault-injected episode and replays two recorded model runs in about two seconds.

Scope

• The services are synthetic and process-local, with a known contract. Results cover held-out compositions, records and wording, not new tools or real incidents.
• One model family, three seeds, one DAgger iteration.
• The two key arms were matched on rows, not tokens.
• Trained adapter weights are not attached; their hashes are in the lock.

Full results: artifacts/v0.4/RESULTS.md · Protocol: docs/casedesk-v2-protocol.md · Changelog: CHANGELOG.md