Repository navigation
Highlights
Sonnet 5 benchmark — no measurable improvement over Sonnet 4.6
Full n=3 run on claude-sonnet-5 (263 unique cases, incl. held-out private, real Docker L1/L2 compile):
| Model | pass@1 (n=3 mean) | 95% CI | Stability |
|---|---|---|---|
| Sonnet 5 | 67.0% | [63.7%, 70.2%] | 82.1% (most stable measured) |
| Sonnet 4.6 | 68.0% | [64.4%, 71.3%] | 87.1% |
| Haiku 4.5 | 56.9% | [53.2%, 60.6%] | 73.0% |
On the 232 cases common to both runs (isolating model change from case-set growth), majority-vote pass@1 is 68.1% → 67.2% = −0.9%p — statistically tied, inside 4.6's own run range. A newer model generation delivered no measurable gain on embedded firmware; the weakest categories (dma, isr-concurrency, threading) are unchanged.
Reports: docs/BENCHMARK-n3-sonnet5.md, docs/BENCHMARK-DELTA-sonnet5-vs-sonnet46.md, docs/BENCHMARK-COMPARISON-2026-04-05.md §10.
Case expansion (185 → 219 public; 267 total with private)
Embedded-Linux Phase A/B/C: kernel drivers, kernel networking, OTA (SWUpdate/RAUC), userspace, U-Boot, Yocto.
New capabilities
- Context Quality Mode — measure a CLAUDE.md/context pack's effect (
context-compare,context-diagnose) - Negatives / mutation oracle (
/negatives) - Hiloop integration — per-check metrics, coverage gate, report schema version
- 2-level SDK-bucket case layout (
cases/<sdk>/<case-id>/)
Fixes
verify_results.pysilently verified 0 cases under the 2-level layout (false all-clear) — now resolves viaiter_case_dirs+--private-cases+ fails loud- 4 public/private case_id collisions renumbered (267 unique)
Full changelog: CHANGELOG.md