Skip to content

v0.2.0 — Sonnet 5 benchmark + Linux case expansion

Latest

Choose a tag to compare

@Ecro Ecro released this 19 Jul 13:02
· 15 commits to main since this release

Highlights

Sonnet 5 benchmark — no measurable improvement over Sonnet 4.6

Full n=3 run on claude-sonnet-5 (263 unique cases, incl. held-out private, real Docker L1/L2 compile):

Model pass@1 (n=3 mean) 95% CI Stability
Sonnet 5 67.0% [63.7%, 70.2%] 82.1% (most stable measured)
Sonnet 4.6 68.0% [64.4%, 71.3%] 87.1%
Haiku 4.5 56.9% [53.2%, 60.6%] 73.0%

On the 232 cases common to both runs (isolating model change from case-set growth), majority-vote pass@1 is 68.1% → 67.2% = −0.9%p — statistically tied, inside 4.6's own run range. A newer model generation delivered no measurable gain on embedded firmware; the weakest categories (dma, isr-concurrency, threading) are unchanged.

Reports: docs/BENCHMARK-n3-sonnet5.md, docs/BENCHMARK-DELTA-sonnet5-vs-sonnet46.md, docs/BENCHMARK-COMPARISON-2026-04-05.md §10.

Case expansion (185 → 219 public; 267 total with private)

Embedded-Linux Phase A/B/C: kernel drivers, kernel networking, OTA (SWUpdate/RAUC), userspace, U-Boot, Yocto.

New capabilities

  • Context Quality Mode — measure a CLAUDE.md/context pack's effect (context-compare, context-diagnose)
  • Negatives / mutation oracle (/negatives)
  • Hiloop integration — per-check metrics, coverage gate, report schema version
  • 2-level SDK-bucket case layout (cases/<sdk>/<case-id>/)

Fixes

  • verify_results.py silently verified 0 cases under the 2-level layout (false all-clear) — now resolves via iter_case_dirs + --private-cases + fails loud
  • 4 public/private case_id collisions renumbered (267 unique)

Full changelog: CHANGELOG.md