v0.2.0 — Sonnet 5 benchmark + Linux case expansion

Highlights

Sonnet 5 benchmark — no measurable improvement over Sonnet 4.6

Full n=3 run on claude-sonnet-5 (263 unique cases, incl. held-out private, real Docker L1/L2 compile):

Model	pass@1 (n=3 mean)	95% CI	Stability
Sonnet 5	67.0%	[63.7%, 70.2%]	82.1% (most stable measured)
Sonnet 4.6	68.0%	[64.4%, 71.3%]	87.1%
Haiku 4.5	56.9%	[53.2%, 60.6%]	73.0%

On the 232 cases common to both runs (isolating model change from case-set growth), majority-vote pass@1 is 68.1% → 67.2% = −0.9%p — statistically tied, inside 4.6's own run range. A newer model generation delivered no measurable gain on embedded firmware; the weakest categories (dma, isr-concurrency, threading) are unchanged.

Reports: docs/BENCHMARK-n3-sonnet5.md, docs/BENCHMARK-DELTA-sonnet5-vs-sonnet46.md, docs/BENCHMARK-COMPARISON-2026-04-05.md §10.

Case expansion (185 → 219 public; 267 total with private)

Embedded-Linux Phase A/B/C: kernel drivers, kernel networking, OTA (SWUpdate/RAUC), userspace, U-Boot, Yocto.

New capabilities

• Context Quality Mode — measure a CLAUDE.md/context pack's effect (context-compare, context-diagnose)
• Negatives / mutation oracle (/negatives)
• Hiloop integration — per-check metrics, coverage gate, report schema version
• 2-level SDK-bucket case layout (cases/<sdk>/<case-id>/)

Fixes

• verify_results.py silently verified 0 cases under the 2-level layout (false all-clear) — now resolves via iter_case_dirs + --private-cases + fails loud
• 4 public/private case_id collisions renumbered (267 unique)

Full changelog: CHANGELOG.md

v0.1.0 — First public release

Highlights

• 233 test cases (185 public + 48 private held-out for contamination prevention)
• 23 evaluation categories across Zephyr RTOS / FreeRTOS / ESP-IDF / STM32 HAL / Linux kernel / Yocto
• 5-layer evaluation pipeline: L0 Static → L1 Compile → L2 Runtime → L3 Behavioral → L4 Mutation
• n=3 benchmark results (statistically significant, non-overlapping 95% CIs):
  • Claude Sonnet 4.6: 68.0% pass@1 [64.4%, 71.3%]
  • Claude Haiku 4.5: 56.9% pass@1 [53.2%, 60.6%]
• HuggingFace Space leaderboard with interactive heatmap and per-category breakdown
• Launch materials: README hero, ROADMAP, issue templates, community posts

Weakest Categories (for Haiku)

dma, isr-concurrency, memory-opt, threading — consistently low across 3 runs

Key Insight

35%p gap between explicit prompts (~95%) and implicit domain-knowledge requirements (~60%) — benchmarks that only test explicit specs may overestimate LLM embedded capability.

Links

• Leaderboard (HF Space)
• Methodology
• Full n=3 Comparison