Repository navigation
v0.1.0 — First public release
Highlights
- 233 test cases (185 public + 48 private held-out for contamination prevention)
- 23 evaluation categories across Zephyr RTOS / FreeRTOS / ESP-IDF / STM32 HAL / Linux kernel / Yocto
- 5-layer evaluation pipeline: L0 Static → L1 Compile → L2 Runtime → L3 Behavioral → L4 Mutation
- n=3 benchmark results (statistically significant, non-overlapping 95% CIs):
- Claude Sonnet 4.6: 68.0% pass@1 [64.4%, 71.3%]
- Claude Haiku 4.5: 56.9% pass@1 [53.2%, 60.6%]
- HuggingFace Space leaderboard with interactive heatmap and per-category breakdown
- Launch materials: README hero, ROADMAP, issue templates, community posts
Weakest Categories (for Haiku)
dma, isr-concurrency, memory-opt, threading — consistently low across 3 runs
Key Insight
35%p gap between explicit prompts (~95%) and implicit domain-knowledge requirements (~60%) — benchmarks that only test explicit specs may overestimate LLM embedded capability.