Skip to content

0.12.0: benchmark on draft quality

Choose a tag to compare

@linit01 linit01 released this 10 Jul 16:40

The model benchmark now scores draft quality instead of review eagerness. It drafts a fixed set of golden fixtures (sun/night condition, weekday time condition, numeric trigger) single-shot and ranks models by clean first-try drafts, then valid-after-repair, then fewest fixer repairs, and only finally speed. This fixes the old benchmark recommending a fast, eager model that needs the fixer safety net over a disciplined one. Verified against a live server: now recommends qwen2.5-coder:14b over an 8B that produced invalid output. Also folds in fixer hardening: drops flattened bare 'and' conditions, converts single sun-word time bounds to the sun.sun state, and collapses duplicate conditions.