Repository navigation
v0.9.5
Benchmarks with a public board, models condensed to fit, and an editor in Build.
- The Bench tab runs local models through nine benchmarks in seven skills: MMLU-Pro, ARC-Challenge, GSM8K, MATH-500, HumanEval, MBPP, IFEval, HellaSwag and TruthfulQA. Every Crucible asks the same items the same way, so a score from one machine stands beside a score from another, and a model that thinks first has room to.
- A spider chart compares up to three results, with the numbers under it, and a result opens to each benchmark with its margin of error and the answers that were wrong.
- The public board, mattsaund/crucible-bench. Post a result and it goes up from your GitHub account, checked against Hugging Face, and sits beside everybody else's in the tab. A result is names and numbers, nothing typed.
- Condense a model from Create: a Q8 that does not fit becomes a Q4 that does, and the largest models go down to IQ2 or IQ1 after a pass over sample text finds the weights that matter most. Every size is measured first and marked by how it fits.
- An editor in Build. Tabs, a file tree, find and replace, search across the project, Ctrl+P to open a file, and a terminal under it.
- A fine-tune is scored on data it never trained on, before and after.
- Quicker where it counts. The delegator decides in a quarter of the time, models like gpt-oss fit a longer context on the same card, and a model split across cards holds back less of each.
Everything else is in the CHANGELOG.