Skip to content

v0.9.5

Choose a tag to compare

@github-actions github-actions released this 26 Oct 18:39
· 14 commits to main since this release

Benchmarks with a public board, models condensed to fit, and an editor in Build.

  • The Bench tab runs local models through nine benchmarks in seven skills: MMLU-Pro, ARC-Challenge, GSM8K, MATH-500, HumanEval, MBPP, IFEval, HellaSwag and TruthfulQA. Every Crucible asks the same items the same way, so a score from one machine stands beside a score from another, and a model that thinks first has room to.
  • A spider chart compares up to three results, with the numbers under it, and a result opens to each benchmark with its margin of error and the answers that were wrong.
  • The public board, mattsaund/crucible-bench. Post a result and it goes up from your GitHub account, checked against Hugging Face, and sits beside everybody else's in the tab. A result is names and numbers, nothing typed.
  • Condense a model from Create: a Q8 that does not fit becomes a Q4 that does, and the largest models go down to IQ2 or IQ1 after a pass over sample text finds the weights that matter most. Every size is measured first and marked by how it fits.
  • An editor in Build. Tabs, a file tree, find and replace, search across the project, Ctrl+P to open a file, and a terminal under it.
  • A fine-tune is scored on data it never trained on, before and after.
  • Quicker where it counts. The delegator decides in a quarter of the time, models like gpt-oss fit a longer context on the same card, and a model split across cards holds back less of each.

Everything else is in the CHANGELOG.