QualityBench 0.3.0

0.3.0 fixes scores that came out too low: GSM8K replies with the right answer marked wrong, and thinking models cut off at the token limit before they answered. It also adds podman, and task scores can now be published without logprobs.

• GSM8K reads the final answer. The grader takes the last \boxed{} or #### N. Failing that, it takes whichever comes last of a bold quantity (**$132.00**) and Answer: N. Failing that, it takes the final sentence's last money amount or number. Answers are compared as numbers, so $7.00 matches 7 and "Judy makes $7,425 in 1 week." reads 7425, not 1.
  • Re-grading the six full Qwopus study runs adds 126–133 correct items per thinking-off run (about +9.5 points; 63 with thinking on). At most 2 items per run lose a correct mark the old scoring had given.
  • The prompt is unchanged. Each item keeps lm-eval's stock result, and the log prints both counts.
  • Runs made with 0.2.0 are re-graded from their saved lm-eval samples when built or compared, so an old reference still pairs fairly. compare warns if one side can't be re-graded.
• Token-limit warnings. Every run counts the replies that stopped at the token limit, per task. A task where 1% or more did gets a caveat in the results saying its score understates the model.
  • A model that reasoned while --thinking was left at the server default gets its own warning and caveat.
  • Preflight also catches reasoning returned inline as <think> by a server with no reasoning parser.
  • --thinking-max-tokens now defaults to 32768 (was 16384).
• Podman. The code tasks run with docker or podman. QUALITYBENCH_CONTAINER=podman picks podman, and podman is also used when docker isn't installed. Podman runs get --userns=keep-id and --security-opt label=disable, so rootless podman on Fedora with SELinux works with no docker shim. The evalplus image name is now fully qualified.
• Task scores without logprobs. On a server without prompt_logprobs (llama.cpp, radiance), leave logprobs out of --tasks. The results no longer say launch80 won't accept them; launch80 publishes them and shows perplexity and KLD as n/a.

GSM8K scores are about 9 points higher than 0.2.0 on chat models. Compare runs made with the same grader; compare re-grades older runs for you when their samples are still in the .qb/ folder. L80 picks up 0.3.0 with its next release.

uv tool install git+https://github.com/launch80/QualityBench@v0.3.0

Community

Questions, results, or a model you want tested? Join the Launch80 Discord: https://discord.gg/launch80

If this work is useful to you, please support the server: https://launch80.com/supporter