QualityBench 0.3.0

0.3.0 fixes scores that came out too low: GSM8K replies with the right answer marked wrong, and thinking models cut off at the token limit before they answered. It also adds podman, and task scores can now be published without logprobs.

• GSM8K reads the final answer. The grader takes the last \boxed{} or #### N. Failing that, it takes whichever comes last of a bold quantity (**$132.00**) and Answer: N. Failing that, it takes the final sentence's last money amount or number. Answers are compared as numbers, so $7.00 matches 7 and "Judy makes $7,425 in 1 week." reads 7425, not 1.
  • Re-grading the six full Qwopus study runs adds 126–133 correct items per thinking-off run (about +9.5 points; 63 with thinking on). At most 2 items per run lose a correct mark the old scoring had given.
  • The prompt is unchanged. Each item keeps lm-eval's stock result, and the log prints both counts.
  • Runs made with 0.2.0 are re-graded from their saved lm-eval samples when built or compared, so an old reference still pairs fairly. compare warns if one side can't be re-graded.
• Token-limit warnings. Every run counts the replies that stopped at the token limit, per task. A task where 1% or more did gets a caveat in the results saying its score understates the model.
  • A model that reasoned while --thinking was left at the server default gets its own warning and caveat.
  • Preflight also catches reasoning returned inline as <think> by a server with no reasoning parser.
  • --thinking-max-tokens now defaults to 32768 (was 16384).
• Podman. The code tasks run with docker or podman. QUALITYBENCH_CONTAINER=podman picks podman, and podman is also used when docker isn't installed. Podman runs get --userns=keep-id and --security-opt label=disable, so rootless podman on Fedora with SELinux works with no docker shim. The evalplus image name is now fully qualified.
• Task scores without logprobs. On a server without prompt_logprobs (llama.cpp, radiance), leave logprobs out of --tasks. The results no longer say launch80 won't accept them; launch80 publishes them and shows perplexity and KLD as n/a.

GSM8K scores are about 9 points higher than 0.2.0 on chat models. Compare runs made with the same grader; compare re-grades older runs for you when their samples are still in the .qb/ folder. L80 picks up 0.3.0 with its next release.

uv tool install git+https://github.com/launch80/QualityBench@v0.3.0

Community

Questions, results, or a model you want tested? Join the Launch80 Discord: https://discord.gg/launch80

If this work is useful to you, please support the server: https://launch80.com/supporter

QualityBench 0.2.0

QualityBench runs take a while, so 0.2.0 tells you how long before it starts, and asks.

• Time estimate. After the warmup, run measures the server for a few seconds (16 parallel chats and one long prompt), scales the reference run's per-task times by the speed difference and prints an estimate with a range. Once a model has a run on record, its real times are used instead. They are kept in timings.jsonl in the QualityBench home and never leave the machine.
• Approval. On a terminal, run asks Start? [y/N] before the first task, and harness setup waits for the answer.
  • --yes skips the question.
  • QUALITYBENCH_CONFIRM=never or over:30m changes when it asks.
  • Runs without a terminal are never asked.
  • --max-minutes N refuses runs that could take longer, with exit 3.
• qualitybench estimate [--json] prints the estimate without running anything.
• Progress: a qb:estimate line before the first task, elapsed_s on task lines, and a run line every 10 seconds with the overall fraction and the time left (eta_s). L80 1.3.1 draws them as an overall progress bar.

The first estimate for a model is rough, because the probe measures speed, not how much the model writes; the range says so. After one run it uses real times. Results files are unchanged: the estimate is kept in <name>.qb/ENV.json, not in the published results.

uv tool install git+https://github.com/launch80/QualityBench@v0.2.0

Community

Questions, results, or a model you want tested? Join the Launch80 Discord: https://discord.gg/launch80

If this work is useful to you, please support the server: https://launch80.com/supporter

QualityBench 0.1.1

A patch release on top of 0.1.0.

• Preflight: a model that thinks by default, run without --thinking, now stops at once and asks you to pass --thinking off (the usual baseline) or --thinking on. Before, it failed with "the model returned no content": its reasoning used up the preflight's tokens, and the harnesses' token limits would have cut its replies off mid-thought anyway.
• README: links to the 2-minute intro video and the launch post.

uv tool install git+https://github.com/launch80/QualityBench@v0.1.1

L80 1.3.0 pins this version, so L80 qualitybench runs it with no install step.

Community

Questions, results, or a model you want tested? Join the Launch80 Discord: https://discord.gg/launch80

If this work is useful to you, please support the server: https://launch80.com/supporter

QualityBench 0.1.0

First release of QualityBench, a repeatable quality baseline and quality-loss bench for any OpenAI-compatible endpoint. It is BetterBench's quality sibling.

Install

uv tool install git+https://github.com/launch80/QualityBench@v0.1.0
qualitybench setup
qualitybench run --endpoint http://127.0.0.1:8000/v1 --model my-model --out results/bf16.json --quick
qualitybench compare results/bf16.json results/int4.json

What's in 0.1.0

• Suite: GSM8K, IFEval, MMLU-Pro (fixed 1000), HumanEval+, MBPP+, BFCL (400, the server's tool parser), LiveCodeBench v6 (175), teacher-forced logprobs (592 × 2048), needle at 32k/128k, and optional spec parity.
• Run controls: --quick (a fixed tenth), --tasks, --thinking on|off (forced through a local proxy, with a preflight check), --deadline with partial results, and resume by rerunning the same command.
• compare: KLD, top-1 agreement, perplexity change, and paired task deltas (exact McNemar and a 10,000-resample bootstrap CI), with threshold verdicts.
• Outputs: a self-contained offline HTML report, and a results file in launch80's bench.qualitybench.v1 format.
• Request lifecycle: one-hour timeouts, at most 3 attempts, and abandoned generations cancelled on the server.
• Sandbox: model-written code runs with no network.
• Tests: re-scoring the Qwopus3.8-27B study's per-item files reproduces its published numbers exactly.

Server notes: logprobs needs vLLM with --max-logprobs 64 and speculative decoding off. BFCL needs the tool-call parser enabled.

Full details are in the README and in CHANGELOG.md.

Community

Questions, results, or a model you want tested? Join the Launch80 Discord: https://discord.gg/launch80

If this work is useful to you, please support the server: https://launch80.com/supporter