Skip to content

QualityBench 0.1.0

Choose a tag to compare

View all tags
@launch80 launch80 released this 06 Nov 15:12
· 8 commits to main since this release
v0.1.0
1f603a6

First release of QualityBench, a repeatable quality baseline and quality-loss bench for any OpenAI-compatible endpoint. It is BetterBench's quality sibling.

Install

uv tool install git+https://github.com/launch80/QualityBench@v0.1.0
qualitybench setup
qualitybench run --endpoint http://127.0.0.1:8000/v1 --model my-model --out results/bf16.json --quick
qualitybench compare results/bf16.json results/int4.json

What's in 0.1.0

  • Suite: GSM8K, IFEval, MMLU-Pro (fixed 1000), HumanEval+, MBPP+, BFCL (400, the server's tool parser), LiveCodeBench v6 (175), teacher-forced logprobs (592 × 2048), needle at 32k/128k, and optional spec parity.
  • Run controls: --quick (a fixed tenth), --tasks, --thinking on|off (forced through a local proxy, with a preflight check), --deadline with partial results, and resume by rerunning the same command.
  • compare: KLD, top-1 agreement, perplexity change, and paired task deltas (exact McNemar and a 10,000-resample bootstrap CI), with threshold verdicts.
  • Outputs: a self-contained offline HTML report, and a results file in launch80's bench.qualitybench.v1 format.
  • Request lifecycle: one-hour timeouts, at most 3 attempts, and abandoned generations cancelled on the server.
  • Sandbox: model-written code runs with no network.
  • Tests: re-scoring the Qwopus3.8-27B study's per-item files reproduces its published numbers exactly.

Server notes: logprobs needs vLLM with --max-logprobs 64 and speculative decoding off. BFCL needs the tool-call parser enabled.

Full details are in the README and in CHANGELOG.md.

Community

Questions, results, or a model you want tested? Join the Launch80 Discord: https://discord.gg/launch80

If this work is useful to you, please support the server: https://launch80.com/supporter

Assets 2
Loading