Skip to content

QualityBench 0.2.0

Choose a tag to compare

View all tags
@launch80 launch80 released this 06 Nov 17:55
· 21 commits to main since this release
v0.2.0
329daf2

QualityBench runs take a while, so 0.2.0 tells you how long before it starts, and asks.

  • Time estimate. After the warmup, run measures the server for a few seconds (16 parallel chats and one long prompt), scales the reference run's per-task times by the speed difference and prints an estimate with a range. Once a model has a run on record, its real times are used instead. They are kept in timings.jsonl in the QualityBench home and never leave the machine.
  • Approval. On a terminal, run asks Start? [y/N] before the first task, and harness setup waits for the answer.
    • --yes skips the question.
    • QUALITYBENCH_CONFIRM=never or over:30m changes when it asks.
    • Runs without a terminal are never asked.
    • --max-minutes N refuses runs that could take longer, with exit 3.
  • qualitybench estimate [--json] prints the estimate without running anything.
  • Progress: a qb:estimate line before the first task, elapsed_s on task lines, and a run line every 10 seconds with the overall fraction and the time left (eta_s). L80 1.3.1 draws them as an overall progress bar.

The first estimate for a model is rough, because the probe measures speed, not how much the model writes; the range says so. After one run it uses real times. Results files are unchanged: the estimate is kept in <name>.qb/ENV.json, not in the published results.

uv tool install git+https://github.com/launch80/QualityBench@v0.2.0

Community

Questions, results, or a model you want tested? Join the Launch80 Discord: https://discord.gg/launch80

If this work is useful to you, please support the server: https://launch80.com/supporter

Assets 2
Loading