QUANTBENCH

run terminal-bench against llama.cpp quants using the pi.dev agent harness

overview

two tiers, joined by an openai-compatible endpoint:

• model tier: llama-server on a rented GPU, serving one GGUF quant
• harness tier: harbor driving terminal-bench task containers locally

the harness needs containers but no GPU; the model tier needs a GPU but no containers. keeping them apart is what makes this tractable -- vast.ai does not support docker-in-docker, so the harness cannot live there.

the model tier is currently built around vast.ai for the GPU rentals. nothing in the benchmark itself is vast-specific, so other providers are a direction worth supporting -- just not built yet.

requirements

• a container runtime with a real docker compose v2 provider. rootless podman works: install a docker-compose binary on PATH and podman prefers it over podman-compose (which lacks cp).
• harbor (>=0.18.0) with this package installed:

uv tool install harbor --with-editable /path/to/quantbench

configure

etc/models.yaml is the source of truth: it defines the sweep matrix and the llama-server flags per model, distilled from llama-swap. quant ladders are capped at what fits a 128gb strixhalo.

var/agent/ is generated, not authored. sweep.py rewrites it per quant and points PI_AGENT_DIR at it, so it is gitignored and holds a live api key. write it by hand only for one-off manual runs.

.env drives the sweep itself; copy .env.example and fill it in. each setting is documented there.

run a sweep

quantbench -j 10 --resume --retry-incomplete

one rental per model, -j of them at a time, each serving its quants over llama-server and benchmarked by harbor locally. --resume skips tasks already scored and is safe to always pass: infra failures re-rent on a fresh host on their own, and --retry-incomplete also re-runs the incomplete outcomes (truncated, overflow, timeout) once you have raised their cause. --model and --quant narrow the matrix; --list prints it and exits.

watch it live in another terminal, which regenerates var/jobs/index.html:

quantbench-dashboard var/jobs --watch

what a rental costs

offers are ranked on total expected cost, not on the hourly rate vast sorts by. a vast bill is only ~65% gpu time: the rest is the weights download (~18%, priced per TB and varying 39x between hosts) and disk (~16%). rentals are short -- median 14 minutes across 52 invoiced ones, most cut short by a host going offline -- so a 100gb download can cost more than the gpu time it buys, and the cheapest $/hr is regularly the dearest rental. ranking on the total takes 26-58% off a rental, measured live across the matrix.

QUANTBENCH_RENTAL_HOURS_EST sets how the two are weighed: raise it to favour cheap hourly rates, lower it to favour cheap bandwidth.

the dashboard's cost estimate -- a column and a summary badge, both hidden by default and toggled from the column picker -- is built from the same three terms. where vast's invoice has settled a rental, the invoiced figure replaces it.

stopping a sweep

two signals, two behaviours:

• SIGTERM or Ctrl-C tears down immediately -- live rentals are destroyed and in-flight harbor runs killed. stop now, stop paying.
• SIGUSR1 drains -- every point currently running finishes and is billed, no new rental starts, and the sweep exits. wind a run down without throwing away the hours already paid for. --resume picks up the rest. prefer this.

infrastructure failures are not the model's

a task whose server was unreachable -- the agent gets a connection error and produces nothing -- is never scored against the model. the dashboard shows ONLY completed runs and what is in flight: an infra failure, or an incomplete run (truncated, overflow, timeout), drops off entirely and reads as still queued rather than as a 0.0 the model did not earn. infra failures re-rent on a fresh host automatically on --resume; the incomplete ones re-run with --retry-incomplete once you have raised their cause. while a point runs the server is health-checked, and if it dies that harbor run is killed so the point re-rents on a fresh host rather than scoring the rest 0.0. before harbor starts, readiness means a real one-token completion, not just that /models answers.

host failures are classified by what a re-run should do -- a corrupt gguf download, a driver too old for the image, a box that runs out of memory on a model it advertised room for -- and each type has a configurable retry (re-rent on a fresh host, or give up) and blacklist action (remember the machine as bad and stop scheduling on it). the blacklist keys on the physical machine and persists across sweeps, with a strike threshold and a TTL so a one-off transient does not ban a host forever. see the QUANTBENCH_HOST_* knobs in .env.example.

run one quant by hand

for a one-off, or to debug a single quant, drive harbor directly:

harbor run \
  -d terminal-bench@2.0 \
  --agent-import-path quantbench:PiAgent \
  -m vast/unsloth/Qwen3.6-27B-GGUF:Q8_0 \
  --allow-agent-host <model-endpoint-host> \
  -i <task-glob> \
  -n 1

--allow-agent-host allowlists the model endpoint for harbor's agent-phase egress policy. it is ignored for tasks whose effective policy is already public, but is needed for tasks that restrict egress.

set -n to match llama-server's --parallel. concurrency beyond that just queues behind a single decode slot.

validate the harness

the oracle runs a task's reference solution with no agent and no model:

harbor run -d terminal-bench@2.0 -a oracle -i fix-git -n 1