localperf v0.3.0

localperf v0.3.0 makes llama.cpp the primary runtime, stops each benchmark point when its number is known, and measures decode at long context from a restored cache instead of a repeated cold prefill. Suite files change to version 3. A suite with version 1 or 2, or a repeats field, no longer loads.

Changes

Adaptive convergence. A point repeats until the 95% confidence interval of its throughput is within ±5%, with 3 to 10 samples and a 600 s limit for each point. The remaining planned samples are skipped with the reason. The report shows the interval and the stop reason in the cell details, and marks a point that stopped without converging with *. localperf artifact check recalculates every stop decision from the stored data. The repeats field is replaced by a convergence block. See docs/2026-10-02-adaptive-convergence.md.

Restored context for decode cases. A decode case with a long prompt now primes its context once, saves the llama.cpp KV cache, and restores it into every slot before each sample, so the sample measures decode only. Every request must report its prompt as cached, or the sample fails. Snapshots are kept under the user cache directory (runner.snapshot_dir, or LOCALPERF_SNAPSHOT_DIR), so later runs skip the prime. Prefill is measured by cold cases. practical-64k gains a prefill-64k case at 1 user, and its full-context rows no longer report prefill at 6 users. On Ternary Bonsai 2 9B at 64k context, decode now measures 31 tok/s. The old method reported 5.7 tok/s, because most of each sample was prefill. See docs/2026-10-02-restored-context-decode.md.

Restored cases need llama.cpp. On vLLM, the built-in suites run only their cold cases until vLLM gets a context preparer.

llama.cpp first. llama.cpp is now the primary runtime in the README, the examples, and the agent skill. A managed llama-server gets its slots, context size, and --slot-save-path from the suite, and the detail view shows the llama.cpp server settings.

Safer runs. bench run refuses to overwrite an existing run without --replace. Resume no longer re-runs short-prompt samples, and keeps the artifact valid while a point runs again.

Mutation testing is now optional. It runs as a weekly scan and by hand, not on every push or pull request.

Install

Download the archive for your system below and check it against SHA256SUMS, or run:

go install github.com/osolmaz/localperf/cmd/localperf@v0.3.0

localperf v0.2.0

localperf v0.2.0 replaces the spec files and the sweep planner of v0.1.0 with named suites and deployments. Older specs and localperf sweep plan are gone along with the localperf-sweep-* helper commands. Reports now accept only validated SQLite artifacts written by localperf bench run.

Changes

localperf bench run --suite <name> --deployment deployment.json is the new way to run a benchmark. The README lists the three built-in suites. A deployment is either a vLLM server that localperf manages or an OpenAI-compatible server that is already running, such as llama.cpp.

Every benchmark point now checks that the requested backend really ran. A result whose backend fell back to something else is marked invalid instead of reported. The built-in HTTP client adds a unique prefix to every request, so a repeated prompt cannot hit the server's prompt cache. Turn it off with "prompt_nonce": false.

Reports gained full-run timing and dedicated prefill numbers, and the HTML report and the viewer now follow the system's dark or light setting. The README has a cover and covers both deployment types.

Install

Download the archive for your system below and check it against SHA256SUMS, or run:

go install github.com/osolmaz/localperf/cmd/localperf@v0.2.0

The archive names changed to localperf-v0.2.0-<os>-<arch>.tar.gz.

localperf v0.1.0

First release of localperf, a local LLM inference benchmark CLI for vLLM-managed runs.

Highlights

• Verified context labels: every workload declares context_target/context_semantics; specs that conflate server capacity with active context are refused before GPU time, and reports label rows only by declared-and-measured active context or the measured token shape — never by max_model_len alone.
• localperf sweep plan: generates the default context/concurrency sweep grid with contract-compliant shapes; generated specs are guaranteed to validate.
• Model-level artifacts: bench run --artifact appends repeated runs of one model into a single SQLite file; artifact merge --into combines existing artifacts idempotently with provenance-checked collision refusal.
• Complete metrics: token-weighted ITL alongside request-weighted TPOT, TTFT/latency p50/p95/p99, RPS, achieved concurrency, failure breakdowns, repeat aggregation with mean ± spread, and optional SLO-based goodput.
• Capture layer: hardware inventory, GPU utilization/memory sampled from all available sources (tegrastats, nvidia-smi), and engine identity probes that verify the served model against the spec.
• Standalone HTML reports rendered from the SQLite artifact, with a generated metric legend; every displayed number is re-derivable from stored raw data.
• Conservative memory safety: MemAvailable floors enforced before and during every step.

Install

Download a binary below, or:

go install github.com/dutifuldev/localperf/cmd/localperf@v0.1.0

Binaries: linux/amd64, linux/arm64, darwin/amd64, darwin/arm64 (pure Go, no CGO).