Repository navigation
Release - v2.1.0
π llama-swap-llm-benchmark v2.1.0 β faster full runs, rankings you can still trust
A full run of ~28 models is almost entirely the GPU generating eval answers. v2.1 cuts that time down without letting a weaker model rank higher. It does this by asking fewer questions of models that are clearly losing, never re-asking a question it already has an answer for, and running several questions at once where the server can take them. A timing report shows where every minute went, and --estimate predicts a run before it starts.
β¨ Key New Features
β±οΈ Timing report and --estimate
- Every model now records its seconds per phase: settle, load, speed tests, prefill-at-length, agent TTFT, cooldown, eval load, long context, and each dataset. Thinking and answer tokens are counted too.
- Where it shows up:
results.json:models[].timing, plusrun.timingwith the speed/eval phase wall times, the per-dataset totals and the token counts.- CSV columns
time_*_sandeval_tok_*. - A new dashboard section, Where the time went.
python benchmark.py --estimatepredicts a run's duration from earlier runs' JSON (measured tok/s, prefill speed, seconds and tokens per question) and any batch-probe result. Models with no history get a rough guess. A one-line estimate is also printed at the start of every run.
π» Funnel profile with confidence-interval gates (EVAL_PROFILE=funnel)
- Every model answers
quick. After each stage, a model goes on only if its upper bound reaches the lower bound of theEVAL_LEADERS-th model. At leastEVAL_LEADERSmodels always go on. - Survivors answer the next stage (
basicβmoderateβ β¦ βEVAL_FUNNEL_FINAL, defaultstandard). They are asked only the new questions. - A plain "top 5 after quick" was tested and rejected. In a simulation with a crowded top, it screened out the true best model in 18β33% of runs. The confidence gates lost it in β€0.1% of runs, picked as well as a full run, and saved ~40β50% of the eval work.
- A screened-out model keeps its stage score but is not ranked and never reaches the podium. First-stage avg compares every model on the same first-stage questions.
πΎ Result cache and resume
- Every answered question is stored in
results/cache/. A rerun, or a run restarted after a crash, asks only what is missing. - An answer is reused only when all of these are identical:
- the model file (size plus a hash of its first and last MiB)
- the launch flags and llama.cpp build
- the loaded context, slots and chat template
- the hardware
- the exact request and sampling settings
- the thinking budget and the retry, salvage and forcing switches
- the grader version
- Speed tests are always measured again.
β‘ Batched evaluation that cannot change a ranking
-
Questions in flight are now capped per model to the server's slots (
/propstotal_slots). With a shared KV pool (--kv-unified, the llama.cpp default when-npis auto) they are also capped to what fits the pool in the worst case. The cap is printed whenever it applies. -
-kvu/-no-kvuare read from the launch command, and a new Unified KV column shows the result. -
When any question ran concurrently, the Agent score uses a single-stream s/q estimate for every model: tokens Γ· speed-test tok/s, plus prompt tokens Γ· prefill tok/s, plus grading time. Measured latency under load includes queueing, so it is no longer used for ranking in that case.
-
New
python benchmark.py --batch-probeasks the same questions at 1/2/4/8 in flight. It reports:- the throughput gain
- per-question latency
- accuracy
- how many answers agree with the one-at-a-time answers
- errors and peak VRAM
It then recommends a level.
π Pipelined grading
- Sandboxed code runs while the next question is generating. The sandbox itself is unchanged: one isolated container per program.
- This replaces the warm container pool from the plan, which would have let one program affect the next.
π
Skip evals of models that cannot be deployed (EVAL_SKIP_INELIGIBLE=auto)
- Models below
MIN_CTX, failing a budget rule, or aboveGOLDILOCKS_MAX_VRAM_PCTare no longer evaluated. They were off the podium anyway. They stay in the table with the reason.
π Long context: one shared document per length
- Each length now builds one document. The questions about it ask different vaults and reuse the document from llama-server's prompt cache.
- The first, cold question doubles as the prefill-at-length measurement. A
PREFILL_CTX_TESTSlength that is also aLONGCTX_LENGTHSlength is no longer sent twice.
π§ Thinking budgets and budget forcing (opt-in)
EVAL_THINK_BUDGET_<DATASET>sets a per-dataset thinking budget.EVAL_BUDGET_FORCING=true: at the cap, the reasoning is kept and closed, and the model is asked for its answer instead of a No-Thinking retry. This uses llama-server's/apply-templateand/completion. The method comes from s1: Simple test-time scaling (Muennighoff et al., 2025).- Both change what is measured, so both are off by default and recorded in
results.json.
π¬ Dashboard: tooltips, column chooser and dataset links
- Every column header has a tooltip: what it measures, where the number comes from and whether higher or lower is better. Eval columns and dataset names also say how big the test is and how it is scored. Budget-rule columns show the rule's limit.
- Hide the columns you don't need:
- Click the Γ in any header to hide that column.
- The new Columns panel toggles single columns or whole groups (Model, Launch, VRAM & memory, Power & cost, Speed, Evals, Long-ctx, Budget, Scores).
- Only this group shows just the model columns plus one group, for example an evals-only view in one click.
- Show all columns brings back the full table.
- The column choice is remembered in your browser, so later dashboards open with the same view. Hiding a column changes only the view, never the data or the scores.
- Test names in the page header link to their datasets. Each dataset name in the header and above the accuracy charts opens the Hugging Face page it was loaded from (the pinned revision when
EVAL_<DATASET>_REVISIONis set). Built-in tool calling links to its source file.
π Rebuild a dashboard without a run (--reload)
python benchmark.py --reloadrebuilds the dashboard of the newest finished run (results/results.json) with the current dashboard code. No server or GPU is needed.python benchmark.py --reload results/benchmark_<stamp>.jsonreloads a specific run.- Nothing is measured or rescored: every number, score and ranking is the one the run saved, and the run's own settings are used rather than your current
.env. - It writes
results/benchmark_dashboard_<run>_reload.html(the run's own dashboard is kept) and updatesbenchmark_dashboard.html. - It is mainly a tool for working on the dashboard. A run writes its JSON when it finishes (with
JSON=true, the default), so an interrupted run can't be reloaded.
π§© LiveCodeBench
EVAL_LCB_SINCE=2025-01-01keeps only newer problems, which follows LiveCodeBench's own contamination method.- A load-time guard refuses a copy whose questions are in Korean. The
bzantium/livecodebenchcard mentions a Korean translation; the data itself is English. EVAL_<DATASET>_REVISIONpins any Hugging Face dataset to a commit.
π Fixes
- A weaker model could rank first when one of its datasets failed (high). The eval average was taken only over the datasets that produced a score. Example: every GPQA request hits a template error. That model was then averaged over easier datasets, and outranked a model that was stronger on everything it answered (Agent 92.8 vs 88.0). Now a model is ranked only with a score on every dataset of the run (β₯90% of each dataset's questions answered without an API error). Otherwise it shows "not ranked: incomplete evals (GPQA 0/96 answered)".
- Profiles were not nested. MATH-500's
quickquestions were not a subset ofstandard's. Every profile now takes a prefix of one seeded question order per dataset. - The first question of each model carried the model load in its latency. A warm-up request now loads the model first. The load is timed separately (
eval_load). --kv-unifiedwas not reflected. The docs said--parallelalways divides the context.- Long context at the full loaded context failed with HTTP 400. Found in a real run: 128k at ctx 131,072 produced prompts of 131,644β132,331 tokens, so two models had no 128k result. The test now keeps 3% of the context back. If the server still reports "exceeds the available context size", it shrinks the document once and retries.
- Long context scored 0% for models that think even when told not to. Found in a real run: LFM replied with reasoning only and an empty answer at 64k and 128k, so its v2.0 long-context score (and the Agent score blended from it) was an artefact. Such a reply is now asked once more with 4,096 tokens; it is noted and marked
thinking_retry. - The batch probe understated the gain. It divided total tokens by wall time, so one long reply finishing alone at the end dominated. It now reports the steady gain (Little's law), writes every answer to
batch_probe_<stamp>_details.jsonl, reads ctx/slots/KV from the server, refuses more than 4 models and warns when another run is still using the server. Agreement is only a rule from 40 questions up. --estimateno longer counts eval time for models that would be skipped, and is calibrated on the tokens per question measured in a real run.- Small fixes: the Agent tie margin counted questions from datasets with no score, and long-context requests let
EVAL_EXTRA_BODYoverridestream.
π Security
EVAL_CODE_SANDBOX=autono longer falls back to an unisolated subprocess when Docker is missing or fails its self-test. The run stops with the reason. SetEVAL_CODE_SANDBOX=subprocessto accept it explicitly.
βοΈ How old and new results compare
- Complete eval results, one request at a time: Agent, Chat, Goldilocks and Sweet-spot scores use the same formulas as v2.0.
- Questions differ from v2.0 (nested sampling). Per-question results are not row-for-row comparable, and scores move within the margin of error. The statistical precision is the same.
- s/q is slightly lower for each model's first dataset, because the model load is no longer counted (β load_s Γ· questions). Slow-loading models gain a little speed credit as a result.
- Models with an incomplete dataset are no longer ranked. Before, they were averaged over the rest.
- With
EVAL_CONCURRENCY> 1, the Agent score uses single-stream s/q. At concurrency 1 nothing changes. - Long-context % comes from one document per length: different needles and positions than v2.0, within noise.
- With
EVAL_LCB_SINCEset, LiveCodeBench scores cover newer, harder problems and are not comparable with runs without it. - Funnel runs: screened-out models have no Agent score.
- Dashboard changes and
--reloadaffect only how results are shown. Scores and rankings are unchanged.
π Updating
- Pull the new version. There are no new dependencies (
pip install -r requirements.txtis unchanged). - Copy the new settings you want from
.env.exampleinto your.env. Every new setting has a safe default. - If you run code datasets without Docker, add
EVAL_CODE_SANDBOX=subprocess, or the run will stop. - Optional: run
python benchmark.py --estimatebefore your next run, and--batch-probebefore raisingEVAL_CONCURRENCY. results/(including the newcache/) stays git-ignored. Don't publish it after a GPQA run.
βοΈ Configuration
EVAL_PROFILE=funnel # or test / quick / ... as before
EVAL_FUNNEL_FINAL=standard
EVAL_LEADERS=5
EVAL_FUNNEL_CONF=0.95
EVAL_SKIP_INELIGIBLE=auto # auto | true | false
EVAL_RESULT_CACHE=true
EVAL_RESULT_CACHE_DIR= # default results/cache
EVAL_SQ_BASIS=auto # auto | measured | single
BATCH_PROBE_LEVELS=1,2,4,8
BATCH_PROBE_QUESTIONS=32
ESTIMATE_GPU_BW_GBS=900
EVAL_THINK_BUDGET_GSM8K= # per-dataset budgets, opt-in
EVAL_BUDGET_FORCING=false
EVAL_LCB_SINCE= # e.g. 2025-01-01
EVAL_LIVECODEBENCH_REVISION= # pin a dataset revision (any dataset: EVAL_<NAME>_REVISION)New commands: --estimate, --batch-probe, --reload.
