Skip to content

Release - v2.1.0

Choose a tag to compare

@Maximo101 Maximo101 released this 08 Nov 12:22
· 53 commits to main since this release
cfb03f2

πŸš€ llama-swap-llm-benchmark v2.1.0 β€” faster full runs, rankings you can still trust

A full run of ~28 models is almost entirely the GPU generating eval answers. v2.1 cuts that time down without letting a weaker model rank higher. It does this by asking fewer questions of models that are clearly losing, never re-asking a question it already has an answer for, and running several questions at once where the server can take them. A timing report shows where every minute went, and --estimate predicts a run before it starts.

✨ Key New Features

⏱️ Timing report and --estimate

  • Every model now records its seconds per phase: settle, load, speed tests, prefill-at-length, agent TTFT, cooldown, eval load, long context, and each dataset. Thinking and answer tokens are counted too.
  • Where it shows up:
    • results.json: models[].timing, plus run.timing with the speed/eval phase wall times, the per-dataset totals and the token counts.
    • CSV columns time_*_s and eval_tok_*.
    • A new dashboard section, Where the time went.
  • python benchmark.py --estimate predicts a run's duration from earlier runs' JSON (measured tok/s, prefill speed, seconds and tokens per question) and any batch-probe result. Models with no history get a rough guess. A one-line estimate is also printed at the start of every run.

πŸ”» Funnel profile with confidence-interval gates (EVAL_PROFILE=funnel)

  • Every model answers quick. After each stage, a model goes on only if its upper bound reaches the lower bound of the EVAL_LEADERS-th model. At least EVAL_LEADERS models always go on.
  • Survivors answer the next stage (basic β†’ moderate β†’ … β†’ EVAL_FUNNEL_FINAL, default standard). They are asked only the new questions.
  • A plain "top 5 after quick" was tested and rejected. In a simulation with a crowded top, it screened out the true best model in 18–33% of runs. The confidence gates lost it in ≀0.1% of runs, picked as well as a full run, and saved ~40–50% of the eval work.
  • A screened-out model keeps its stage score but is not ranked and never reaches the podium. First-stage avg compares every model on the same first-stage questions.

πŸ’Ύ Result cache and resume

  • Every answered question is stored in results/cache/. A rerun, or a run restarted after a crash, asks only what is missing.
  • An answer is reused only when all of these are identical:
    • the model file (size plus a hash of its first and last MiB)
    • the launch flags and llama.cpp build
    • the loaded context, slots and chat template
    • the hardware
    • the exact request and sampling settings
    • the thinking budget and the retry, salvage and forcing switches
    • the grader version
  • Speed tests are always measured again.

⚑ Batched evaluation that cannot change a ranking

  • Questions in flight are now capped per model to the server's slots (/props total_slots). With a shared KV pool (--kv-unified, the llama.cpp default when -np is auto) they are also capped to what fits the pool in the worst case. The cap is printed whenever it applies.

  • -kvu / -no-kvu are read from the launch command, and a new Unified KV column shows the result.

  • When any question ran concurrently, the Agent score uses a single-stream s/q estimate for every model: tokens Γ· speed-test tok/s, plus prompt tokens Γ· prefill tok/s, plus grading time. Measured latency under load includes queueing, so it is no longer used for ranking in that case.

  • New python benchmark.py --batch-probe asks the same questions at 1/2/4/8 in flight. It reports:

    • the throughput gain
    • per-question latency
    • accuracy
    • how many answers agree with the one-at-a-time answers
    • errors and peak VRAM

    It then recommends a level.

πŸ”€ Pipelined grading

  • Sandboxed code runs while the next question is generating. The sandbox itself is unchanged: one isolated container per program.
  • This replaces the warm container pool from the plan, which would have let one program affect the next.

πŸ™… Skip evals of models that cannot be deployed (EVAL_SKIP_INELIGIBLE=auto)

  • Models below MIN_CTX, failing a budget rule, or above GOLDILOCKS_MAX_VRAM_PCT are no longer evaluated. They were off the podium anyway. They stay in the table with the reason.

πŸ“œ Long context: one shared document per length

  • Each length now builds one document. The questions about it ask different vaults and reuse the document from llama-server's prompt cache.
  • The first, cold question doubles as the prefill-at-length measurement. A PREFILL_CTX_TESTS length that is also a LONGCTX_LENGTHS length is no longer sent twice.

🧠 Thinking budgets and budget forcing (opt-in)

  • EVAL_THINK_BUDGET_<DATASET> sets a per-dataset thinking budget.
  • EVAL_BUDGET_FORCING=true: at the cap, the reasoning is kept and closed, and the model is asked for its answer instead of a No-Thinking retry. This uses llama-server's /apply-template and /completion. The method comes from s1: Simple test-time scaling (Muennighoff et al., 2025).
  • Both change what is measured, so both are off by default and recorded in results.json.

πŸ’¬ Dashboard: tooltips, column chooser and dataset links

  • Every column header has a tooltip: what it measures, where the number comes from and whether higher or lower is better. Eval columns and dataset names also say how big the test is and how it is scored. Budget-rule columns show the rule's limit.
  • Hide the columns you don't need:
    • Click the Γ— in any header to hide that column.
    • The new Columns panel toggles single columns or whole groups (Model, Launch, VRAM & memory, Power & cost, Speed, Evals, Long-ctx, Budget, Scores).
    • Only this group shows just the model columns plus one group, for example an evals-only view in one click.
    • Show all columns brings back the full table.
  • The column choice is remembered in your browser, so later dashboards open with the same view. Hiding a column changes only the view, never the data or the scores.
  • Test names in the page header link to their datasets. Each dataset name in the header and above the accuracy charts opens the Hugging Face page it was loaded from (the pinned revision when EVAL_<DATASET>_REVISION is set). Built-in tool calling links to its source file.

πŸ” Rebuild a dashboard without a run (--reload)

  • python benchmark.py --reload rebuilds the dashboard of the newest finished run (results/results.json) with the current dashboard code. No server or GPU is needed.
  • python benchmark.py --reload results/benchmark_<stamp>.json reloads a specific run.
  • Nothing is measured or rescored: every number, score and ranking is the one the run saved, and the run's own settings are used rather than your current .env.
  • It writes results/benchmark_dashboard_<run>_reload.html (the run's own dashboard is kept) and updates benchmark_dashboard.html.
  • It is mainly a tool for working on the dashboard. A run writes its JSON when it finishes (with JSON=true, the default), so an interrupted run can't be reloaded.

🧩 LiveCodeBench

  • EVAL_LCB_SINCE=2025-01-01 keeps only newer problems, which follows LiveCodeBench's own contamination method.
  • A load-time guard refuses a copy whose questions are in Korean. The bzantium/livecodebench card mentions a Korean translation; the data itself is English.
  • EVAL_<DATASET>_REVISION pins any Hugging Face dataset to a commit.

🐞 Fixes

  • A weaker model could rank first when one of its datasets failed (high). The eval average was taken only over the datasets that produced a score. Example: every GPQA request hits a template error. That model was then averaged over easier datasets, and outranked a model that was stronger on everything it answered (Agent 92.8 vs 88.0). Now a model is ranked only with a score on every dataset of the run (β‰₯90% of each dataset's questions answered without an API error). Otherwise it shows "not ranked: incomplete evals (GPQA 0/96 answered)".
  • Profiles were not nested. MATH-500's quick questions were not a subset of standard's. Every profile now takes a prefix of one seeded question order per dataset.
  • The first question of each model carried the model load in its latency. A warm-up request now loads the model first. The load is timed separately (eval_load).
  • --kv-unified was not reflected. The docs said --parallel always divides the context.
  • Long context at the full loaded context failed with HTTP 400. Found in a real run: 128k at ctx 131,072 produced prompts of 131,644–132,331 tokens, so two models had no 128k result. The test now keeps 3% of the context back. If the server still reports "exceeds the available context size", it shrinks the document once and retries.
  • Long context scored 0% for models that think even when told not to. Found in a real run: LFM replied with reasoning only and an empty answer at 64k and 128k, so its v2.0 long-context score (and the Agent score blended from it) was an artefact. Such a reply is now asked once more with 4,096 tokens; it is noted and marked thinking_retry.
  • The batch probe understated the gain. It divided total tokens by wall time, so one long reply finishing alone at the end dominated. It now reports the steady gain (Little's law), writes every answer to batch_probe_<stamp>_details.jsonl, reads ctx/slots/KV from the server, refuses more than 4 models and warns when another run is still using the server. Agreement is only a rule from 40 questions up.
  • --estimate no longer counts eval time for models that would be skipped, and is calibrated on the tokens per question measured in a real run.
  • Small fixes: the Agent tie margin counted questions from datasets with no score, and long-context requests let EVAL_EXTRA_BODY override stream.

πŸ”’ Security

  • EVAL_CODE_SANDBOX=auto no longer falls back to an unisolated subprocess when Docker is missing or fails its self-test. The run stops with the reason. Set EVAL_CODE_SANDBOX=subprocess to accept it explicitly.

βš–οΈ How old and new results compare

  • Complete eval results, one request at a time: Agent, Chat, Goldilocks and Sweet-spot scores use the same formulas as v2.0.
  • Questions differ from v2.0 (nested sampling). Per-question results are not row-for-row comparable, and scores move within the margin of error. The statistical precision is the same.
  • s/q is slightly lower for each model's first dataset, because the model load is no longer counted (β‰ˆ load_s Γ· questions). Slow-loading models gain a little speed credit as a result.
  • Models with an incomplete dataset are no longer ranked. Before, they were averaged over the rest.
  • With EVAL_CONCURRENCY > 1, the Agent score uses single-stream s/q. At concurrency 1 nothing changes.
  • Long-context % comes from one document per length: different needles and positions than v2.0, within noise.
  • With EVAL_LCB_SINCE set, LiveCodeBench scores cover newer, harder problems and are not comparable with runs without it.
  • Funnel runs: screened-out models have no Agent score.
  • Dashboard changes and --reload affect only how results are shown. Scores and rankings are unchanged.

πŸ”„ Updating

  1. Pull the new version. There are no new dependencies (pip install -r requirements.txt is unchanged).
  2. Copy the new settings you want from .env.example into your .env. Every new setting has a safe default.
  3. If you run code datasets without Docker, add EVAL_CODE_SANDBOX=subprocess, or the run will stop.
  4. Optional: run python benchmark.py --estimate before your next run, and --batch-probe before raising EVAL_CONCURRENCY.
  5. results/ (including the new cache/) stays git-ignored. Don't publish it after a GPQA run.

βš™οΈ Configuration

EVAL_PROFILE=funnel              # or test / quick / ... as before
EVAL_FUNNEL_FINAL=standard
EVAL_LEADERS=5
EVAL_FUNNEL_CONF=0.95
EVAL_SKIP_INELIGIBLE=auto        # auto | true | false
EVAL_RESULT_CACHE=true
EVAL_RESULT_CACHE_DIR=           # default results/cache
EVAL_SQ_BASIS=auto               # auto | measured | single
BATCH_PROBE_LEVELS=1,2,4,8
BATCH_PROBE_QUESTIONS=32
ESTIMATE_GPU_BW_GBS=900
EVAL_THINK_BUDGET_GSM8K=         # per-dataset budgets, opt-in
EVAL_BUDGET_FORCING=false
EVAL_LCB_SINCE=                  # e.g. 2025-01-01
EVAL_LIVECODEBENCH_REVISION=     # pin a dataset revision (any dataset: EVAL_<NAME>_REVISION)

New commands: --estimate, --batch-probe, --reload.

image