Skip to content

Releases

v2.4.1: `--diff-runs`, what changed between runs

Choose a tag to compare

@Maximo101 Maximo101 released this 20 Nov 06:36
aea1cc9

🧰 Local-LLM-Benchmark v2.4.1: --diff-runs, what changed between runs

A report that follows the same models across finished runs and says which changes are real and which are noise. Nothing is benchmarked, and no score changes.

✨ What changes

🔍 --diff-runs

  • python benchmark.py --diff-runs compares the two newest finished runs; --diff-runs 4 the four newest; --diff-runs A B [C ...] these runs (a stamp, a benchmark_<stamp>.json / rescored_<stamp>.json, or a folder with results.json). The first run is the baseline.
  • Per model: rank and Agent score in every run (with a trend line from three runs), the change in Agent quality, each dataset's accuracy, long context per length, generation, prefill, prefill at 64k and at depth, agent TTFT, seconds per question, load time, VRAM peak, GPU power and energy per token. Launch changes are listed (context, quant, KV type, --n-cpu-moe, ngl, batch sizes, flash attention, llama.cpp build, sampling), and so are the run settings that differ.
  • Noise first. Every change from the baseline to the latest run is marked ↑ better, ↓ worse or ≈ within noise:
    • accuracy is paired when both runs' eval_details_<stamp>.jsonl are found (exact McNemar test on the questions that flipped; the result cache and seed give the same questions), otherwise an unpaired 95% Agresti-Caffo interval. A details file recorded under the server's path is also looked for next to the JSON, so a report built on the main PC over SMB still pairs;
    • tokens/s and seconds: at least 5%, or twice the larger run-to-run spread of the timed samples; power and energy 5%; VRAM 0.1 GB; Agent score and quality 0.5 points;
    • rank agreement between consecutive runs as Kendall's τ-b.
  • Honest about comparability. Runs that asked or graded the questions differently (the checks --compare refuses on) get a "Not like-for-like" banner instead of a refusal. Runs scored by different versions are noted, and --diff-rescore scores every run with this version first (in memory; no file is written but the report). The report says that the Agent score and quality are relative to each run's best finalist.
  • Output: results/run_diff_<stamp>.html (self-contained, no CDN, a Content-Security-Policy that blocks any network access; sortable tables, a model filter, an "only changes beyond noise" switch; light and dark), results/run_diff_<stamp>.json with every number, and a terminal summary with the biggest movers.
  • The report reads verdicts and aggregates only. No question, reply or reference answer is read into it, so it is safe with GPQA enabled; it still lives in the git-ignored results/ folder.
  • --forget-run leaves diff reports alone (they belong to no single run).

🧪 Tests

  • selftests/run_factory.py writes synthetic finished runs with the real JSON writer, plus matching per-question details, with planted changes.
  • selftests/test_v241_rundiff.py (21 tests) covers:
    • the three statistical tests against known values;
    • that the planted changes are found and the 2-question wobble is not;
    • the unpaired fallback;
    • a run diffed against itself reports nothing;
    • three runs;
    • settings and version mismatches, --diff-rescore, and that no run file is touched;
    • run selection;
    • HTML escaping of model names, no question text in the report, the page script parsing under Node, and the table shape;
    • the command line;
    • an end-to-end diff of two evals-only runs against the mock llama-swap server.
  • 360 selftests pass, 1 skipped.

How old and new results compare

No scoring, grading, prompt or cache-key change: every run scores exactly as with v2.4.0. GRADER_VERSION is unchanged and the result cache stays valid.

Credits

New methods credited in the README: McNemar (1947), Agresti and Caffo (2000), Kendall (1938, 1945). No dataset, library or third-party code was added.

v2.4.0: a fair Agent score across runs, `--rescore`, steadier long context

Choose a tag to compare

@Maximo101 Maximo101 released this 20 Nov 05:20
b6a5e13

🧰 Local-LLM-Benchmark v2.4.0: a fair Agent score across runs, --rescore, steadier long context

✨ What changes

⚖️ Long context: fixed lengths per model (AGENT_LONGCTX_BASIS)

  • The problem. The long-context part of the Agent score used only the lengths every ranked model ran.
  • Now: each model is judged on the lengths in AGENT_LONGCTX_BASIS (default 64k,90k) that its loaded context fits. A 64k model is judged on 64k; a 90k or 128k model on 64k and 90k. The context factor (AGENT_CTX_CURVE) still carries the preference for longer context, and the usable-context rule is unchanged.
  • The tie-zone bar is still the best finalist, but only models judged on every basis length can set it. Otherwise a 64k model with a perfect, easier 64k test could raise the bar for everyone.
  • 128k is not in the default basis. A 128k model that does badly at 128k already loses through its usable context (it counts as a 90k model). Set AGENT_LONGCTX_BASIS=all to count every tested length, or common for the v2.3 rule.

🎯 Long context: more questions near the pass mark (LONGCTX_BORDERLINE_TRIALS)

  • With 8 codes per length, 75% is the step between 6/8 and 5/8: one code decides whether a model's usable context is 64k or 90k.
  • Now a result within one code of AGENT_CTX_PASS_PCT (8 codes at 75%: 5/8 or 6/8) is asked LONGCTX_BORDERLINE_TRIALS more questions (default 2, i.e. 8 more codes) before it stands. The extra questions reuse the cached document, so they cost generation time only.
  • The codes for the extra questions are always in the document, so it is the same document whether or not they are asked. The log notes when it happens (64k: 5/8 is within one code of 75%, asked 8 more). The per-length count is stored as longctx_<length>_n and used for the error.

🔁 --rescore

  • python benchmark.py --rescore [JSON] scores a finished run again with this version's rules. It keeps every measurement and answer and prints old vs new rank, Agent score and quality per model.
  • It writes rescored_<run>.json and its dashboard. The run's own files, results.json and the latest dashboard are left alone.
  • A scoring setting the run did not record (AGENT_LONGCTX_BASIS for runs before v2.4) comes from today's .env and is named in the output.

🧹 Housekeeping

  • --forget-run STAMP removes one run's files from OUTPUT_DIR and its rows in the history CSV (.bak kept). If results.json was that run, the newest other run takes its place. Without --force it only lists what would go. The result cache is kept.
  • --compare now also refuses runs graded differently: strict tool calls, code-test timeout, LiveCodeBench tests per problem and date, extra request fields, long-context thinking. Runs too old to record a setting are not refused for it. Different long-context trials, keys or borderline trials are noted.
  • --estimate:
    • The funnel's pass share was a fixed 50%. With no earlier funnel run it assumes 85%.
    • A --resume run no longer counts the speed test (and the long-context test) of models the rows file already holds.
  • "GLM shown as Dense" is struck from the plan: both November runs label it MoE (read from the GGUF header).

🔁 How old and new results compare

  • Stored results do not change. --reload keeps stored scores; --rescore shows the new ones.

    Model Rank old → new Agent score old → new Why
    Groot 1 → 1 93.80 → 94.47 its 90k counts
    Qwen3.6-MTP 10 → 9 86.46 → 87.08 its 90k counts (100%)
    Qwen3.6 Q4_K_M 9 → 10 86.83 → 86.83 unchanged
    Ornith-9B / Qwen3.8-27B 12 / 11 → 11 / 12 83.22 → 83.84 / 83.73
    flint 13 → 14 82.55 → 76.63 its 90k is 87.5%
    Wicked-Stuff 15 → 15 77.32 → 71.47 its 90k is 75%
    Empero-Qwen3.8-9B 18 → 16 64.45 → 71.16 its 90k is 100%
    The other 10 same rank +0 to +0.6

    Every change comes from the model's own 90k answers, now counted. With this, the 16 Nov and 19 Nov rankings agree on the order of the models in both runs.

  • New runs:

    • Long-context results near the pass mark get more codes, so a borderline model's usable context is decided on 16 codes instead of 8.
    • The long-context document holds the extra codes, so v2.4 long-context results are not like-for-like with v2.3.x runs. They are never cached anyway.

⚠️ Worth knowing

  • New settings:

    • AGENT_LONGCTX_BASIS (default 64k,90k; common = v2.3; all)
    • LONGCTX_BORDERLINE_TRIALS (default 2; 0 = off)

    Both are in .env.example PART 3.

  • --compare of a v2.4 run with an older run uses the v2.4 rule for both: each model's own lengths.

Credits

  • No dataset, benchmark method, library or other third-party work is added or changed. The README credits section is unchanged.

v2.3.7: fairer long prompts, a steadier long-context test, cheaper cache changes

Choose a tag to compare

@Maximo101 Maximo101 released this 19 Nov 21:45
49ad769

🧰 Local-LLM-Benchmark v2.3.7: fairer long prompts, a steadier long-context test, cheaper cache changes

✨ What changes

📏 Long prompts are sized without the chat template's own tokens

  • The benchmark builds its long prompts (prefill at 64k/90k/128k, prefill at depth, the 20k agent prompt, the long-context document) from a measured tokens per word. v2.3.6 measured it as prompt tokens ÷ words of one 1,200-word prompt, so the template's fixed tokens were counted as words.
  • Most templates add 10-20 tokens and the error was about 1%. Cyber-Tiel-Coder's template adds ~180 tokens to every request (a default system prompt), so it read 1.33 tokens per word instead of ~1.18 and every long prompt came out about 12% short: its "128k" long-context test was 111k tokens (others ~125k), its 90k test 80k, its agent prompt 18.1k instead of 20k.
  • Now the figure is the slope between the short tech prompt and the prefill prompt, so the template's tokens cancel out. The log says so when a template adds 64 tokens or more ([info] the chat template adds ~180 tokens to every prompt), and the JSON stores it as prompt_overhead_tokens.

🧷 The long-context document stays the same between runs

  • The seeded document's length, and so where its codes sit, followed each run's measured tokens per word (1.172 one run, 1.177 the next), so most models got a slightly different document every run. The figure is now rounded up to a 0.02 step before the document is built: the same document while the measurement wobbles, and never longer than before (at most ~1.7% shorter).
  • It does not remove llama.cpp's own run-to-run differences: on 2026-11-19 Bolt-E4B dropped one of 8 codes at 64k with an identical document. For a steadier long-context score now, set LONGCTX_TRIALS=4 (16 codes per length instead of 8). The extra questions reuse the cached document, so they cost generation time only.

♻️ Cached answers survive a change of thinking budget

  • The thinking budget and max_tokens are part of each question's cache key, so changing EVAL_THINK_BUDGET (or the context, which moves the auto budget) meant asking everything again.
  • Now an answer that came out of one attempt that ended on its own (no thinking cap, timeout, loop or length cut) is reused under another budget when it used no more thinking tokens and no more tokens than the new limits. Neither limit was reached, so it could not have changed. Capped, retried or cut answers are asked again.
  • The result line shows it: [96 from cache, 12 asked under another thinking budget].
  • Only answers stored by v2.3.7 or later carry the extra key (akey in the cache file); older answers keep working as before.
  • Why it matters: on the 2026-11-16 fresh run, 111 of 7,386 questions (1.5%) hit the 32k thinking cap or the 600 s timeout. They took 19% of the question time and 25 of them ended correct. Only 1.2% of correct answers thought for more than 16k tokens. A lower budget is now a cheap experiment instead of a full re-run.

⚠️ A llama.cpp update is announced before it empties the cache

  • The llama.cpp build stays part of every model's fingerprint (a new build can change answers), so an update means every question is asked again.
  • --estimate reads the current build from a model llama-swap already has loaded (nothing is loaded for it), compares it with each model's last results and leaves those models' cached answers out of the estimate: llama.cpp build changed (b12316 -> b12407): the result cache will NOT be reused for 22 of 22 model(s). With no model loaded it cannot tell and says nothing.
  • The run checks again after the speed test, when every model's build is known, and prints a [warn] line before the evals start.

🔧 Small things

  • Tool calling: a model that calls a tool although a required value is missing (the missing cases) is now told called a tool although a required value was not given (it should ask the user); it used to get the irrelevance wording "needs none". Grading is unchanged. Answers already in the cache keep their old text until asked again.
  • The "in flight: 3 (capped: KV pool ... -kvu unknown)" line was printed before every dataset of every stage (16 times per model). It is now printed once per model, as In flight: 3 per dataset (...), and again only if the number changes.

🔁 How old and new results compare

  • Scoring code is unchanged. Rescoring the 2026-11-19 run gives the stored scores exactly. Stored results, --reload and --compare are unaffected.
  • New runs measure slightly different prompts:
    • Models with a plain template: long prompts about 1% longer, closer to the target lengths (most landed 1-3% short).
    • The long-context document changes once for every model (new rounding), so long-context results of the first v2.3.7 run are not like-for-like with v2.3.6 runs. They are never cached anyway.
  • Result cache: nothing is invalidated. Question keys are the same as v2.3.6; new answers just carry one extra key.

Credits

  • No dataset, benchmark method, library or other third-party work is added or changed. The README credits section is unchanged.

v2.3.6: tool calling grows with the profile, finalists-only accuracy charts

Choose a tag to compare

@Maximo101 Maximo101 released this 15 Nov 10:20
51a5819

🧰 Local-LLM-Benchmark v2.3.6: tool calling grows with the profile, finalists-only accuracy charts

Two changes from the first full v2.3.5 baseline on the server (2026-11-15, 22 models, funnel to moderate, resumed twice after UPS shutdowns), plus the runner fix from the v2.3.5 review.

✨ Key New Features

🛠️ Tool calling: 96 cases, more as the profile grows

  • The built-in tool-calling set grows from 32 to 96 cases. A stage now asks max(32, its rows) of them: 32 at quick and basic, 43 at moderate, all 96 from standard up. The cases are asked in a fixed order, so every stage is a prefix and the first 32 are always the v2.3 core set.
  • MIN_TOOLCALL_PCT is still decided on those 32 at the first funnel stage (and checked again on the final count when the run ends).
  • Two new categories, after the ideas of BFCL V3 (in a much simpler single-step form):
    • missing: a tool fits but the user left out a required value (no e-mail address, no room, no repo). The right answer is to ask, not to call with a made-up value.
    • chained: the conversation already holds the model's earlier call and the tool's result; the next call must use that result (convert the share price it just got, read the newest of the listed files, set the thermostat only because it is 11 °C), or no call when the result already answers the question.
  • Harder cases in the old categories: close tools side by side (forecast vs current weather, exchange rate vs conversion, turn off vs turn on, search files vs list a directory), three calls in one reply, mixed tools in parallel, dates relative to a given "today", numbers in words ("two and a half thousand", "a quarter of full volume").
  • New tool definitions (forecast, lights off, locks, exchange rate, file search/write, shell command, Docker restart/logs, restaurant booking, GitHub issue, volume, calendar lookup, order status, meetings) are only offered in the new cases.
  • multiple-10 is replaced by multiple-10b. The old case ("How do you say 'thank you' in German?" with a translate tool offered) was answered directly by 10 of 22 models. Answering a well-known fact yourself is exactly what the irrelevance cases reward, so the case measured style, not tool use. The new one asks for the translation tool explicitly.
  • The funnel gate counts tool calling's error while the final stage will still ask a model more tool cases (it used to be complete at the first stage). Bounds get slightly wider: the safe side.

📊 Accuracy charts show the finalists only

  • With the funnel, a model that stopped early answered only that stage's questions (11 per dataset at quick) while the finalists answered 43, so flint's and LFM2.5 Q4_K_XL's 11/11 sat above finalists' 41/43. Such models are now left out of the dataset accuracy / pass-rate / tool-calling charts, the Quality vs speed chart and the quant table's eval average.
  • A line above the charts says how many stopped and where; Show them brings them back faded, with [quick] after the name and the reason in the tooltip.
  • Long-context retrieval is asked in full at the first stage, so its chart keeps every model. The table and the Agent leaderboard are unchanged. Display only: no score changes.

🔁 How old and new results compare

  • Existing results do not change. The 22 models of the 2026-11-15 run rescored under v2.3.6 give the same Agent scores, quality and rank states as stored (--reload keeps stored scores anyway).
  • The gate: replayed on the 2026-11-15 run with its own answers: the same 18 of 22 models pass quick and 18 of 18 pass basic. The one screened-out model (Empero-Qwen3.8-4B) moves from 0.94 to 0.18 points below the bar.
  • New runs: tool-calling percentages are not like-for-like with earlier runs. Finalists of a moderate funnel answer 43 tool cases instead of 32, 11 of them harder. --compare notes the different question counts. How much the new cases move the ranking can only be seen after a run.
  • multiple-10b, simulated on the 2026-11-15 run (counting it as passed for the 10 models that answered multiple-10 directly): every other model keeps its relative order, but Empero-Qwen3.8-9B drops from #2 (89.4) to #11 (85.2) and Nemotron-3.5-Lightning from 85.6 to 81.6. Neither model's own answers change. The best composite rises, and both fall just outside the tie zone. That is the honest result of removing a case that penalised direct answers, and it also shows how compressed the field is: one tool case moves a model nine places.
  • Result cache: 31 of the 32 core cases keep their cache keys. A first v2.3.6 run asks multiple-10b of every model and the 11 new cases of every moderate finalist. Tool cases are short (about 130 tokens), so this is roughly a minute per model.

⚠️ Worth knowing

  • The missing and chained cases follow the system prompt the tool always sends ("call it with exactly the arguments the user gave"). A model that fills a required value with a guess fails them. That is deliberate for agent use.
  • Chained cases send assistant messages with tool_calls and tool messages. llama.cpp with --jinja should render these for any template with tool support (not checked on every model here). If a model scores 0 on every chained case, look at those requests in eval_details_*.jsonl before blaming the model.
  • The defaults are unchanged: EVAL_FUNNEL_FINAL defaults to standard, so a default funnel run asks all 96 tool cases of its finalists.

Fixes

  • evals/runner.py held its second half twice (the pause work merged onto a pre-v2.3.3 runner). Runs used the older gate, so hard-rule models were logged as drop and stored as screened out. Fixed. Gate decisions and scores were not affected. selftests/test_v236_runner.py guards it (from the v2.3.5 review).

Credits

  • The tool-calling categories and AST-style matching follow the Berkeley Function Calling Leaderboard (Patil, Mao, Yan, Ji, Suresh, Stoica, Gonzalez, ICML 2025, PMLR 267:48371-48392; Apache 2.0). The new categories follow BFCL V3's Missing Parameters and multi-turn ideas. No BFCL data is used: all 96 cases were written for this tool.

v2.3.5: settings named for their role, a 90k long-context length, safer defaults

Choose a tag to compare

@Maximo101 Maximo101 released this 13 Nov 04:05
2ac16b2

🪟 llama-swap-llm-benchmark v2.3.5: settings named for their role, a 90k long-context length, safer defaults

A tidy-up release before a fresh baseline run: the endpoint settings no longer carry product names, long-context gets the 90k step the 27B models are loaded at, and the dashboard shows VRAM pressure the way you read it. Nothing about how a model is measured or graded changes, so existing Agent scores are identical (rescored: 26 server + 7 main-PC models from the 2026-11-12 runs).

✨ Key New Features

🏷️ Settings renamed (old names keep working)

  • LLAMA_SWAP_URL / LLAMA_SWAP_API_KEY are now SERVER_URL / SERVER_API_KEY (any OpenAI-compatible server works, llama-swap is one of them).
  • STRATA_URL (and the older STRATA_SWAP_URL), STRATA_API_KEY, STRATA_MODEL, STRATA_PARAMS_B, STRATA_ACTIVE_B, STRATA_QUANT, STRATA_CTX are now SECONDARY_*.
  • An old name in .env, a favourite or the shell still works and prints one line saying what it is called now; --check-env lists it; the settings page renames it when you save. If both names are set, the new one wins.
  • Not renamed on purpose: names inside result files (strata_api_key, strata_details, columns) so older results reload and compare; --probe-strata; the words "llama-swap" and "Strata" in text.

📏 Long-context at 90k

  • LONGCTX_LENGTHS defaults to 65536,92160,131072 (new longctx_90k_pct column; the history CSV is rewritten under the new header, a .bak is kept). A length larger than the model's loaded context is skipped as before, so a model loaded at -c 90112 (88k) is not asked the 92160 length.
  • The tooltip on every context-size setting now reads: "Suggested context sizes: 4096 (4k) ... 1048576 (1M) | Enter the whole number (bracket value are just for help - do not use them in the fields)".

🛡️ Defaults

  • MAX_GPU_TEMP=85 (was off). The run pauses at 85 C and carries on 10 C lower. With the server on another machine, or no readable GPU sensor, the existing "not enforced" warning is printed. MAX_GPU_TEMP=0 turns it off.
  • EVAL_SANITY_PCT stays 0 (off). The settings page now has a guide under it: what it does, and how the rate halves per model while answers match (20 -> 10 -> 5 -> 2 -> 0) and resets on a difference.

🎨 Dashboard

  • VRAM bar above VRAM_SWEET_HIGH: red, darkening evenly to black at 100% (no pink step). The hazard sign still starts at VRAM_SWEET_MAX.
  • Ctx and Max ctx tooltips give the exact size: 90k (92160 tokens).

🔁 How old and new results compare

  • Agent scores, usable context and rank states of existing result sets are identical under v2.3.3 and v2.3.5 (checked on the 33 models of the 2026-11-12 runs; they hold no 90k results).
  • New runs with a 90k result: a model loaded at 92160 or more that passes 90k but fails 128k now has usable context 92160 (context factor about 0.978) instead of 64k (0.95), a small rise it has earned. A model at 92160 or more with no 90k result counts as "not passed" and drops to 64k, so --compare of an old run against a new one can penalise it; re-run the old one. The composite stays on 64k while any ranked model is loaded at 64k or 90112.
  • The result cache is not keyed on the tool version, so v2.3.5 reuses a matching v2.3.x cache. For a clean baseline move results/cache/ aside (or set EVAL_RESULT_CACHE=refresh for the first run).

⚠️ Worth knowing

  • Load the "90k" models at -c 92160, not 90112, if you want them in the 90k test and the same dashboard label.
  • The long-context borderline cliff (8 retrievals per length) and the v2.4 MMLU-Pro tie-breaker are unchanged from v2.3.3.
  • No new dataset, library or third-party work: the README credits section is unchanged.

Fixes / Security

  • None in this release (no security-relevant change).

v2.3.3: the funnel says what happened to every model

Choose a tag to compare

@Maximo101 Maximo101 released this 12 Nov 23:05
96b153d

🪟 llama-swap-llm-benchmark v2.3.3: the funnel says what happened to every model

Real runs (2026-11-12, two PCs, 33 models) showed the funnel pruned 1 model of 26, and that one (flint: 100% on the evals, 0% at 128k) was labelled "screened out" although it could never have had an Agent score. v2.3.3 separates rules from comparisons.

✨ Key New Features

🚦 Three explicit states: Ranked, Screened out, No Agent score

  • Every evaluated model ends in exactly one state: Ranked (has an Agent score), No Agent score (fails a hard rule) or Screened out (the funnel gate dropped it as clearly below the leaders). rank_state / rank_reason are in results.json (scores group), shown as a badge on the model name, a Rank filter (with counts), the leaderboard note, the legend and the markdown report. If an older result is both, No Agent score wins.
  • Hard rules are decided outside the funnel: loaded or usable context below AGENT_CTX_MIN, long-context below AGENT_CTX_PASS_PCT at every length that counts, tool calling below MIN_TOOLCALL_PCT. They are plain thresholds on the model's own numbers: no estimate, no margin of error, never compared with other models, and a failing model no longer lifts the leaders' bar.
  • EVAL_HARD_FAIL=stop|continue (new, default stop): a model that fails a hard rule at a gate is not asked further stages. continue keeps asking, so its later-stage scores exist.

🔬 A funnel that prunes

  • The gate's bounds use the reducible error (the evals part; tool calling and long-context are asked in full at the first stage and do not shrink) plus EVAL_FUNNEL_MARGIN points (new, default 3). Rule: score + z × SE_evals + margin ≥ score_k − z × SE_evals_k for the EVAL_LEADERS-th model. This replaces the EVAL_FUNNEL_CONF=0.85 stopgap.
  • Decimal gaps: the log shows, per model and gate, KEEP / SCREENED OUT / NO AGENT SCORE, score ± the error the gate used (and the error of all parts) and the signed gap to the leaders' bar to two decimals. evaluation.funnel.gates in the JSON carries state, se_red, ub, lb, gap; the run header records margin and hard_fail.

🧠 Thinking budget: auto is the baseline

EVAL_THINK_BUDGET=auto was already the code default and is the documented norm. If your .env still has EVAL_THINK_BUDGET=16384, delete the line: the first run then re-asks the questions whose budget changed (it is part of the result-cache key), and --compare refuses to mix runs with different budgets.

🔁 How old and new results compare

  • Agent scores of an existing result set do not change (python benchmark.py --reload on the 2026-11-12 runs gives the same score_agent for every ranked model). Only labels change: on the server run flint and LFM2.5-24B-UD-Q3_K_XL now show No Agent score (long-context 50% / 62% at 64k, below the 75% pass mark) instead of flint's "screened out"; 24 models stay Ranked.
  • New runs can screen out more models (that is the point), but only models clearly below the leaders, and a model's stage scores are unchanged. No weaker model can rank higher because of this change: a model that is dropped is never ranked, and hard-rule models never enter the bar.
  • One run set (v2.3.1) is the only evidence for the margin: nothing within 3 composite points of the final 5th-best was pruned at any gate in a retrospective, and 9 models would have been. Treat it as indicative.

⚠️ Worth knowing

  • AGENT_CTX_MIN defaults to 64k, so a model that passes 64k but fails 128k keeps an Agent score (lower context factor). AGENT_CTX_MIN=131072 makes 128k mandatory but would remove 13 of 26 models of the server run (loaded at 64k-90k). The default is not changed.
  • The long-context borderline problem (8 retrievals per length, so 75% is a cliff between 6/8 and 5/8) is not addressed here.
  • The deeper MMLU-Pro tie-breaker stage is planned for v2.4.
  • New settings: EVAL_FUNNEL_MARGIN, EVAL_HARD_FAIL (in .env.example).

🙏 Credits

No dataset, benchmark method or library was added or changed.

v2.3.2 - Pause Function and adjusted thinking cap

Choose a tag to compare

@Maximo101 Maximo101 released this 12 Nov 06:44
055f347

Pause/resume, thermal guard, auto thinking budget (work on v2.3.2, 2026-11-12)

Pause / resume

  • Same-process pause controlled by run_control.json in OUTPUT_DIR (polled ~1 s). Works on Windows and for a run on another machine sharing OUTPUT_DIR. CLI: --pause, --resume (no arg + live run); settings page: Pause / Resume buttons; /api/pause, /api/resume.
  • Temporary state: run_<stamp>.pause.json + the existing run_<stamp>.rows.json. Both deleted when the run ends or resumes. --resume <run> still recovers a process lost while paused.
  • Manual pause: current model's speed phase finishes; evals stop taking new questions, in-flight ones finish and are graded. The batch probe cannot be paused.
  • Thermal: MAX_GPU_TEMP (0 = off, else 40-110), GPU_TEMP_RESUME_DELTA (default 10), GPU_TEMP_MAX_RESTARTS (default 3). Own GPU monitor thread at 500 ms (never inside token generation); trip is latched; resume after 3 consecutive cool samples. Not enforced, with a loud warning, when the server is remote or no temperature can be read.
  • Fresh restart: PauseRestart(BaseException) raised at checkpoints only inside CONTROL.restartable(). Speed test restarts wholesale; long-context restores its row snapshot; an interrupted eval DatasetRun is discarded and re-run (answered questions come from the result cache). After GPU_TEMP_MAX_RESTARTS restarts of one line, PauseGaveUp ends the run like Ctrl+C (exit 130, resumable).
  • Time paused is recorded in phases.paused_s (it is also inside speed_s / eval_s).

Thinking budget

  • EVAL_THINK_BUDGET=auto is now the default: EVAL_THINK_BUDGET_PCT (25) % of the model's own context, raised to EVAL_THINK_BUDGET_MIN (4096) and held to EVAL_THINK_BUDGET_MAX_PCT (60) % of the context (ceiling wins on a tiny context), rounded down to a multiple of 1024. Unknown context: 16384. A number or 0 still fixes it. Validation: PCT 5-90, MIN >= 256, MAX_PCT 10-90, PCT <= MAX_PCT.
  • Logged per model: thinking budget 32,768 tokens (25% of 131,072 ctx) (or "floor ..." / "ceiling ..."). Exported in results.json / dashboard meta (think_budget, think_budget_pct, _min, _max_pct); --compare refuses runs that differ.
  • Settings page: hover tip on the four keys lists the resulting budget for 4k..1M contexts from the current field values (JS twin of auto_think_budget, tested under Node).
  • Consequences: the budget is in the result-cache key, so models whose budget changed are re-asked (EVAL_THINK_BUDGET=16384 keeps old cache and comparability). A bigger budget fits fewer requests in the shared KV pool: in-flight drops (4 -> 3 at 128k in the mock). Scores of previously capped models may rise; not measured.
  • Evidence (user's 2026-11-12 server log): many models hit the 16,384 cap on ~5-15% of questions; most recovered by the No-Thinking retry (time cost), the LFM2.5 builds show "0 recovered, N counted as wrong".

Rebrand (display strings only)

"Local LLM Benchmark Suite" in the dashboard, settings page, markdown report, argparse description, .env.example header, __init__. Left for the owner to decide: GitHub repo / Project name, REPO_URL and clone URLs, tool.name in the JSON and the notify body, the markdown front-matter tag, historical release notes, regenerated demo dashboards.

Open items

  • Not verified: real GPU / NVML thermal behaviour, a real llama-server, Docker, Hugging Face.
  • Suggested commits: feat: pause/resume and GPU thermal guard, feat: auto thinking budget with floor and ceiling, fix: rebrand display strings, and docs: (README) separately.
  • No new third-party dataset, method or library: no credits/licence change.

v2.3.1: Agent score, a settings page that saves settings and runs the test, numerous improvements (v2.1 → v2.3.1)

Choose a tag to compare

@Maximo101 Maximo101 released this 10 Nov 12:21
098d910

First release since v2.1.0. It rolls up v2.2.0, v2.2.1, v2.3.0 and v2.3.1.

🚀 llama-swap-llm-benchmark v2.2.0 — reuse what you have, check it, and compare across servers

v2.1 introduced the result cache, so a rerun asks only what is missing. v2.2 adds a full retest, a sanity check of cached answers, an --estimate that knows what the cache already holds, and --compare, which puts runs from different servers in one dashboard (for example a model that only fits on Strata next to your llama-swap models). Scores are computed exactly as in v2.1.

✨ Key New Features

🔁 Full retest (EVAL_RESULT_CACHE=refresh)

  • EVAL_RESULT_CACHE now has three modes:
    • true (default): reuse cached answers; only missing or failed questions are asked.
    • refresh: full retest. Every question is asked again, and the new answers replace the cached ones, so the next run can reuse them.
    • false: no cache at all; nothing is reused or stored.
  • Speed tests and the long-context test are always measured again, in every mode.

🩺 Sanity check of cached answers (EVAL_SANITY_PCT)

  • EVAL_SANITY_PCT (0–100, default 0 = off) re-asks that share of the cached answers at random and compares each with the cached one:
    • Code and tool calls must have the same pass/fail; multiple choice and math the same extracted answer and verdict.
    • The new answer is the one scored and stored.
    • Differences are listed in the log ([sanity] <question>: cached ..., now ...), summed per model at the end of the run, and shown in a new Sanity diff dashboard column.
  • EVAL_SANITY_ADAPTIVE=true (default): each model starts at EVAL_SANITY_PCT. Every matching answer halves its rate in whole percents, and below 2% the checks stop for that model for the rest of the run (from 20%: 20 → 10 → 5 → 2 → 0). A different answer puts it back to the full rate. Models that keep answering the same are checked less and less; models that change keep being checked. Each run starts again at the full rate.
  • The questions picked differ per run (the seed is printed), so repeated runs check different questions.

🎛️ Each model's own sampling (EVAL_TEMPERATURE=server, EVAL_SERVER_SAMPLING_MODELS)

  • Until now every eval request carried EVAL_TEMPERATURE (default 0), which overrides the --temp in your llama-swap commands. Some reasoning models, Gemma-family especially, loop under greedy decoding: in a real run Gemma 4 26B and a Gemma 4 fine-tune needed a retry on 32% and 48% of questions, failed the retry budget rule and were dropped from the ranking, although every answer they finished cleanly was right.
  • EVAL_TEMPERATURE=server sends no temperature, so every model uses the sampling in its own llama-swap command, with EVAL_SEED as the sampling seed. This measures each model as you deploy it.
  • EVAL_SERVER_SAMPLING_MODELS=<regex> does this only for the matching models; the rest keep EVAL_TEMPERATURE, and only the matching models are asked again.
  • A new Sampling column shows what each model was evaluated with, e.g. temp 0 or server: temp 1.0, top-k 64, top-p 0.95.

⏱️ --estimate knows what the cache holds

  • It says whether the run will reuse cached answers or is a full re-run (refresh / false).
  • Per model, it shows how many answers it expects to reuse against the questions it expects to ask, and counts time only for the rest, plus any sanity-check re-asks.
  • The one-line estimate at the start of every run says the same.

⚖️ Compare runs in one dashboard (--compare)

  • python benchmark.py --compare combines the newest llama-swap run with the newest Strata run. python benchmark.py --compare A.json B.json ... combines the runs you name; the first is the base.
  • Every model keeps its own measurements. The scores that compare models with each other (Agent score, sweet spot, budget eligibility) are computed again over all models together, so the ranking matches a single run with all of them.
  • Runs whose eval settings differ (seed, temperature, thinking budget, streaming switches) are refused, because they asked different questions or measured differently. --force combines them anyway, and the dashboard notes it.
  • A model missing a dataset or questions shows "not ranked: incomplete evals", as within one run. A model in both runs is shown twice, labelled with its run.
  • The dashboard gets a Run column and a Run filter to show or hide each run's models.
  • Output: results/benchmark_compare_<stamp>.html, copied to benchmark_compare.html.
  • Tip: run both sides with the same eval settings. With EVAL_PROFILE=funnel, a run with a single model goes straight to the final profile, so a lone Strata model answers the same questions as a llama-swap run's finalists.

🧹 Small changes

  • Two .env.example section headings carried stale "(v2.2)" tags for settings that shipped in v2.0; removed.
  • results.json records the cache mode and sanity settings (evaluation.result_cache_mode, sanity_pct, sanity_adaptive) and, per model, sanity_checked and sanity_different. Per-question logs mark re-asked questions with sanity (same / different) and the cached answer in sanity_cached.

⚖️ How old and new results compare

  • Scoring is unchanged. Runs with the same settings give the same scores as v2.1.
  • Cached answers from v2.1 runs stay valid. The cache key is unchanged, so a v2.2 run reuses everything a v2.1.0 run answered.
  • With EVAL_SANITY_PCT > 0, a re-asked question is scored on its new answer. When decoding is not fully deterministic (batched requests, temperature > 0), a score can move by the questions that changed, the same noise a fresh run would show.
  • With own sampling (EVAL_TEMPERATURE=server or EVAL_SERVER_SAMPLING_MODELS), those models are measured with their deployed sampling instead of greedy decoding. Their answers are no longer bit-identical between runs, so small differences mean less. Models left at EVAL_TEMPERATURE are unchanged and keep their cached answers.
  • --compare recomputes the comparative scores over the combined set. A model's Agent score there can differ from its score in its own run, because the best model it is compared with can be different.

🔄 Updating

  1. Pull the new version. There are no new dependencies.
  2. Optional: copy the new settings from .env.example into your .env. The defaults behave like v2.1 (EVAL_RESULT_CACHE=true, EVAL_SANITY_PCT=0).
  3. If you used EVAL_RESULT_CACHE=false to force a full retest, use refresh instead, so the new answers are kept for later runs.
  4. Keep the llama-swap / llama.cpp image unchanged between runs you want to reuse: the llama.cpp build is part of the cache key.

⚙️ Configuration

EVAL_RESULT_CACHE=true           # true | refresh (full retest) | false
EVAL_TEMPERATURE=0               # a number, or server = each model's own llama-swap sampling
EVAL_SERVER_SAMPLING_MODELS=     # regex: only these models use their own sampling
EVAL_SANITY_PCT=0                # % of cached answers re-asked and compared (0-100)
EVAL_SANITY_ADAPTIVE=true        # halve a model's rate after each matching answer

New commands: --compare (with --force).

🚀 llama-swap-llm-benchmark v2.2.1: a settings file you can read at a glance, and a check for your .env

A tidy-up release. .env.example is reorganised so it is clear which settings you must check, which ones you change per run, and which are optional, with the built-in default shown for every one. A new --check-env compares your .env with it. Measurements, scores and the result cache are unchanged.

This prepares v2.3, which will add a settings page that reads .env.example in the same way.

✨ Key New Features

🗂️ .env.example in three parts

  • PART 1 · Your setup: check every line once per machine. Covers the server address, the RAM process pattern, the budget profile, the VRAM sweet spot and Docker model-file paths.
  • PART 2 · Your run: what you change between runs. Covers models, datasets, profile / funnel, concurrency, prefill lengths, cache, sampling, the thinking budget and output files.
  • PART 3 · Optional: everything else, grouped by topic. Every line is commented out and already at its built-in default.
  • One line per setting, in one format:
    • KEY=value # what it does [default: x]: an active line; [default] is what you get if you delete it.
    • # KEY=value # what it does: a commented line; the value shown is the built-in default.
    • # KEY= # what it does: not set by default; the text says what happens then.
  • All 141 settings the code reads are listed (plus the BUDGET_<PROFILE>_<RULE> and EVAL_THINK_BUDGET_<DATASET> families). The old file was missing EVAL_<DATASET>_REVISION for five of the seven datasets.

🩺 python benchmark.py --check-env [file]

Reads files only; it changes nothing and contacts no server. It lists:

  • Unknown settings (ignored by the benchmark), with the closest real name: EVAL_TOOLCAL -> EVAL_TOOLCALL?. Budget rules and per-dataset thinking budgets are checked by name too.
  • Values that won't work as intended:
    • tue or another non-boolean (read as false);
    • a word where a number is expected (the run would stop at start-up);
    • KEY= # comment, which python-dotenv reads as the value # comment.
  • Old names (STRATA_SWAP_URL), keys set twice (the last one wins), and keys your shell also sets (the shell wins).
  • PART 1 / PART 2 settings missing from your .env (usually new in this version), with the default they fall back to.
  • Everything you changed from the built-in defaults.

It exits with 1 if it finds an unknown setting or an unusable value, so it can run before a scheduled run.

🔧 Fixes

  • Documented defaults that didn't match the code (the code was right; the file was wrong):
    • the c...
Read more

Release - v2.1.0

Choose a tag to compare

@Maximo101 Maximo101 released this 08 Nov 12:22
cfb03f2

🚀 llama-swap-llm-benchmark v2.1.0 — faster full runs, rankings you can still trust

A full run of ~28 models is almost entirely the GPU generating eval answers. v2.1 cuts that time down without letting a weaker model rank higher. It does this by asking fewer questions of models that are clearly losing, never re-asking a question it already has an answer for, and running several questions at once where the server can take them. A timing report shows where every minute went, and --estimate predicts a run before it starts.

✨ Key New Features

⏱️ Timing report and --estimate

  • Every model now records its seconds per phase: settle, load, speed tests, prefill-at-length, agent TTFT, cooldown, eval load, long context, and each dataset. Thinking and answer tokens are counted too.
  • Where it shows up:
    • results.json: models[].timing, plus run.timing with the speed/eval phase wall times, the per-dataset totals and the token counts.
    • CSV columns time_*_s and eval_tok_*.
    • A new dashboard section, Where the time went.
  • python benchmark.py --estimate predicts a run's duration from earlier runs' JSON (measured tok/s, prefill speed, seconds and tokens per question) and any batch-probe result. Models with no history get a rough guess. A one-line estimate is also printed at the start of every run.

🔻 Funnel profile with confidence-interval gates (EVAL_PROFILE=funnel)

  • Every model answers quick. After each stage, a model goes on only if its upper bound reaches the lower bound of the EVAL_LEADERS-th model. At least EVAL_LEADERS models always go on.
  • Survivors answer the next stage (basic → moderate → … → EVAL_FUNNEL_FINAL, default standard). They are asked only the new questions.
  • A plain "top 5 after quick" was tested and rejected. In a simulation with a crowded top, it screened out the true best model in 18–33% of runs. The confidence gates lost it in ≤0.1% of runs, picked as well as a full run, and saved ~40–50% of the eval work.
  • A screened-out model keeps its stage score but is not ranked and never reaches the podium. First-stage avg compares every model on the same first-stage questions.

💾 Result cache and resume

  • Every answered question is stored in results/cache/. A rerun, or a run restarted after a crash, asks only what is missing.
  • An answer is reused only when all of these are identical:
    • the model file (size plus a hash of its first and last MiB)
    • the launch flags and llama.cpp build
    • the loaded context, slots and chat template
    • the hardware
    • the exact request and sampling settings
    • the thinking budget and the retry, salvage and forcing switches
    • the grader version
  • Speed tests are always measured again.

⚡ Batched evaluation that cannot change a ranking

  • Questions in flight are now capped per model to the server's slots (/props total_slots). With a shared KV pool (--kv-unified, the llama.cpp default when -np is auto) they are also capped to what fits the pool in the worst case. The cap is printed whenever it applies.

  • -kvu / -no-kvu are read from the launch command, and a new Unified KV column shows the result.

  • When any question ran concurrently, the Agent score uses a single-stream s/q estimate for every model: tokens ÷ speed-test tok/s, plus prompt tokens ÷ prefill tok/s, plus grading time. Measured latency under load includes queueing, so it is no longer used for ranking in that case.

  • New python benchmark.py --batch-probe asks the same questions at 1/2/4/8 in flight. It reports:

    • the throughput gain
    • per-question latency
    • accuracy
    • how many answers agree with the one-at-a-time answers
    • errors and peak VRAM

    It then recommends a level.

🔀 Pipelined grading

  • Sandboxed code runs while the next question is generating. The sandbox itself is unchanged: one isolated container per program.
  • This replaces the warm container pool from the plan, which would have let one program affect the next.

🙅 Skip evals of models that cannot be deployed (EVAL_SKIP_INELIGIBLE=auto)

  • Models below MIN_CTX, failing a budget rule, or above GOLDILOCKS_MAX_VRAM_PCT are no longer evaluated. They were off the podium anyway. They stay in the table with the reason.

📜 Long context: one shared document per length

  • Each length now builds one document. The questions about it ask different vaults and reuse the document from llama-server's prompt cache.
  • The first, cold question doubles as the prefill-at-length measurement. A PREFILL_CTX_TESTS length that is also a LONGCTX_LENGTHS length is no longer sent twice.

🧠 Thinking budgets and budget forcing (opt-in)

  • EVAL_THINK_BUDGET_<DATASET> sets a per-dataset thinking budget.
  • EVAL_BUDGET_FORCING=true: at the cap, the reasoning is kept and closed, and the model is asked for its answer instead of a No-Thinking retry. This uses llama-server's /apply-template and /completion. The method comes from s1: Simple test-time scaling (Muennighoff et al., 2025).
  • Both change what is measured, so both are off by default and recorded in results.json.

💬 Dashboard: tooltips, column chooser and dataset links

  • Every column header has a tooltip: what it measures, where the number comes from and whether higher or lower is better. Eval columns and dataset names also say how big the test is and how it is scored. Budget-rule columns show the rule's limit.
  • Hide the columns you don't need:
    • Click the × in any header to hide that column.
    • The new Columns panel toggles single columns or whole groups (Model, Launch, VRAM & memory, Power & cost, Speed, Evals, Long-ctx, Budget, Scores).
    • Only this group shows just the model columns plus one group, for example an evals-only view in one click.
    • Show all columns brings back the full table.
  • The column choice is remembered in your browser, so later dashboards open with the same view. Hiding a column changes only the view, never the data or the scores.
  • Test names in the page header link to their datasets. Each dataset name in the header and above the accuracy charts opens the Hugging Face page it was loaded from (the pinned revision when EVAL_<DATASET>_REVISION is set). Built-in tool calling links to its source file.

🔁 Rebuild a dashboard without a run (--reload)

  • python benchmark.py --reload rebuilds the dashboard of the newest finished run (results/results.json) with the current dashboard code. No server or GPU is needed.
  • python benchmark.py --reload results/benchmark_<stamp>.json reloads a specific run.
  • Nothing is measured or rescored: every number, score and ranking is the one the run saved, and the run's own settings are used rather than your current .env.
  • It writes results/benchmark_dashboard_<run>_reload.html (the run's own dashboard is kept) and updates benchmark_dashboard.html.
  • It is mainly a tool for working on the dashboard. A run writes its JSON when it finishes (with JSON=true, the default), so an interrupted run can't be reloaded.

🧩 LiveCodeBench

  • EVAL_LCB_SINCE=2025-01-01 keeps only newer problems, which follows LiveCodeBench's own contamination method.
  • A load-time guard refuses a copy whose questions are in Korean. The bzantium/livecodebench card mentions a Korean translation; the data itself is English.
  • EVAL_<DATASET>_REVISION pins any Hugging Face dataset to a commit.

🐞 Fixes

  • A weaker model could rank first when one of its datasets failed (high). The eval average was taken only over the datasets that produced a score. Example: every GPQA request hits a template error. That model was then averaged over easier datasets, and outranked a model that was stronger on everything it answered (Agent 92.8 vs 88.0). Now a model is ranked only with a score on every dataset of the run (≥90% of each dataset's questions answered without an API error). Otherwise it shows "not ranked: incomplete evals (GPQA 0/96 answered)".
  • Profiles were not nested. MATH-500's quick questions were not a subset of standard's. Every profile now takes a prefix of one seeded question order per dataset.
  • The first question of each model carried the model load in its latency. A warm-up request now loads the model first. The load is timed separately (eval_load).
  • --kv-unified was not reflected. The docs said --parallel always divides the context.
  • Long context at the full loaded context failed with HTTP 400. Found in a real run: 128k at ctx 131,072 produced prompts of 131,644–132,331 tokens, so two models had no 128k result. The test now keeps 3% of the context back. If the server still reports "exceeds the available context size", it shrinks the document once and retries.
  • Long context scored 0% for models that think even when told not to. Found in a real run: LFM replied with reasoning only and an empty answer at 64k and 128k, so its v2.0 long-context score (and the Agent score blended from it) was an artefact. Such a reply is now asked once more with 4,096 tokens; it is noted and marked thinking_retry.
  • The batch probe understated the gain. It divided total tokens by wall time, so one long reply finishing alone at the end dominated. It now reports the steady gain (Little's law), writes every answer to batch_probe_<stamp>_details.jsonl, reads ctx/slots/KV from the server, refuses more than 4 models and warns when another run is still using the server. Agreement is only a rule from 40 questions up.
  • --estimate no longer counts eval time for models that would be skipped, and is calibrated on the tokens per question measured in a real run.
  • Small fixes: the Agent tie margin counted questions from datasets with no score, and long-context requests let EVAL_EXTRA_BODY override stream.

🔒 Security

  • EVAL_CODE_SANDBOX=auto no longer falls back to an unis...
Read more

Release - v2.0.0

Choose a tag to compare

@Maximo101 Maximo101 released this 07 Nov 13:24
3a5cfc1

🚀 Release - v2.0.0

We are excited to release v2.0.0 of the llama-swap-llm-benchmark suite! v1.5 taught the tool to grade model intelligence. v2.0 makes those grades trustworthy and decisive, and measures models the way you actually deploy them: at your real context size, with your real KV cache settings, and against your machine's own VRAM budget.

d1

✨ Key New Features & Capabilities

🧠 Smarter, more reliable evaluations

  • Math evals: GSM8K and MATH-500 join GPQA Diamond, MMLU-Pro, HumanEval, MBPP and LiveCodeBench. Answers are normalised (fractions, \frac, units, thousands separators) and matched numerically.
  • Tool-calling eval (BFCL-style): 32 built-in cases (no download) send native OpenAI tools requests in four categories: simple, multiple, parallel and "should not call". Arguments are graded field by field. Calls written as plain text instead of native tool_calls are caught, which usually means a missing --jinja or a chat template without tool support. Off by default (EVAL_TOOLCALL=true).
  • Long-context retrieval: a multi-key needle test at 64k and 128k tokens, with decoys. The benchmark reads each model's KV cache type and flags (or skips) results below LONGCTX_KV_MIN (default q8_0).
  • Streamed evals with loop abort: a reasoning model stuck in a repetition loop is stopped within seconds (Fail: Degradation) instead of hitting a 10-minute timeout. Thinking is capped per question (Fail: Think cap), and a stopped reply whose visible answer is already complete is still graded.
  • Margin of error shown with every score: the dashboard shows ± points next to the test profile. test and quick profiles carry a "rough guide only, re-test your finalists" banner.
  • Retries tracked and penalised: questions that needed the No-Thinking retry are counted, shown next to the scores, and lower the Agent score. More than 20% is flagged ⚠, because it usually points to a template or thinking setting rather than a weak model.
  • Concurrent evals: EVAL_CONCURRENCY keeps several questions in flight for servers running --parallel N.

🔍 Knows what your server actually loaded

  • Launch flags recorded per model: KV cache type (-ctk/-ctv), flash attention, -ub/-b, slots, mmproj, --jinja and CPU offload, read from the llama-server command line.
  • GGUF header reader: file size, bits per weight, total and active parameters, expert count, and attention type (GQA / MLA / sliding window / hybrid SSM) come from the model file itself, not the filename. A MoE without "A3B" in its name is now correctly reported as MoE.
  • KV cost and "Max ctx @85%": KV cache size per token and the largest context that fits in 85% of VRAM, for every model.

📏 Tested at your deployed context

  • Minimum context rule: models loaded below MIN_CTX are flagged and kept off the podium, or skipped entirely (MIN_CTX_ACTION=refuse).
  • Prefill at real prompt lengths: prefill speed at 8k / 32k / 64k tokens. Offloaded MoEs that look fast at 1.4k often slow down sharply here.
  • Agent time to first token: TTFT for a ~20k-token agent-style prompt, cold and on the next turn with the prefix cached.

🏆 Fairer rankings

  • Rebuilt Agent score (0–100, quality first): accuracy decides, and speed can only separate models of similar quality. Prefill speed and agent TTFT now count most within the speed part. VRAM is no longer part of the Agent score.
  • Per-machine budget profiles: BUDGET_PROFILE=server or main (or your own), with rules for minimum context, VRAM reserve (e.g. for Immich or the desktop), minimum KV type, retries and more. The dashboard shows a pass/fail column per rule, and failing models stay off the podium.
  • VRAM sweet spot: Agent score × VRAM fit finds the most capable model that still leaves sensible headroom. Under-using the GPU is now only a small penalty.

📊 Dashboard & outputs

  • New columns, charts and filters for everything above, plus click-to-highlight: click a model name anywhere and it is highlighted across every chart and table.
  • Structured JSON export: results.json (always the latest run, written atomically) and benchmark_<timestamp>.json, with null for missing values instead of NaN.
  • Markdown report with a new Deployment fit section.

🔒 Safety & quality

  • Hardened code sandbox: Docker runs as a non-root user with no network, a read-only filesystem and resource limits. A self-test runs at start-up and in --check. Programs that exit before their tests run no longer count as passes.
  • NVML telemetry in-process via nvidia-ml-py, with automatic fallback to nvidia-smi.
  • Refactored into the llmbench/ package with a 90-test self-test suite (python -m pytest selftests/, no GPU or server needed).
  • Credits & licences: a new README section credits every dataset's authors and licence, and documents the GPQA authors' request that questions are never published online.

⚠️ Behaviour changes

  • Scores are not comparable with v1.5. The Agent score, VRAM fit and eval average are all calculated differently, and enabling GSM8K, MATH or tool calling changes the eval average.
  • Bug fix: a model whose evals all failed used to receive a score on a different scale and could rank first. It now gets no Agent score and is labelled "no eval score".
  • The history CSV gains new columns. It is migrated automatically, and a .bak copy is kept.
  • Keep EVAL_CONCURRENCY=1 for runs you want to compare with older results.

📦 Updating to v2.0.0

# Pull the latest changes
git pull origin main

# Install new dependencies (adds nvidia-ml-py and aiohttp; both optional at runtime)
pip install -r requirements.txt

# Update your configuration (back up your old .env first)
cp .env .env.v1.5.bak
cp .env.example .env

# Confirm telemetry, launch flags and the GGUF reader on your machine
python benchmark.py --check

🛠️ Configuration

Review the new sections in .env.example:

  • PREFILL_CTX_TESTS, AGENT_PROMPT_TOKENS, MIN_CTX (deployed context)
  • EVAL_TOOLCALL, EVAL_LONGCTX, LONGCTX_* (new evals)
  • EVAL_STREAM, EVAL_THINK_BUDGET (streaming)
  • BUDGET_PROFILE (per-machine rules)
  • GGUF_PATH_MAP (if llama-swap runs in Docker and the benchmark isn't run as root)

Refer to the README.md for complete details.

🧪 Known limitations

  • Reading launch flags and GGUF files from a llama-server inside Docker needs root, or GGUF_PATH_MAP. Run --check to confirm.
  • test and quick profiles are for shortlisting only. Use standard or thorough before picking a model.