Repository navigation
Releases
v2.4.1: `--diff-runs`, what changed between runs
🧰 Local-LLM-Benchmark v2.4.1: --diff-runs, what changed between runs
A report that follows the same models across finished runs and says which changes are real and which are noise. Nothing is benchmarked, and no score changes.
✨ What changes
🔍 --diff-runs
python benchmark.py --diff-runscompares the two newest finished runs;--diff-runs 4the four newest;--diff-runs A B [C ...]these runs (a stamp, abenchmark_<stamp>.json/rescored_<stamp>.json, or a folder withresults.json). The first run is the baseline.- Per model: rank and Agent score in every run (with a trend line from three runs), the change in Agent quality, each dataset's accuracy, long context per length, generation, prefill, prefill at 64k and at depth, agent TTFT, seconds per question, load time, VRAM peak, GPU power and energy per token. Launch changes are listed (context, quant, KV type,
--n-cpu-moe, ngl, batch sizes, flash attention, llama.cpp build, sampling), and so are the run settings that differ. - Noise first. Every change from the baseline to the latest run is marked ↑ better, ↓ worse or ≈ within noise:
- accuracy is paired when both runs'
eval_details_<stamp>.jsonlare found (exact McNemar test on the questions that flipped; the result cache and seed give the same questions), otherwise an unpaired 95% Agresti-Caffo interval. A details file recorded under the server's path is also looked for next to the JSON, so a report built on the main PC over SMB still pairs; - tokens/s and seconds: at least 5%, or twice the larger run-to-run spread of the timed samples; power and energy 5%; VRAM 0.1 GB; Agent score and quality 0.5 points;
- rank agreement between consecutive runs as Kendall's τ-b.
- accuracy is paired when both runs'
- Honest about comparability. Runs that asked or graded the questions differently (the checks
--comparerefuses on) get a "Not like-for-like" banner instead of a refusal. Runs scored by different versions are noted, and--diff-rescorescores every run with this version first (in memory; no file is written but the report). The report says that the Agent score and quality are relative to each run's best finalist. - Output:
results/run_diff_<stamp>.html(self-contained, no CDN, a Content-Security-Policy that blocks any network access; sortable tables, a model filter, an "only changes beyond noise" switch; light and dark),results/run_diff_<stamp>.jsonwith every number, and a terminal summary with the biggest movers. - The report reads verdicts and aggregates only. No question, reply or reference answer is read into it, so it is safe with GPQA enabled; it still lives in the git-ignored
results/folder. --forget-runleaves diff reports alone (they belong to no single run).
🧪 Tests
selftests/run_factory.pywrites synthetic finished runs with the real JSON writer, plus matching per-question details, with planted changes.selftests/test_v241_rundiff.py(21 tests) covers:- the three statistical tests against known values;
- that the planted changes are found and the 2-question wobble is not;
- the unpaired fallback;
- a run diffed against itself reports nothing;
- three runs;
- settings and version mismatches,
--diff-rescore, and that no run file is touched; - run selection;
- HTML escaping of model names, no question text in the report, the page script parsing under Node, and the table shape;
- the command line;
- an end-to-end diff of two evals-only runs against the mock llama-swap server.
- 360 selftests pass, 1 skipped.
How old and new results compare
No scoring, grading, prompt or cache-key change: every run scores exactly as with v2.4.0. GRADER_VERSION is unchanged and the result cache stays valid.
Credits
New methods credited in the README: McNemar (1947), Agresti and Caffo (2000), Kendall (1938, 1945). No dataset, library or third-party code was added.
v2.4.0: a fair Agent score across runs, `--rescore`, steadier long context
🧰 Local-LLM-Benchmark v2.4.0: a fair Agent score across runs, --rescore, steadier long context
✨ What changes
⚖️ Long context: fixed lengths per model (AGENT_LONGCTX_BASIS)
- The problem. The long-context part of the Agent score used only the lengths every ranked model ran.
- Now: each model is judged on the lengths in
AGENT_LONGCTX_BASIS(default64k,90k) that its loaded context fits. A 64k model is judged on 64k; a 90k or 128k model on 64k and 90k. The context factor (AGENT_CTX_CURVE) still carries the preference for longer context, and the usable-context rule is unchanged. - The tie-zone bar is still the best finalist, but only models judged on every basis length can set it. Otherwise a 64k model with a perfect, easier 64k test could raise the bar for everyone.
- 128k is not in the default basis. A 128k model that does badly at 128k already loses through its usable context (it counts as a 90k model). Set
AGENT_LONGCTX_BASIS=allto count every tested length, orcommonfor the v2.3 rule.
🎯 Long context: more questions near the pass mark (LONGCTX_BORDERLINE_TRIALS)
- With 8 codes per length, 75% is the step between 6/8 and 5/8: one code decides whether a model's usable context is 64k or 90k.
- Now a result within one code of
AGENT_CTX_PASS_PCT(8 codes at 75%: 5/8 or 6/8) is askedLONGCTX_BORDERLINE_TRIALSmore questions (default 2, i.e. 8 more codes) before it stands. The extra questions reuse the cached document, so they cost generation time only. - The codes for the extra questions are always in the document, so it is the same document whether or not they are asked. The log notes when it happens (
64k: 5/8 is within one code of 75%, asked 8 more). The per-length count is stored aslongctx_<length>_nand used for the error.
🔁 --rescore
python benchmark.py --rescore [JSON]scores a finished run again with this version's rules. It keeps every measurement and answer and prints old vs new rank, Agent score and quality per model.- It writes
rescored_<run>.jsonand its dashboard. The run's own files,results.jsonand the latest dashboard are left alone. - A scoring setting the run did not record (
AGENT_LONGCTX_BASISfor runs before v2.4) comes from today's.envand is named in the output.
🧹 Housekeeping
--forget-run STAMPremoves one run's files fromOUTPUT_DIRand its rows in the history CSV (.bakkept). Ifresults.jsonwas that run, the newest other run takes its place. Without--forceit only lists what would go. The result cache is kept.--comparenow also refuses runs graded differently: strict tool calls, code-test timeout, LiveCodeBench tests per problem and date, extra request fields, long-context thinking. Runs too old to record a setting are not refused for it. Different long-context trials, keys or borderline trials are noted.--estimate:- The funnel's pass share was a fixed 50%. With no earlier funnel run it assumes 85%.
- A
--resumerun no longer counts the speed test (and the long-context test) of models the rows file already holds.
- "GLM shown as Dense" is struck from the plan: both November runs label it MoE (read from the GGUF header).
🔁 How old and new results compare
-
Stored results do not change.
--reloadkeeps stored scores;--rescoreshows the new ones.Model Rank old → new Agent score old → new Why Groot 1 → 1 93.80 → 94.47 its 90k counts Qwen3.6-MTP 10 → 9 86.46 → 87.08 its 90k counts (100%) Qwen3.6 Q4_K_M 9 → 10 86.83 → 86.83 unchanged Ornith-9B / Qwen3.8-27B 12 / 11 → 11 / 12 83.22 → 83.84 / 83.73 flint 13 → 14 82.55 → 76.63 its 90k is 87.5% Wicked-Stuff 15 → 15 77.32 → 71.47 its 90k is 75% Empero-Qwen3.8-9B 18 → 16 64.45 → 71.16 its 90k is 100% The other 10 same rank +0 to +0.6 Every change comes from the model's own 90k answers, now counted. With this, the 16 Nov and 19 Nov rankings agree on the order of the models in both runs.
-
New runs:
- Long-context results near the pass mark get more codes, so a borderline model's usable context is decided on 16 codes instead of 8.
- The long-context document holds the extra codes, so v2.4 long-context results are not like-for-like with v2.3.x runs. They are never cached anyway.
⚠️ Worth knowing
-
New settings:
AGENT_LONGCTX_BASIS(default64k,90k;common= v2.3;all)LONGCTX_BORDERLINE_TRIALS(default 2; 0 = off)
Both are in
.env.examplePART 3. -
--compareof a v2.4 run with an older run uses the v2.4 rule for both: each model's own lengths.
Credits
- No dataset, benchmark method, library or other third-party work is added or changed. The README credits section is unchanged.
v2.3.7: fairer long prompts, a steadier long-context test, cheaper cache changes
🧰 Local-LLM-Benchmark v2.3.7: fairer long prompts, a steadier long-context test, cheaper cache changes
✨ What changes
📏 Long prompts are sized without the chat template's own tokens
- The benchmark builds its long prompts (prefill at 64k/90k/128k, prefill at depth, the 20k agent prompt, the long-context document) from a measured tokens per word. v2.3.6 measured it as prompt tokens ÷ words of one 1,200-word prompt, so the template's fixed tokens were counted as words.
- Most templates add 10-20 tokens and the error was about 1%. Cyber-Tiel-Coder's template adds ~180 tokens to every request (a default system prompt), so it read 1.33 tokens per word instead of ~1.18 and every long prompt came out about 12% short: its "128k" long-context test was 111k tokens (others ~125k), its 90k test 80k, its agent prompt 18.1k instead of 20k.
- Now the figure is the slope between the short tech prompt and the prefill prompt, so the template's tokens cancel out. The log says so when a template adds 64 tokens or more (
[info] the chat template adds ~180 tokens to every prompt), and the JSON stores it asprompt_overhead_tokens.
🧷 The long-context document stays the same between runs
- The seeded document's length, and so where its codes sit, followed each run's measured tokens per word (1.172 one run, 1.177 the next), so most models got a slightly different document every run. The figure is now rounded up to a 0.02 step before the document is built: the same document while the measurement wobbles, and never longer than before (at most ~1.7% shorter).
- It does not remove llama.cpp's own run-to-run differences: on 2026-11-19 Bolt-E4B dropped one of 8 codes at 64k with an identical document. For a steadier long-context score now, set
LONGCTX_TRIALS=4(16 codes per length instead of 8). The extra questions reuse the cached document, so they cost generation time only.
♻️ Cached answers survive a change of thinking budget
- The thinking budget and
max_tokensare part of each question's cache key, so changingEVAL_THINK_BUDGET(or the context, which moves theautobudget) meant asking everything again. - Now an answer that came out of one attempt that ended on its own (no thinking cap, timeout, loop or length cut) is reused under another budget when it used no more thinking tokens and no more tokens than the new limits. Neither limit was reached, so it could not have changed. Capped, retried or cut answers are asked again.
- The result line shows it:
[96 from cache, 12 asked under another thinking budget]. - Only answers stored by v2.3.7 or later carry the extra key (
akeyin the cache file); older answers keep working as before. - Why it matters: on the 2026-11-16 fresh run, 111 of 7,386 questions (1.5%) hit the 32k thinking cap or the 600 s timeout. They took 19% of the question time and 25 of them ended correct. Only 1.2% of correct answers thought for more than 16k tokens. A lower budget is now a cheap experiment instead of a full re-run.
⚠️ A llama.cpp update is announced before it empties the cache
- The llama.cpp build stays part of every model's fingerprint (a new build can change answers), so an update means every question is asked again.
--estimatereads the current build from a model llama-swap already has loaded (nothing is loaded for it), compares it with each model's last results and leaves those models' cached answers out of the estimate:llama.cpp build changed (b12316 -> b12407): the result cache will NOT be reused for 22 of 22 model(s). With no model loaded it cannot tell and says nothing.- The run checks again after the speed test, when every model's build is known, and prints a
[warn]line before the evals start.
🔧 Small things
- Tool calling: a model that calls a tool although a required value is missing (the
missingcases) is now toldcalled a tool although a required value was not given (it should ask the user); it used to get the irrelevance wording "needs none". Grading is unchanged. Answers already in the cache keep their old text until asked again. - The "in flight: 3 (capped: KV pool ... -kvu unknown)" line was printed before every dataset of every stage (16 times per model). It is now printed once per model, as
In flight: 3 per dataset (...), and again only if the number changes.
🔁 How old and new results compare
- Scoring code is unchanged. Rescoring the 2026-11-19 run gives the stored scores exactly. Stored results,
--reloadand--compareare unaffected. - New runs measure slightly different prompts:
- Models with a plain template: long prompts about 1% longer, closer to the target lengths (most landed 1-3% short).
- The long-context document changes once for every model (new rounding), so long-context results of the first v2.3.7 run are not like-for-like with v2.3.6 runs. They are never cached anyway.
- Result cache: nothing is invalidated. Question keys are the same as v2.3.6; new answers just carry one extra key.
Credits
- No dataset, benchmark method, library or other third-party work is added or changed. The README credits section is unchanged.
v2.3.6: tool calling grows with the profile, finalists-only accuracy charts
🧰 Local-LLM-Benchmark v2.3.6: tool calling grows with the profile, finalists-only accuracy charts
Two changes from the first full v2.3.5 baseline on the server (2026-11-15, 22 models, funnel to moderate, resumed twice after UPS shutdowns), plus the runner fix from the v2.3.5 review.
✨ Key New Features
🛠️ Tool calling: 96 cases, more as the profile grows
- The built-in tool-calling set grows from 32 to 96 cases. A stage now asks max(32, its rows) of them: 32 at
quickandbasic, 43 atmoderate, all 96 fromstandardup. The cases are asked in a fixed order, so every stage is a prefix and the first 32 are always the v2.3 core set. MIN_TOOLCALL_PCTis still decided on those 32 at the first funnel stage (and checked again on the final count when the run ends).- Two new categories, after the ideas of BFCL V3 (in a much simpler single-step form):
- missing: a tool fits but the user left out a required value (no e-mail address, no room, no repo). The right answer is to ask, not to call with a made-up value.
- chained: the conversation already holds the model's earlier call and the tool's result; the next call must use that result (convert the share price it just got, read the newest of the listed files, set the thermostat only because it is 11 °C), or no call when the result already answers the question.
- Harder cases in the old categories: close tools side by side (forecast vs current weather, exchange rate vs conversion, turn off vs turn on, search files vs list a directory), three calls in one reply, mixed tools in parallel, dates relative to a given "today", numbers in words ("two and a half thousand", "a quarter of full volume").
- New tool definitions (forecast, lights off, locks, exchange rate, file search/write, shell command, Docker restart/logs, restaurant booking, GitHub issue, volume, calendar lookup, order status, meetings) are only offered in the new cases.
multiple-10is replaced bymultiple-10b. The old case ("How do you say 'thank you' in German?" with a translate tool offered) was answered directly by 10 of 22 models. Answering a well-known fact yourself is exactly what the irrelevance cases reward, so the case measured style, not tool use. The new one asks for the translation tool explicitly.- The funnel gate counts tool calling's error while the final stage will still ask a model more tool cases (it used to be complete at the first stage). Bounds get slightly wider: the safe side.
📊 Accuracy charts show the finalists only
- With the funnel, a model that stopped early answered only that stage's questions (11 per dataset at
quick) while the finalists answered 43, so flint's and LFM2.5 Q4_K_XL's 11/11 sat above finalists' 41/43. Such models are now left out of the dataset accuracy / pass-rate / tool-calling charts, the Quality vs speed chart and the quant table's eval average. - A line above the charts says how many stopped and where; Show them brings them back faded, with
[quick]after the name and the reason in the tooltip. - Long-context retrieval is asked in full at the first stage, so its chart keeps every model. The table and the Agent leaderboard are unchanged. Display only: no score changes.
🔁 How old and new results compare
- Existing results do not change. The 22 models of the 2026-11-15 run rescored under v2.3.6 give the same Agent scores, quality and rank states as stored (
--reloadkeeps stored scores anyway). - The gate: replayed on the 2026-11-15 run with its own answers: the same 18 of 22 models pass
quickand 18 of 18 passbasic. The one screened-out model (Empero-Qwen3.8-4B) moves from 0.94 to 0.18 points below the bar. - New runs: tool-calling percentages are not like-for-like with earlier runs. Finalists of a
moderatefunnel answer 43 tool cases instead of 32, 11 of them harder.--comparenotes the different question counts. How much the new cases move the ranking can only be seen after a run. multiple-10b, simulated on the 2026-11-15 run (counting it as passed for the 10 models that answeredmultiple-10directly): every other model keeps its relative order, but Empero-Qwen3.8-9B drops from #2 (89.4) to #11 (85.2) and Nemotron-3.5-Lightning from 85.6 to 81.6. Neither model's own answers change. The best composite rises, and both fall just outside the tie zone. That is the honest result of removing a case that penalised direct answers, and it also shows how compressed the field is: one tool case moves a model nine places.- Result cache: 31 of the 32 core cases keep their cache keys. A first v2.3.6 run asks
multiple-10bof every model and the 11 new cases of everymoderatefinalist. Tool cases are short (about 130 tokens), so this is roughly a minute per model.
⚠️ Worth knowing
- The missing and chained cases follow the system prompt the tool always sends ("call it with exactly the arguments the user gave"). A model that fills a required value with a guess fails them. That is deliberate for agent use.
- Chained cases send
assistantmessages withtool_callsandtoolmessages. llama.cpp with--jinjashould render these for any template with tool support (not checked on every model here). If a model scores 0 on every chained case, look at those requests ineval_details_*.jsonlbefore blaming the model. - The defaults are unchanged:
EVAL_FUNNEL_FINALdefaults tostandard, so a default funnel run asks all 96 tool cases of its finalists.
Fixes
evals/runner.pyheld its second half twice (the pause work merged onto a pre-v2.3.3 runner). Runs used the older gate, so hard-rule models were logged asdropand stored as screened out. Fixed. Gate decisions and scores were not affected.selftests/test_v236_runner.pyguards it (from the v2.3.5 review).
Credits
- The tool-calling categories and AST-style matching follow the Berkeley Function Calling Leaderboard (Patil, Mao, Yan, Ji, Suresh, Stoica, Gonzalez, ICML 2025, PMLR 267:48371-48392; Apache 2.0). The new categories follow BFCL V3's Missing Parameters and multi-turn ideas. No BFCL data is used: all 96 cases were written for this tool.
v2.3.5: settings named for their role, a 90k long-context length, safer defaults
🪟 llama-swap-llm-benchmark v2.3.5: settings named for their role, a 90k long-context length, safer defaults
A tidy-up release before a fresh baseline run: the endpoint settings no longer carry product names, long-context gets the 90k step the 27B models are loaded at, and the dashboard shows VRAM pressure the way you read it. Nothing about how a model is measured or graded changes, so existing Agent scores are identical (rescored: 26 server + 7 main-PC models from the 2026-11-12 runs).
✨ Key New Features
🏷️ Settings renamed (old names keep working)
LLAMA_SWAP_URL/LLAMA_SWAP_API_KEYare nowSERVER_URL/SERVER_API_KEY(any OpenAI-compatible server works, llama-swap is one of them).STRATA_URL(and the olderSTRATA_SWAP_URL),STRATA_API_KEY,STRATA_MODEL,STRATA_PARAMS_B,STRATA_ACTIVE_B,STRATA_QUANT,STRATA_CTXare nowSECONDARY_*.- An old name in
.env, a favourite or the shell still works and prints one line saying what it is called now;--check-envlists it; the settings page renames it when you save. If both names are set, the new one wins. - Not renamed on purpose: names inside result files (
strata_api_key,strata_details, columns) so older results reload and compare;--probe-strata; the words "llama-swap" and "Strata" in text.
📏 Long-context at 90k
LONGCTX_LENGTHSdefaults to 65536,92160,131072 (newlongctx_90k_pctcolumn; the history CSV is rewritten under the new header, a.bakis kept). A length larger than the model's loaded context is skipped as before, so a model loaded at-c 90112(88k) is not asked the 92160 length.- The tooltip on every context-size setting now reads: "Suggested context sizes: 4096 (4k) ... 1048576 (1M) | Enter the whole number (bracket value are just for help - do not use them in the fields)".
🛡️ Defaults
MAX_GPU_TEMP=85(was off). The run pauses at 85 C and carries on 10 C lower. With the server on another machine, or no readable GPU sensor, the existing "not enforced" warning is printed.MAX_GPU_TEMP=0turns it off.EVAL_SANITY_PCTstays 0 (off). The settings page now has a guide under it: what it does, and how the rate halves per model while answers match (20 -> 10 -> 5 -> 2 -> 0) and resets on a difference.
🎨 Dashboard
- VRAM bar above
VRAM_SWEET_HIGH: red, darkening evenly to black at 100% (no pink step). The hazard sign still starts atVRAM_SWEET_MAX. - Ctx and Max ctx tooltips give the exact size:
90k (92160 tokens).
🔁 How old and new results compare
- Agent scores, usable context and rank states of existing result sets are identical under v2.3.3 and v2.3.5 (checked on the 33 models of the 2026-11-12 runs; they hold no 90k results).
- New runs with a 90k result: a model loaded at 92160 or more that passes 90k but fails 128k now has usable context 92160 (context factor about 0.978) instead of 64k (0.95), a small rise it has earned. A model at 92160 or more with no 90k result counts as "not passed" and drops to 64k, so
--compareof an old run against a new one can penalise it; re-run the old one. The composite stays on 64k while any ranked model is loaded at 64k or 90112. - The result cache is not keyed on the tool version, so v2.3.5 reuses a matching v2.3.x cache. For a clean baseline move
results/cache/aside (or setEVAL_RESULT_CACHE=refreshfor the first run).
⚠️ Worth knowing
- Load the "90k" models at
-c 92160, not 90112, if you want them in the 90k test and the same dashboard label. - The long-context borderline cliff (8 retrievals per length) and the v2.4 MMLU-Pro tie-breaker are unchanged from v2.3.3.
- No new dataset, library or third-party work: the README credits section is unchanged.
Fixes / Security
- None in this release (no security-relevant change).
v2.3.3: the funnel says what happened to every model
🪟 llama-swap-llm-benchmark v2.3.3: the funnel says what happened to every model
Real runs (2026-11-12, two PCs, 33 models) showed the funnel pruned 1 model of 26, and that one (flint: 100% on the evals, 0% at 128k) was labelled "screened out" although it could never have had an Agent score. v2.3.3 separates rules from comparisons.
✨ Key New Features
🚦 Three explicit states: Ranked, Screened out, No Agent score
- Every evaluated model ends in exactly one state: Ranked (has an Agent score), No Agent score (fails a hard rule) or Screened out (the funnel gate dropped it as clearly below the leaders).
rank_state/rank_reasonare inresults.json(scoresgroup), shown as a badge on the model name, a Rank filter (with counts), the leaderboard note, the legend and the markdown report. If an older result is both, No Agent score wins. - Hard rules are decided outside the funnel: loaded or usable context below
AGENT_CTX_MIN, long-context belowAGENT_CTX_PASS_PCTat every length that counts, tool calling belowMIN_TOOLCALL_PCT. They are plain thresholds on the model's own numbers: no estimate, no margin of error, never compared with other models, and a failing model no longer lifts the leaders' bar. EVAL_HARD_FAIL=stop|continue(new, defaultstop): a model that fails a hard rule at a gate is not asked further stages.continuekeeps asking, so its later-stage scores exist.
🔬 A funnel that prunes
- The gate's bounds use the reducible error (the evals part; tool calling and long-context are asked in full at the first stage and do not shrink) plus
EVAL_FUNNEL_MARGINpoints (new, default 3). Rule:score + z × SE_evals + margin ≥ score_k − z × SE_evals_kfor theEVAL_LEADERS-th model. This replaces theEVAL_FUNNEL_CONF=0.85stopgap. - Decimal gaps: the log shows, per model and gate,
KEEP / SCREENED OUT / NO AGENT SCORE, score ± the error the gate used (and the error of all parts) and the signed gap to the leaders' bar to two decimals.evaluation.funnel.gatesin the JSON carriesstate,se_red,ub,lb,gap; the run header recordsmarginandhard_fail.
🧠 Thinking budget: auto is the baseline
EVAL_THINK_BUDGET=auto was already the code default and is the documented norm. If your .env still has EVAL_THINK_BUDGET=16384, delete the line: the first run then re-asks the questions whose budget changed (it is part of the result-cache key), and --compare refuses to mix runs with different budgets.
🔁 How old and new results compare
- Agent scores of an existing result set do not change (
python benchmark.py --reloadon the 2026-11-12 runs gives the samescore_agentfor every ranked model). Only labels change: on the server run flint and LFM2.5-24B-UD-Q3_K_XL now show No Agent score (long-context 50% / 62% at 64k, below the 75% pass mark) instead of flint's "screened out"; 24 models stay Ranked. - New runs can screen out more models (that is the point), but only models clearly below the leaders, and a model's stage scores are unchanged. No weaker model can rank higher because of this change: a model that is dropped is never ranked, and hard-rule models never enter the bar.
- One run set (v2.3.1) is the only evidence for the margin: nothing within 3 composite points of the final 5th-best was pruned at any gate in a retrospective, and 9 models would have been. Treat it as indicative.
⚠️ Worth knowing
AGENT_CTX_MINdefaults to 64k, so a model that passes 64k but fails 128k keeps an Agent score (lower context factor).AGENT_CTX_MIN=131072makes 128k mandatory but would remove 13 of 26 models of the server run (loaded at 64k-90k). The default is not changed.- The long-context borderline problem (8 retrievals per length, so 75% is a cliff between 6/8 and 5/8) is not addressed here.
- The deeper MMLU-Pro tie-breaker stage is planned for v2.4.
- New settings:
EVAL_FUNNEL_MARGIN,EVAL_HARD_FAIL(in.env.example).
🙏 Credits
No dataset, benchmark method or library was added or changed.
v2.3.2 - Pause Function and adjusted thinking cap
Pause/resume, thermal guard, auto thinking budget (work on v2.3.2, 2026-11-12)
Pause / resume
- Same-process pause controlled by
run_control.jsonin OUTPUT_DIR (polled ~1 s). Works on Windows and for a run on another machine sharing OUTPUT_DIR. CLI:--pause,--resume(no arg + live run); settings page: Pause / Resume buttons;/api/pause,/api/resume. - Temporary state:
run_<stamp>.pause.json+ the existingrun_<stamp>.rows.json. Both deleted when the run ends or resumes.--resume <run>still recovers a process lost while paused. - Manual pause: current model's speed phase finishes; evals stop taking new questions, in-flight ones finish and are graded. The batch probe cannot be paused.
- Thermal:
MAX_GPU_TEMP(0 = off, else 40-110),GPU_TEMP_RESUME_DELTA(default 10),GPU_TEMP_MAX_RESTARTS(default 3). Own GPU monitor thread at 500 ms (never inside token generation); trip is latched; resume after 3 consecutive cool samples. Not enforced, with a loud warning, when the server is remote or no temperature can be read. - Fresh restart:
PauseRestart(BaseException)raised at checkpoints only insideCONTROL.restartable(). Speed test restarts wholesale; long-context restores its row snapshot; an interrupted evalDatasetRunis discarded and re-run (answered questions come from the result cache). After GPU_TEMP_MAX_RESTARTS restarts of one line,PauseGaveUpends the run like Ctrl+C (exit 130, resumable). - Time paused is recorded in
phases.paused_s(it is also inside speed_s / eval_s).
Thinking budget
EVAL_THINK_BUDGET=autois now the default:EVAL_THINK_BUDGET_PCT(25) % of the model's own context, raised toEVAL_THINK_BUDGET_MIN(4096) and held toEVAL_THINK_BUDGET_MAX_PCT(60) % of the context (ceiling wins on a tiny context), rounded down to a multiple of 1024. Unknown context: 16384. A number or 0 still fixes it. Validation: PCT 5-90, MIN >= 256, MAX_PCT 10-90, PCT <= MAX_PCT.- Logged per model:
thinking budget 32,768 tokens (25% of 131,072 ctx)(or "floor ..." / "ceiling ..."). Exported in results.json / dashboard meta (think_budget,think_budget_pct,_min,_max_pct);--comparerefuses runs that differ. - Settings page: hover tip on the four keys lists the resulting budget for 4k..1M contexts from the current field values (JS twin of
auto_think_budget, tested under Node). - Consequences: the budget is in the result-cache key, so models whose budget changed are re-asked (
EVAL_THINK_BUDGET=16384keeps old cache and comparability). A bigger budget fits fewer requests in the shared KV pool: in-flight drops (4 -> 3 at 128k in the mock). Scores of previously capped models may rise; not measured. - Evidence (user's 2026-11-12 server log): many models hit the 16,384 cap on ~5-15% of questions; most recovered by the No-Thinking retry (time cost), the LFM2.5 builds show "0 recovered, N counted as wrong".
Rebrand (display strings only)
"Local LLM Benchmark Suite" in the dashboard, settings page, markdown report, argparse description, .env.example header, __init__. Left for the owner to decide: GitHub repo / Project name, REPO_URL and clone URLs, tool.name in the JSON and the notify body, the markdown front-matter tag, historical release notes, regenerated demo dashboards.
Open items
- Not verified: real GPU / NVML thermal behaviour, a real llama-server, Docker, Hugging Face.
- Suggested commits:
feat: pause/resume and GPU thermal guard,feat: auto thinking budget with floor and ceiling,fix: rebrand display strings, anddocs:(README) separately. - No new third-party dataset, method or library: no credits/licence change.
v2.3.1: Agent score, a settings page that saves settings and runs the test, numerous improvements (v2.1 → v2.3.1)
First release since v2.1.0. It rolls up v2.2.0, v2.2.1, v2.3.0 and v2.3.1.
🚀 llama-swap-llm-benchmark v2.2.0 — reuse what you have, check it, and compare across servers
v2.1 introduced the result cache, so a rerun asks only what is missing. v2.2 adds a full retest, a sanity check of cached answers, an --estimate that knows what the cache already holds, and --compare, which puts runs from different servers in one dashboard (for example a model that only fits on Strata next to your llama-swap models). Scores are computed exactly as in v2.1.
✨ Key New Features
🔁 Full retest (EVAL_RESULT_CACHE=refresh)
EVAL_RESULT_CACHEnow has three modes:true(default): reuse cached answers; only missing or failed questions are asked.refresh: full retest. Every question is asked again, and the new answers replace the cached ones, so the next run can reuse them.false: no cache at all; nothing is reused or stored.
- Speed tests and the long-context test are always measured again, in every mode.
🩺 Sanity check of cached answers (EVAL_SANITY_PCT)
EVAL_SANITY_PCT(0–100, default 0 = off) re-asks that share of the cached answers at random and compares each with the cached one:- Code and tool calls must have the same pass/fail; multiple choice and math the same extracted answer and verdict.
- The new answer is the one scored and stored.
- Differences are listed in the log (
[sanity] <question>: cached ..., now ...), summed per model at the end of the run, and shown in a new Sanity diff dashboard column.
EVAL_SANITY_ADAPTIVE=true(default): each model starts atEVAL_SANITY_PCT. Every matching answer halves its rate in whole percents, and below 2% the checks stop for that model for the rest of the run (from 20%: 20 → 10 → 5 → 2 → 0). A different answer puts it back to the full rate. Models that keep answering the same are checked less and less; models that change keep being checked. Each run starts again at the full rate.- The questions picked differ per run (the seed is printed), so repeated runs check different questions.
🎛️ Each model's own sampling (EVAL_TEMPERATURE=server, EVAL_SERVER_SAMPLING_MODELS)
- Until now every eval request carried
EVAL_TEMPERATURE(default 0), which overrides the--tempin your llama-swap commands. Some reasoning models, Gemma-family especially, loop under greedy decoding: in a real run Gemma 4 26B and a Gemma 4 fine-tune needed a retry on 32% and 48% of questions, failed the retry budget rule and were dropped from the ranking, although every answer they finished cleanly was right. EVAL_TEMPERATURE=serversends no temperature, so every model uses the sampling in its own llama-swap command, withEVAL_SEEDas the sampling seed. This measures each model as you deploy it.EVAL_SERVER_SAMPLING_MODELS=<regex>does this only for the matching models; the rest keepEVAL_TEMPERATURE, and only the matching models are asked again.- A new Sampling column shows what each model was evaluated with, e.g.
temp 0orserver: temp 1.0, top-k 64, top-p 0.95.
⏱️ --estimate knows what the cache holds
- It says whether the run will reuse cached answers or is a full re-run (
refresh/false). - Per model, it shows how many answers it expects to reuse against the questions it expects to ask, and counts time only for the rest, plus any sanity-check re-asks.
- The one-line estimate at the start of every run says the same.
⚖️ Compare runs in one dashboard (--compare)
python benchmark.py --comparecombines the newest llama-swap run with the newest Strata run.python benchmark.py --compare A.json B.json ...combines the runs you name; the first is the base.- Every model keeps its own measurements. The scores that compare models with each other (Agent score, sweet spot, budget eligibility) are computed again over all models together, so the ranking matches a single run with all of them.
- Runs whose eval settings differ (seed, temperature, thinking budget, streaming switches) are refused, because they asked different questions or measured differently.
--forcecombines them anyway, and the dashboard notes it. - A model missing a dataset or questions shows "not ranked: incomplete evals", as within one run. A model in both runs is shown twice, labelled with its run.
- The dashboard gets a Run column and a Run filter to show or hide each run's models.
- Output:
results/benchmark_compare_<stamp>.html, copied tobenchmark_compare.html. - Tip: run both sides with the same eval settings. With
EVAL_PROFILE=funnel, a run with a single model goes straight to the final profile, so a lone Strata model answers the same questions as a llama-swap run's finalists.
🧹 Small changes
- Two
.env.examplesection headings carried stale "(v2.2)" tags for settings that shipped in v2.0; removed. results.jsonrecords the cache mode and sanity settings (evaluation.result_cache_mode,sanity_pct,sanity_adaptive) and, per model,sanity_checkedandsanity_different. Per-question logs mark re-asked questions withsanity(same/different) and the cached answer insanity_cached.
⚖️ How old and new results compare
- Scoring is unchanged. Runs with the same settings give the same scores as v2.1.
- Cached answers from v2.1 runs stay valid. The cache key is unchanged, so a v2.2 run reuses everything a v2.1.0 run answered.
- With
EVAL_SANITY_PCT> 0, a re-asked question is scored on its new answer. When decoding is not fully deterministic (batched requests, temperature > 0), a score can move by the questions that changed, the same noise a fresh run would show. - With own sampling (
EVAL_TEMPERATURE=serverorEVAL_SERVER_SAMPLING_MODELS), those models are measured with their deployed sampling instead of greedy decoding. Their answers are no longer bit-identical between runs, so small differences mean less. Models left atEVAL_TEMPERATUREare unchanged and keep their cached answers. --comparerecomputes the comparative scores over the combined set. A model's Agent score there can differ from its score in its own run, because the best model it is compared with can be different.
🔄 Updating
- Pull the new version. There are no new dependencies.
- Optional: copy the new settings from
.env.exampleinto your.env. The defaults behave like v2.1 (EVAL_RESULT_CACHE=true,EVAL_SANITY_PCT=0). - If you used
EVAL_RESULT_CACHE=falseto force a full retest, userefreshinstead, so the new answers are kept for later runs. - Keep the llama-swap / llama.cpp image unchanged between runs you want to reuse: the llama.cpp build is part of the cache key.
⚙️ Configuration
EVAL_RESULT_CACHE=true # true | refresh (full retest) | false
EVAL_TEMPERATURE=0 # a number, or server = each model's own llama-swap sampling
EVAL_SERVER_SAMPLING_MODELS= # regex: only these models use their own sampling
EVAL_SANITY_PCT=0 # % of cached answers re-asked and compared (0-100)
EVAL_SANITY_ADAPTIVE=true # halve a model's rate after each matching answerNew commands: --compare (with --force).
🚀 llama-swap-llm-benchmark v2.2.1: a settings file you can read at a glance, and a check for your .env
A tidy-up release. .env.example is reorganised so it is clear which settings you must check, which ones you change per run, and which are optional, with the built-in default shown for every one. A new --check-env compares your .env with it. Measurements, scores and the result cache are unchanged.
This prepares v2.3, which will add a settings page that reads .env.example in the same way.
✨ Key New Features
🗂️ .env.example in three parts
- PART 1 · Your setup: check every line once per machine. Covers the server address, the RAM process pattern, the budget profile, the VRAM sweet spot and Docker model-file paths.
- PART 2 · Your run: what you change between runs. Covers models, datasets, profile / funnel, concurrency, prefill lengths, cache, sampling, the thinking budget and output files.
- PART 3 · Optional: everything else, grouped by topic. Every line is commented out and already at its built-in default.
- One line per setting, in one format:
KEY=value # what it does [default: x]: an active line;[default]is what you get if you delete it.# KEY=value # what it does: a commented line; the value shown is the built-in default.# KEY= # what it does: not set by default; the text says what happens then.
- All 141 settings the code reads are listed (plus the
BUDGET_<PROFILE>_<RULE>andEVAL_THINK_BUDGET_<DATASET>families). The old file was missingEVAL_<DATASET>_REVISIONfor five of the seven datasets.
🩺 python benchmark.py --check-env [file]
Reads files only; it changes nothing and contacts no server. It lists:
- Unknown settings (ignored by the benchmark), with the closest real name:
EVAL_TOOLCAL->EVAL_TOOLCALL?. Budget rules and per-dataset thinking budgets are checked by name too. - Values that won't work as intended:
tueor another non-boolean (read as false);- a word where a number is expected (the run would stop at start-up);
KEY= # comment, which python-dotenv reads as the value# comment.
- Old names (
STRATA_SWAP_URL), keys set twice (the last one wins), and keys your shell also sets (the shell wins). - PART 1 / PART 2 settings missing from your
.env(usually new in this version), with the default they fall back to. - Everything you changed from the built-in defaults.
It exits with 1 if it finds an unknown setting or an unusable value, so it can run before a scheduled run.
🔧 Fixes
- Documented defaults that didn't match the code (the code was right; the file was wrong):
- the c...
Release - v2.1.0
🚀 llama-swap-llm-benchmark v2.1.0 — faster full runs, rankings you can still trust
A full run of ~28 models is almost entirely the GPU generating eval answers. v2.1 cuts that time down without letting a weaker model rank higher. It does this by asking fewer questions of models that are clearly losing, never re-asking a question it already has an answer for, and running several questions at once where the server can take them. A timing report shows where every minute went, and --estimate predicts a run before it starts.
✨ Key New Features
⏱️ Timing report and --estimate
- Every model now records its seconds per phase: settle, load, speed tests, prefill-at-length, agent TTFT, cooldown, eval load, long context, and each dataset. Thinking and answer tokens are counted too.
- Where it shows up:
results.json:models[].timing, plusrun.timingwith the speed/eval phase wall times, the per-dataset totals and the token counts.- CSV columns
time_*_sandeval_tok_*. - A new dashboard section, Where the time went.
python benchmark.py --estimatepredicts a run's duration from earlier runs' JSON (measured tok/s, prefill speed, seconds and tokens per question) and any batch-probe result. Models with no history get a rough guess. A one-line estimate is also printed at the start of every run.
🔻 Funnel profile with confidence-interval gates (EVAL_PROFILE=funnel)
- Every model answers
quick. After each stage, a model goes on only if its upper bound reaches the lower bound of theEVAL_LEADERS-th model. At leastEVAL_LEADERSmodels always go on. - Survivors answer the next stage (
basic→moderate→ … →EVAL_FUNNEL_FINAL, defaultstandard). They are asked only the new questions. - A plain "top 5 after quick" was tested and rejected. In a simulation with a crowded top, it screened out the true best model in 18–33% of runs. The confidence gates lost it in ≤0.1% of runs, picked as well as a full run, and saved ~40–50% of the eval work.
- A screened-out model keeps its stage score but is not ranked and never reaches the podium. First-stage avg compares every model on the same first-stage questions.
💾 Result cache and resume
- Every answered question is stored in
results/cache/. A rerun, or a run restarted after a crash, asks only what is missing. - An answer is reused only when all of these are identical:
- the model file (size plus a hash of its first and last MiB)
- the launch flags and llama.cpp build
- the loaded context, slots and chat template
- the hardware
- the exact request and sampling settings
- the thinking budget and the retry, salvage and forcing switches
- the grader version
- Speed tests are always measured again.
⚡ Batched evaluation that cannot change a ranking
-
Questions in flight are now capped per model to the server's slots (
/propstotal_slots). With a shared KV pool (--kv-unified, the llama.cpp default when-npis auto) they are also capped to what fits the pool in the worst case. The cap is printed whenever it applies. -
-kvu/-no-kvuare read from the launch command, and a new Unified KV column shows the result. -
When any question ran concurrently, the Agent score uses a single-stream s/q estimate for every model: tokens ÷ speed-test tok/s, plus prompt tokens ÷ prefill tok/s, plus grading time. Measured latency under load includes queueing, so it is no longer used for ranking in that case.
-
New
python benchmark.py --batch-probeasks the same questions at 1/2/4/8 in flight. It reports:- the throughput gain
- per-question latency
- accuracy
- how many answers agree with the one-at-a-time answers
- errors and peak VRAM
It then recommends a level.
🔀 Pipelined grading
- Sandboxed code runs while the next question is generating. The sandbox itself is unchanged: one isolated container per program.
- This replaces the warm container pool from the plan, which would have let one program affect the next.
🙅 Skip evals of models that cannot be deployed (EVAL_SKIP_INELIGIBLE=auto)
- Models below
MIN_CTX, failing a budget rule, or aboveGOLDILOCKS_MAX_VRAM_PCTare no longer evaluated. They were off the podium anyway. They stay in the table with the reason.
📜 Long context: one shared document per length
- Each length now builds one document. The questions about it ask different vaults and reuse the document from llama-server's prompt cache.
- The first, cold question doubles as the prefill-at-length measurement. A
PREFILL_CTX_TESTSlength that is also aLONGCTX_LENGTHSlength is no longer sent twice.
🧠 Thinking budgets and budget forcing (opt-in)
EVAL_THINK_BUDGET_<DATASET>sets a per-dataset thinking budget.EVAL_BUDGET_FORCING=true: at the cap, the reasoning is kept and closed, and the model is asked for its answer instead of a No-Thinking retry. This uses llama-server's/apply-templateand/completion. The method comes from s1: Simple test-time scaling (Muennighoff et al., 2025).- Both change what is measured, so both are off by default and recorded in
results.json.
💬 Dashboard: tooltips, column chooser and dataset links
- Every column header has a tooltip: what it measures, where the number comes from and whether higher or lower is better. Eval columns and dataset names also say how big the test is and how it is scored. Budget-rule columns show the rule's limit.
- Hide the columns you don't need:
- Click the × in any header to hide that column.
- The new Columns panel toggles single columns or whole groups (Model, Launch, VRAM & memory, Power & cost, Speed, Evals, Long-ctx, Budget, Scores).
- Only this group shows just the model columns plus one group, for example an evals-only view in one click.
- Show all columns brings back the full table.
- The column choice is remembered in your browser, so later dashboards open with the same view. Hiding a column changes only the view, never the data or the scores.
- Test names in the page header link to their datasets. Each dataset name in the header and above the accuracy charts opens the Hugging Face page it was loaded from (the pinned revision when
EVAL_<DATASET>_REVISIONis set). Built-in tool calling links to its source file.
🔁 Rebuild a dashboard without a run (--reload)
python benchmark.py --reloadrebuilds the dashboard of the newest finished run (results/results.json) with the current dashboard code. No server or GPU is needed.python benchmark.py --reload results/benchmark_<stamp>.jsonreloads a specific run.- Nothing is measured or rescored: every number, score and ranking is the one the run saved, and the run's own settings are used rather than your current
.env. - It writes
results/benchmark_dashboard_<run>_reload.html(the run's own dashboard is kept) and updatesbenchmark_dashboard.html. - It is mainly a tool for working on the dashboard. A run writes its JSON when it finishes (with
JSON=true, the default), so an interrupted run can't be reloaded.
🧩 LiveCodeBench
EVAL_LCB_SINCE=2025-01-01keeps only newer problems, which follows LiveCodeBench's own contamination method.- A load-time guard refuses a copy whose questions are in Korean. The
bzantium/livecodebenchcard mentions a Korean translation; the data itself is English. EVAL_<DATASET>_REVISIONpins any Hugging Face dataset to a commit.
🐞 Fixes
- A weaker model could rank first when one of its datasets failed (high). The eval average was taken only over the datasets that produced a score. Example: every GPQA request hits a template error. That model was then averaged over easier datasets, and outranked a model that was stronger on everything it answered (Agent 92.8 vs 88.0). Now a model is ranked only with a score on every dataset of the run (≥90% of each dataset's questions answered without an API error). Otherwise it shows "not ranked: incomplete evals (GPQA 0/96 answered)".
- Profiles were not nested. MATH-500's
quickquestions were not a subset ofstandard's. Every profile now takes a prefix of one seeded question order per dataset. - The first question of each model carried the model load in its latency. A warm-up request now loads the model first. The load is timed separately (
eval_load). --kv-unifiedwas not reflected. The docs said--parallelalways divides the context.- Long context at the full loaded context failed with HTTP 400. Found in a real run: 128k at ctx 131,072 produced prompts of 131,644–132,331 tokens, so two models had no 128k result. The test now keeps 3% of the context back. If the server still reports "exceeds the available context size", it shrinks the document once and retries.
- Long context scored 0% for models that think even when told not to. Found in a real run: LFM replied with reasoning only and an empty answer at 64k and 128k, so its v2.0 long-context score (and the Agent score blended from it) was an artefact. Such a reply is now asked once more with 4,096 tokens; it is noted and marked
thinking_retry. - The batch probe understated the gain. It divided total tokens by wall time, so one long reply finishing alone at the end dominated. It now reports the steady gain (Little's law), writes every answer to
batch_probe_<stamp>_details.jsonl, reads ctx/slots/KV from the server, refuses more than 4 models and warns when another run is still using the server. Agreement is only a rule from 40 questions up. --estimateno longer counts eval time for models that would be skipped, and is calibrated on the tokens per question measured in a real run.- Small fixes: the Agent tie margin counted questions from datasets with no score, and long-context requests let
EVAL_EXTRA_BODYoverridestream.
🔒 Security
EVAL_CODE_SANDBOX=autono longer falls back to an unis...
Release - v2.0.0
🚀 Release - v2.0.0
We are excited to release v2.0.0 of the llama-swap-llm-benchmark suite! v1.5 taught the tool to grade model intelligence. v2.0 makes those grades trustworthy and decisive, and measures models the way you actually deploy them: at your real context size, with your real KV cache settings, and against your machine's own VRAM budget.
✨ Key New Features & Capabilities
🧠 Smarter, more reliable evaluations
- Math evals: GSM8K and MATH-500 join GPQA Diamond, MMLU-Pro, HumanEval, MBPP and LiveCodeBench. Answers are normalised (fractions,
\frac, units, thousands separators) and matched numerically. - Tool-calling eval (BFCL-style): 32 built-in cases (no download) send native OpenAI
toolsrequests in four categories: simple, multiple, parallel and "should not call". Arguments are graded field by field. Calls written as plain text instead of nativetool_callsare caught, which usually means a missing--jinjaor a chat template without tool support. Off by default (EVAL_TOOLCALL=true). - Long-context retrieval: a multi-key needle test at 64k and 128k tokens, with decoys. The benchmark reads each model's KV cache type and flags (or skips) results below
LONGCTX_KV_MIN(defaultq8_0). - Streamed evals with loop abort: a reasoning model stuck in a repetition loop is stopped within seconds (
Fail: Degradation) instead of hitting a 10-minute timeout. Thinking is capped per question (Fail: Think cap), and a stopped reply whose visible answer is already complete is still graded. - Margin of error shown with every score: the dashboard shows ± points next to the test profile.
testandquickprofiles carry a "rough guide only, re-test your finalists" banner. - Retries tracked and penalised: questions that needed the No-Thinking retry are counted, shown next to the scores, and lower the Agent score. More than 20% is flagged ⚠, because it usually points to a template or thinking setting rather than a weak model.
- Concurrent evals:
EVAL_CONCURRENCYkeeps several questions in flight for servers running--parallel N.
🔍 Knows what your server actually loaded
- Launch flags recorded per model: KV cache type (
-ctk/-ctv), flash attention,-ub/-b, slots, mmproj,--jinjaand CPU offload, read from the llama-server command line. - GGUF header reader: file size, bits per weight, total and active parameters, expert count, and attention type (GQA / MLA / sliding window / hybrid SSM) come from the model file itself, not the filename. A MoE without "A3B" in its name is now correctly reported as MoE.
- KV cost and "Max ctx @85%": KV cache size per token and the largest context that fits in 85% of VRAM, for every model.
📏 Tested at your deployed context
- Minimum context rule: models loaded below
MIN_CTXare flagged and kept off the podium, or skipped entirely (MIN_CTX_ACTION=refuse). - Prefill at real prompt lengths: prefill speed at 8k / 32k / 64k tokens. Offloaded MoEs that look fast at 1.4k often slow down sharply here.
- Agent time to first token: TTFT for a ~20k-token agent-style prompt, cold and on the next turn with the prefix cached.
🏆 Fairer rankings
- Rebuilt Agent score (0–100, quality first): accuracy decides, and speed can only separate models of similar quality. Prefill speed and agent TTFT now count most within the speed part. VRAM is no longer part of the Agent score.
- Per-machine budget profiles:
BUDGET_PROFILE=serverormain(or your own), with rules for minimum context, VRAM reserve (e.g. for Immich or the desktop), minimum KV type, retries and more. The dashboard shows a pass/fail column per rule, and failing models stay off the podium. - VRAM sweet spot:
Agent score × VRAM fitfinds the most capable model that still leaves sensible headroom. Under-using the GPU is now only a small penalty.
📊 Dashboard & outputs
- New columns, charts and filters for everything above, plus click-to-highlight: click a model name anywhere and it is highlighted across every chart and table.
- Structured JSON export:
results.json(always the latest run, written atomically) andbenchmark_<timestamp>.json, withnullfor missing values instead ofNaN. - Markdown report with a new Deployment fit section.
🔒 Safety & quality
- Hardened code sandbox: Docker runs as a non-root user with no network, a read-only filesystem and resource limits. A self-test runs at start-up and in
--check. Programs that exit before their tests run no longer count as passes. - NVML telemetry in-process via
nvidia-ml-py, with automatic fallback tonvidia-smi. - Refactored into the
llmbench/package with a 90-test self-test suite (python -m pytest selftests/, no GPU or server needed). - Credits & licences: a new README section credits every dataset's authors and licence, and documents the GPQA authors' request that questions are never published online.
⚠️ Behaviour changes
- Scores are not comparable with v1.5. The Agent score, VRAM fit and eval average are all calculated differently, and enabling GSM8K, MATH or tool calling changes the eval average.
- Bug fix: a model whose evals all failed used to receive a score on a different scale and could rank first. It now gets no Agent score and is labelled "no eval score".
- The history CSV gains new columns. It is migrated automatically, and a
.bakcopy is kept. - Keep
EVAL_CONCURRENCY=1for runs you want to compare with older results.
📦 Updating to v2.0.0
# Pull the latest changes
git pull origin main
# Install new dependencies (adds nvidia-ml-py and aiohttp; both optional at runtime)
pip install -r requirements.txt
# Update your configuration (back up your old .env first)
cp .env .env.v1.5.bak
cp .env.example .env
# Confirm telemetry, launch flags and the GGUF reader on your machine
python benchmark.py --check🛠️ Configuration
Review the new sections in .env.example:
PREFILL_CTX_TESTS,AGENT_PROMPT_TOKENS,MIN_CTX(deployed context)EVAL_TOOLCALL,EVAL_LONGCTX,LONGCTX_*(new evals)EVAL_STREAM,EVAL_THINK_BUDGET(streaming)BUDGET_PROFILE(per-machine rules)GGUF_PATH_MAP(if llama-swap runs in Docker and the benchmark isn't run as root)
Refer to the README.md for complete details.
🧪 Known limitations
- Reading launch flags and GGUF files from a llama-server inside Docker needs root, or
GGUF_PATH_MAP. Run--checkto confirm. testandquickprofiles are for shortlisting only. Usestandardorthoroughbefore picking a model.