Repository navigation
v2.3.1: Agent score, a settings page that saves settings and runs the test, numerous improvements (v2.1 → v2.3.1)
First release since v2.1.0. It rolls up v2.2.0, v2.2.1, v2.3.0 and v2.3.1.
🚀 llama-swap-llm-benchmark v2.2.0 — reuse what you have, check it, and compare across servers
v2.1 introduced the result cache, so a rerun asks only what is missing. v2.2 adds a full retest, a sanity check of cached answers, an --estimate that knows what the cache already holds, and --compare, which puts runs from different servers in one dashboard (for example a model that only fits on Strata next to your llama-swap models). Scores are computed exactly as in v2.1.
✨ Key New Features
🔁 Full retest (EVAL_RESULT_CACHE=refresh)
EVAL_RESULT_CACHEnow has three modes:true(default): reuse cached answers; only missing or failed questions are asked.refresh: full retest. Every question is asked again, and the new answers replace the cached ones, so the next run can reuse them.false: no cache at all; nothing is reused or stored.
- Speed tests and the long-context test are always measured again, in every mode.
🩺 Sanity check of cached answers (EVAL_SANITY_PCT)
EVAL_SANITY_PCT(0–100, default 0 = off) re-asks that share of the cached answers at random and compares each with the cached one:- Code and tool calls must have the same pass/fail; multiple choice and math the same extracted answer and verdict.
- The new answer is the one scored and stored.
- Differences are listed in the log (
[sanity] <question>: cached ..., now ...), summed per model at the end of the run, and shown in a new Sanity diff dashboard column.
EVAL_SANITY_ADAPTIVE=true(default): each model starts atEVAL_SANITY_PCT. Every matching answer halves its rate in whole percents, and below 2% the checks stop for that model for the rest of the run (from 20%: 20 → 10 → 5 → 2 → 0). A different answer puts it back to the full rate. Models that keep answering the same are checked less and less; models that change keep being checked. Each run starts again at the full rate.- The questions picked differ per run (the seed is printed), so repeated runs check different questions.
🎛️ Each model's own sampling (EVAL_TEMPERATURE=server, EVAL_SERVER_SAMPLING_MODELS)
- Until now every eval request carried
EVAL_TEMPERATURE(default 0), which overrides the--tempin your llama-swap commands. Some reasoning models, Gemma-family especially, loop under greedy decoding: in a real run Gemma 4 26B and a Gemma 4 fine-tune needed a retry on 32% and 48% of questions, failed the retry budget rule and were dropped from the ranking, although every answer they finished cleanly was right. EVAL_TEMPERATURE=serversends no temperature, so every model uses the sampling in its own llama-swap command, withEVAL_SEEDas the sampling seed. This measures each model as you deploy it.EVAL_SERVER_SAMPLING_MODELS=<regex>does this only for the matching models; the rest keepEVAL_TEMPERATURE, and only the matching models are asked again.- A new Sampling column shows what each model was evaluated with, e.g.
temp 0orserver: temp 1.0, top-k 64, top-p 0.95.
⏱️ --estimate knows what the cache holds
- It says whether the run will reuse cached answers or is a full re-run (
refresh/false). - Per model, it shows how many answers it expects to reuse against the questions it expects to ask, and counts time only for the rest, plus any sanity-check re-asks.
- The one-line estimate at the start of every run says the same.
⚖️ Compare runs in one dashboard (--compare)
python benchmark.py --comparecombines the newest llama-swap run with the newest Strata run.python benchmark.py --compare A.json B.json ...combines the runs you name; the first is the base.- Every model keeps its own measurements. The scores that compare models with each other (Agent score, sweet spot, budget eligibility) are computed again over all models together, so the ranking matches a single run with all of them.
- Runs whose eval settings differ (seed, temperature, thinking budget, streaming switches) are refused, because they asked different questions or measured differently.
--forcecombines them anyway, and the dashboard notes it. - A model missing a dataset or questions shows "not ranked: incomplete evals", as within one run. A model in both runs is shown twice, labelled with its run.
- The dashboard gets a Run column and a Run filter to show or hide each run's models.
- Output:
results/benchmark_compare_<stamp>.html, copied tobenchmark_compare.html. - Tip: run both sides with the same eval settings. With
EVAL_PROFILE=funnel, a run with a single model goes straight to the final profile, so a lone Strata model answers the same questions as a llama-swap run's finalists.
🧹 Small changes
- Two
.env.examplesection headings carried stale "(v2.2)" tags for settings that shipped in v2.0; removed. results.jsonrecords the cache mode and sanity settings (evaluation.result_cache_mode,sanity_pct,sanity_adaptive) and, per model,sanity_checkedandsanity_different. Per-question logs mark re-asked questions withsanity(same/different) and the cached answer insanity_cached.
⚖️ How old and new results compare
- Scoring is unchanged. Runs with the same settings give the same scores as v2.1.
- Cached answers from v2.1 runs stay valid. The cache key is unchanged, so a v2.2 run reuses everything a v2.1.0 run answered.
- With
EVAL_SANITY_PCT> 0, a re-asked question is scored on its new answer. When decoding is not fully deterministic (batched requests, temperature > 0), a score can move by the questions that changed, the same noise a fresh run would show. - With own sampling (
EVAL_TEMPERATURE=serverorEVAL_SERVER_SAMPLING_MODELS), those models are measured with their deployed sampling instead of greedy decoding. Their answers are no longer bit-identical between runs, so small differences mean less. Models left atEVAL_TEMPERATUREare unchanged and keep their cached answers. --comparerecomputes the comparative scores over the combined set. A model's Agent score there can differ from its score in its own run, because the best model it is compared with can be different.
🔄 Updating
- Pull the new version. There are no new dependencies.
- Optional: copy the new settings from
.env.exampleinto your.env. The defaults behave like v2.1 (EVAL_RESULT_CACHE=true,EVAL_SANITY_PCT=0). - If you used
EVAL_RESULT_CACHE=falseto force a full retest, userefreshinstead, so the new answers are kept for later runs. - Keep the llama-swap / llama.cpp image unchanged between runs you want to reuse: the llama.cpp build is part of the cache key.
⚙️ Configuration
EVAL_RESULT_CACHE=true # true | refresh (full retest) | false
EVAL_TEMPERATURE=0 # a number, or server = each model's own llama-swap sampling
EVAL_SERVER_SAMPLING_MODELS= # regex: only these models use their own sampling
EVAL_SANITY_PCT=0 # % of cached answers re-asked and compared (0-100)
EVAL_SANITY_ADAPTIVE=true # halve a model's rate after each matching answerNew commands: --compare (with --force).
🚀 llama-swap-llm-benchmark v2.2.1: a settings file you can read at a glance, and a check for your .env
A tidy-up release. .env.example is reorganised so it is clear which settings you must check, which ones you change per run, and which are optional, with the built-in default shown for every one. A new --check-env compares your .env with it. Measurements, scores and the result cache are unchanged.
This prepares v2.3, which will add a settings page that reads .env.example in the same way.
✨ Key New Features
🗂️ .env.example in three parts
- PART 1 · Your setup: check every line once per machine. Covers the server address, the RAM process pattern, the budget profile, the VRAM sweet spot and Docker model-file paths.
- PART 2 · Your run: what you change between runs. Covers models, datasets, profile / funnel, concurrency, prefill lengths, cache, sampling, the thinking budget and output files.
- PART 3 · Optional: everything else, grouped by topic. Every line is commented out and already at its built-in default.
- One line per setting, in one format:
KEY=value # what it does [default: x]: an active line;[default]is what you get if you delete it.# KEY=value # what it does: a commented line; the value shown is the built-in default.# KEY= # what it does: not set by default; the text says what happens then.
- All 141 settings the code reads are listed (plus the
BUDGET_<PROFILE>_<RULE>andEVAL_THINK_BUDGET_<DATASET>families). The old file was missingEVAL_<DATASET>_REVISIONfor five of the seven datasets.
🩺 python benchmark.py --check-env [file]
Reads files only; it changes nothing and contacts no server. It lists:
- Unknown settings (ignored by the benchmark), with the closest real name:
EVAL_TOOLCAL->EVAL_TOOLCALL?. Budget rules and per-dataset thinking budgets are checked by name too. - Values that won't work as intended:
tueor another non-boolean (read as false);- a word where a number is expected (the run would stop at start-up);
KEY= # comment, which python-dotenv reads as the value# comment.
- Old names (
STRATA_SWAP_URL), keys set twice (the last one wins), and keys your shell also sets (the shell wins). - PART 1 / PART 2 settings missing from your
.env(usually new in this version), with the default they fall back to. - Everything you changed from the built-in defaults.
It exits with 1 if it finds an unknown setting or an unusable value, so it can run before a scheduled run.
🔧 Fixes
- Documented defaults that didn't match the code (the code was right; the file was wrong):
- the commented
VRAM_SWEET_LOW=82/HIGH=89.5/MAX=95lines suggested defaults of 82 / 89.5 / 95. The built-in values are 80 / 90 / 94. - the commented
LOGIC_PROMPTshowed a "Physics Handbook" prompt; the built-in prompt is the "Biology textbook" one.
- the commented
- Built-in
.envreader (used only when python-dotenv isn't installed): a key set twice now takes the last value, as python-dotenv does. It used to take the first. Values with an inline comment are read exactly as python-dotenv reads them.
⚖️ How old and new results compare
Nothing that is measured or scored changed, and the result-cache key and GRADER_VERSION are untouched, so cached answers from v2.1.0 / v2.2.0 are reused.
Two values in the new .env.example differ from the old one. They matter only if you build a fresh .env from it:
VRAM_LIMIT_GBmoved to PART 3 at its default of 16. The old file set12. It is only used when nvidia-smi cannot report the VRAM total, so it does not affect a normal run.VRAM_SWEET_LOW=82/VRAM_SWEET_HIGH=89.5are now active lines in PART 1. Before, they were commented out, so the sweet spot used 80 / 90. A fresh.envbuilt from the new file therefore scores the sweet spot on 82–89.5. Your existing.envkeeps whatever it has. This changes the Sweet-spot score only, not the Agent score or any measurement.
🔄 Updating
- Copy the changed files over your install.
- Run
python benchmark.py --check-env. - Then either keep your
.envand add any settings it reports as missing, or start from the new layout:cp .env.example .env.new, copy your values across, and rename it.--check-envon the new file should then list only your intended changes.
⚙️ Configuration
- New command-line option:
--check-env [file]. - No new settings.
.env.examplenow carries a short header that explains the format.
🚀 llama-swap-llm-benchmark v2.3.0: an Agent score built for agents, prefill at depth, a clearer dashboard, and a settings page
The Agent score now ranks models the way an agent uses them: it requires enough context, prefers more, scores tool calling as its own part, and compares long-context retrieval on the same questions for every model. A new speed test measures what an agent waits for on every turn: new tokens on top of a long, cached conversation. The dashboard gets six colour tiers, a light theme, model families and a Help page. A settings page (python benchmark.py --ui) edits .env, keeps favourite setups, and starts or stops runs from the browser.
The Agent score changes in this release. How old and new results compare below shows what that does to rankings, using two real runs.
✨ Key New Features
🤖 Agent score for agentic use
-
Usable context, a minimum and a mild preference. The Agent score now uses the context a model can actually work with, not just the one it was loaded with. The long-context lengths of the run are walked upwards; every length that fits the loaded context must reach
AGENT_CTX_PASS_PCT(75%). If all of them pass, the usable context is the loaded context (a 90k model that finds 100% at 64k counts as 90k). If one fails, the usable context is the longest length passed before it (a 128k model that finds 12% at 128k works like a 64k model). A length without a result does not count as passed. A model whose usable context is belowAGENT_CTX_MIN(default 64k) gets no Agent score, so a model must pass the 64k test to be ranked. Above it, a mild context factor fromAGENT_CTX_CURVEmultiplies the score: 128k and up 1.0, 77k 0.97, 64k 0.95, linear in between (90k ≈ 0.98). The three settings are in PART 1, because what an agent needs depends on your setup: a very smart model that only fits with a short context can still be ranked by loweringAGENT_CTX_MINand adding an anchor.AGENT_CTX_PASS_PCT=0uses the loaded context;AGENT_CTX_MIN=0andAGENT_CTX_CURVE=offswitch the rest off. New column Usable ctx. -
Composite quality. The Agent score now combines three parts:
- evals (every dataset except tool calling) 45%;
- tool calling 30% (
AGENT_TOOL_WEIGHT); - long-context retrieval 25% (
AGENT_LONGCTX_WEIGHT).
A part the run doesn't have gives its weight to the evals. Before, tool calling was one dataset among many in the eval average, and long-context retrieval was only blended when every model had it.
-
Long-context retrieval on the same questions for every model. Only the lengths that every ranked model ran count. v2.2 averaged each model over its own lengths, so a 90k model that skipped the hard 128k test could beat a 128k model that passed it only partly. A model with no long-context result at 64k is not ranked (see above), so the others keep the long-context part. Only with
AGENT_CTX_PASS_PCT=0can a ranked model lack long-context; then it is left out for everyone and the dashboard says why. -
Tool-calling minimum. Below
MIN_TOOLCALL_PCT(default 70%), a model gets no Agent score: it can't reliably drive an agent. A model within its margin of error of the minimum is marked ≈ borderline. Tool calling now runs in full (all 32 cases) at every profile and funnel stage, so its margin is the same everywhere. -
Tie zone from the composite's own noise. A model within one standard error of the best composite counts as a tie at 100. The standard error comes from each part's question count; tool calling has only 32 cases, so it carries more noise than a large eval set. The funnel's confidence gates rank on the same composite.
-
Speed factor. It is now weighted prompt processing 50%, seconds per question 30%, generation 20% (before: 40 / 35 / 25). Prompt processing uses prefill at depth (60%) and the cold agent-prompt TTFT (40%), whichever exist.
AGENT_SPEED_WEIGHTstays 0.3: speed can still take off at most 30% and never lifts a clearly weaker model. -
New columns Composite, Long-ctx (Agent), Usable ctx and Ctx factor, plus a note above the table with the weights the run used.
⏱️ Prefill at depth (PREFILL_DEPTH, default auto)
- The speed phase keeps a long conversation in llama-server's prompt cache: with
auto, per model the largest of 8k / 16k / 32k / 64k that fits the loaded context with the 4k turn: a model loaded at 64k is measured at 32k, every 90k+ model at 64k (a number fixes the depth; 0 switches the test off). It then sends ~4,096 new tokens on top (a tool result) and measures only those new tokens. This is the wait on every turn of a long agent session, which the cold 8k–128k prefill tests don't show. - New columns Prefill @Depth t/s and @Depth TTFT s (CSV, JSON
context.prefill_depth_*, Markdown). Older history CSVs are migrated. - The result is recorded only when the server reused at least 90% of the cached prefix. When it didn't, the run says how much was reused. New column Depth shows the depth each model was measured at.
- It is part of the Agent score's speed factor: the credit halves at about 1,333 tok/s. Prefill is slower deeper into a context, so the score only compares models measured at the same depth (the one most models had); a model measured at another depth counts its cold agent-prompt TTFT instead, with a note.
🎨 Dashboard: colours, light theme, families, Help
-
Six tiers per metric, worst to best: red with a ⚠ sign, pink, amber, green, blue, purple.
- Generation tok/s: 30 / 40 / 50 / 80 / 100.
- Prefill tok/s: 500 / 1,000 / 1,500 / 2,500 / 4,000.
- Accuracy and scores: 50 / 60 / 70 / 80 / 90.
- Reversed scales for times, retries and power.
VRAM keeps its sweet-spot colours. The worst tier always shows ⚠, so colour is never the only signal.
-
Bands or Smooth (flat tiers, or a gradient between them) and Absolute or Relative (tiers spread over this run's own range, so a slow GPU still shows which models are faster). These are toggles at the top right and apply to the table and all charts.
COLOUR_STYLEandCOLOUR_MODEset the defaults;COLOUR_<METRIC>=a,b,c,d,esets your own thresholds (e.g.COLOUR_GEN=10,15,20,30,40for a small GPU). -
Light and dark theme. Auto follows your system. The light theme has its own tier colours, each with at least 4.5:1 contrast on white.
-
Model families. Chips come from
DASH_FAMILIES(Gemma=*gemma-4*;BigBang=*bigbang*) and automatically from model names. A wildcard box takes patterns:*gemma-4*also findsagent-gemma-4-26B…. -
? Help lists every threshold for the run, explains the toggles, and has a rough speed-by-setup table to read the tiers against your hardware.
-
"How scores work" describes the v2.3 Agent score.
🖥️ Settings page (python benchmark.py --ui)
- Shows every
.envsetting by PART and section, with its description, default,.env.examplevalue and allowed choices. Values are checked as you type. - Save checks the values, writes
.envin the.env.examplelayout and keeps the old file as.env.bak. Return to defaults asks first and keeps your tokens. - Tokens and keys are masked; show fetches one on request.
- Favourites live in
configs/<name>.env. They hold only the settings that differ from.env.example, never secrets. Your current.envis saved asmainon the first visit. Loading a favourite into a newer version follows renamed settings and asks you when both sides changed a value. - Models lists the server's models with tick boxes for include / exclude, their file size, parameters, trained context and architecture (GGUF header, else the last run), and last run's VRAM, context and Agent score.
- On opening, the page lists the server's models and runs
--check(once per browser session). - Estimate (with unsaved settings), Run --check again, Reload dashboard, Compare, Open latest dashboard, ? Help.
- Run and Stop. Full run, speed only or evals only. The page follows the run's log and picks it back up after a reload. Stop is Ctrl+C for the run, so partial results are written.
- Port 41010, localhost only by default. See Security below.
- Editing
.envby hand and running from the command line or Unraid User Scripts work exactly as before.
🧭 New defaults and guides
EVAL_PROFILE=funnelis the default and the recommended profile (wasquick). Every model answersquick; confidence gates drop the clearly weaker ones; the rest go on up toEVAL_FUNNEL_FINAL.EVAL_TEMPERATURE=serverandTEMPERATURE=serverare the defaults (were 0 and 0.2): no temperature is sent, so every model uses the sampling in its llama-swap config, as you deploy it. Put each model's recommended sampling in its llama-swap command.ESTIMATE_GPU_BW_GBS=auto, now in PART 1 (was 900 in PART 3). It looks up the detected NVIDIA GPU in a table of 22 common cards (RTX 3060 to 5090), else uses 670 GB/s; a number overrides it.- Guides on the settings page:
EVAL_PROFILE: why the funnel, how it works, the margin of error per test for every profile, and an Estimate button;EVAL_FUNNEL_FINAL: the stages and the finalists' margin for each choice;ESTIMATE_GPU_BW_GBS: the GPU table (click a row to use it);- both temperatures.
- A Batch probe button starts
--batch-probefrom the page.--batch-probenow writes a log (batch_probe_<stamp>.log) and the run lock, like a run.
🧰 Command line
--env FILE: use another complete settings file instead of.env(works with every flag).--validate: check that the settings load; prints OK or the error.--ui: the settings page.- A run now marks itself in
OUTPUT_DIR/.benchmark_run.json, so the settings page can see and stop it. A second run started on the same GPU prints a warning.
🔧 Fixes
- llama-swap on another machine:
--check, the start of a run with speed tests, and the settings page warn whenLLAMA_SWAP_URLpoints at another machine. The speed test reads GPU, VRAM, power and RAM locally, so it must run on the machine with the GPU; evals-only runs are fine from elsewhere. The page asks before starting a speed run. --reload,--compareand runs no longer fail when the "latest" copy next tobenchmark.pycan't be written (e.g. root-owned on a share). They warn and keep the dashboard inresults/; the settings page opens the newest one. The page's "Open the combined dashboard" link after Compare now works.- A model that fails to load is flagged Failed to load (cause) in the log and on the dashboard (badge with the full hint). Causes: out of memory, bad launch flag, model file not found, unsupported model, load timed out, or not reported. Each comes with its fix; out of memory suggests
--fit on(with-c/-nglremoved) or a lower-c. It shows llama-swap's own error lines and, when the GGUF is readable, its size, trained context and KV cost. --reload/--compareof an older run. Rescoring a v2.0 JSON could crash on a staleeval_quality;finalize()now clears it first.- Docs: interrupted runs. The v2.1 release notes said an interrupted run can't be reloaded. A run stopped with Ctrl+C writes its JSON, CSV and dashboard with the models finished so far, and
--reloadworks on it. Only a hard kill (kill -9, a closed terminal, a power cut) leaves nothing to reload. The v2.1 notes and the README are corrected. GOLDILOCKS_MODEis now checked (linearorlog); a typo used to fall back silently.UI_HOSTwritten as a URL (http://192.168.2.200) failed with "Name or service not known". It is now cut down to the address with a note,--check-envand the settings page flag it, and the error names the real cause: not this machine's address, port in use, or port not allowed. The LAN token is saved to.envonly once the page has opened.--demowrote its synthetic results intoresults/:results.json, abenchmark_<stamp>.jsonand the latest dashboard. A following--reloadrebuilt the demo instead of the last real run, and--estimateand the settings page read the demo files. The demo now writes toresults/demo/and leaves the latest dashboard alone.- New
requirements-dev.txt(the normal requirements plus pytest) for running the self-tests.
🔒 Security (settings page)
- Binds
127.0.0.1unlessUI_HOSTis set. For the LAN, a token is required: it is generated on the first start, saved asUI_TOKEN, and kept as an HttpOnly, SameSite=Strict cookie after the first visit. - Requests whose
Hostisn't an IP address,localhostor this machine's name are refused (DNS rebinding). - Every change needs the page's own header and a matching
Origin, so other websites in your browser can't drive it. - Only a fixed set of the tool's own commands can run. The page never takes a program or a file path, and runs to compare are picked from its own list.
- Unsaved settings for Estimate go to a private temporary file (mode 0600) that is deleted straight after.
- Plain HTTP: on the LAN the token, and any token you reveal, travel unencrypted. Fine on a home network; use a reverse proxy with HTTPS for more, and don't forward the port to the internet.
Defaults agreed for this release: AGENT_CTX_MIN=65536, AGENT_CTX_PASS_PCT=75, AGENT_CTX_CURVE=131072:1.0,78848:0.97,65536:0.95 (agreed 2026-11-10: usable context, mild preference), AGENT_TOOL_WEIGHT=0.30, AGENT_LONGCTX_WEIGHT=0.25 (unchanged), EVAL_PROFILE=funnel (was quick), EVAL_TEMPERATURE=server (was 0), TEMPERATURE=server (was 0.2), ESTIMATE_GPU_BW_GBS=auto (was 900), MIN_TOOLCALL_PCT=70, AGENT_SPEED_WEIGHT=0.3 (unchanged), speed factor 0.5 / 0.3 / 0.2, PREFILL_DEPTH=auto, COLOUR_STYLE=bands, COLOUR_MODE=absolute, UI_PORT=41010, localhost only.
Other display changes: the retry warning (⚠) now shows above 25% retried (was 20%), the red tier of the new retry scale; set COLOUR_RETRY to change it.
🔄 Updating
- Copy the changed files over your install.
llmbench/ui/,llmbench/agent.py,llmbench/envfile.py,llmbench/gpubw.py,llmbench/reporting/colours.pyandconfigs/are new. - Run
python benchmark.py --check-env. It lists the new PART 1 settingsAGENT_CTX_MIN,AGENT_CTX_PASS_PCTandAGENT_CTX_CURVE, and the PART 2 settingPREFILL_DEPTH. Or runpython benchmark.py --ui: the page shows the same list, and Save adds them for you (your old file stays as.env.bak). - Decide whether 64k is the right minimum for your agents. If a model has to run at 32k (a large, smart model that only fits that way), set
AGENT_CTX_MIN=32768and add a32768:…point toAGENT_CTX_CURVE; with long-context tests in the run, it must still pass every length it can hold. - Sampling: your
.envkeeps whateverEVAL_TEMPERATUREit has. To move to the new default, setEVAL_TEMPERATURE=serveronce each model's llama-swap command carries its recommended sampling; expect one full re-ask of the evals. --reloadof an older run keeps the scores it stored.--compareof two older runs rescores both with the v2.3 rules (without prefill at depth, which they didn't measure).
⚙️ Configuration
- New in PART 1:
AGENT_CTX_MIN,AGENT_CTX_PASS_PCT,AGENT_CTX_CURVE;ESTIMATE_GPU_BW_GBSmoved here from PART 3. - Changed defaults:
EVAL_PROFILEfunnel,EVAL_TEMPERATUREandTEMPERATUREserver,ESTIMATE_GPU_BW_GBSauto (see above). - New in PART 2:
PREFILL_DEPTH. - New in PART 3:
AGENT_TOOL_WEIGHT,MIN_TOOLCALL_PCT(Agent score);COLOUR_STYLE,COLOUR_MODE,COLOUR_<METRIC>(GEN, PREFILL, ACC, SCORE, TTFT_MS, AGENT_TTFT, DEPTH_TTFT, SQ, RETRY, POWER),DASH_FAMILIES(dashboard view);UI_PORT,UI_HOST,UI_TOKEN(settings page).
.env.examplelists 153 settings plus three families (141 before).--check-envand the settings page know the allowed values of every setting that has a fixed set.- New command-line options:
--ui,--env FILE,--validate. - No third-party code or data was added. The settings page uses only Python's standard library; the dashboard still loads Chart.js from cdnjs.
🛟 llama-swap-llm-benchmark v2.3.1: runs that survive trouble, --resume, honest timings, a harder settings page
A long run used to have three ways to lose or distort its results when something went wrong: an unexpected error threw away the JSON and CSV, a llama-swap restart cost the models tested in that window their ranking, and one failed dataset quietly changed which latency every model was scored on. This release fixes all three, adds --resume, and tightens the settings page and the code sandbox.
✨ Key New Features
🛟 Nothing measured is lost, and --resume
- Rows are saved after every model.
results/run_<stamp>.rows.jsonis rewritten after each model of the speed phase and each model of every eval stage. A crash,kill -9or a power cut loses at most the model being tested. python benchmark.py --resumecontinues the newest run inOUTPUT_DIR(or--resume 20261110_103848):- models whose speed test finished are kept and not benchmarked again;
- models that failed, or were never reached, are benchmarked;
- the result cache answers every eval question already asked, so only the missing ones are sent;
- a long-context result that is complete is kept; a missing one is run.
- It also repairs a finished run. A finished run keeps its rows file, so
--resumere-tests only what went wrong in it: a model that failed to load, lost its long-context test, or has incomplete evals. - The resumed run gets its own stamp and files;
artifacts.resumed_fromin the JSON names the run it continued. Runs made before v2.3.1 have no rows file and cannot be resumed (their eval answers are still in the result cache). - An unexpected internal error no longer discards the run. In the speed phase the model is marked
errorand the next one runs. In the eval phase everything measured so far is written (statusinterrupted). - SIGTERM is handled like Ctrl+C (Unraid "abort script",
docker stop, systemd): partial results are written. - Exit codes:
0complete,2stopped early by an error with partial results written,130stopped by Ctrl+C / Stop / SIGTERM,1could not run.
🔌 llama-swap goes away mid-run
SERVER_WAIT_SECONDS(default 600; 0 = off). When requests fail because the server is not there (refused, dropped, 5xx), the run waits for it to list its models again, reloads the model, and:- asks the failed questions, and the ones never asked because the dataset stopped, once more;
- repeats a long-context test that failed on a connection error;
- benchmarks a model again whose speed test failed that way.
- Timeouts are the model's own behaviour and are not asked again. Re-asked questions are counted in
eval_reasked_nand marked"reasked": trueineval_details_*.jsonl.
⏱️ Timings measured over the right window
- A retry is no longer inside the measurement. Latency, energy, J/token and GPU watts of a speed request are taken over the attempt that answered, not over a failed attempt plus its backoff.
speed_retriescounts them. - Durations use a clock that cannot step. Every interval and telemetry window uses a monotonic clock; an NTP correction during a long run no longer shifts them.
- "TTFT ms" is now "Prompt ms". The column was always llama-server's prompt-processing time, not what a client waits. The evals now record what the client sees through llama-swap, per dataset in the JSON:
ttft(median seconds to the first token),ttfa(to the first visible answer token, after the thinking) andttfa95. Display only: no score uses them. - Spread between runs.
tps_sampleskeeps every speed run and TPS spread % shows how far apart they were. - How tokens/s was obtained (
tps_basis):serverfrom llama-server's timings,wallwhen a backend reports none (that figure includes prompt processing). - Change since the last run. TPS vs last % compares with the previous run of the same model file at the same context and offload; changes of 5% or more are printed with the llama.cpp builds (
build_info). - A deferred prefill figure (taken from the long-context test) is recorded only when the prompt was cold; very short requests no longer read a telemetry sample from before they started.
📟 Following a run
results/run_status.json: phase, model i of N, dataset, questions done, started / updated, host, pid. The settings page shows it above the log; a script or a Home Assistant sensor can read it.NOTIFY_URL: a short JSON summary is POSTed there when a run ends (status, duration, models, failed models, top three). No settings, tokens or answers.- New flags for one-off runs:
--models,--exclude,--profile,--output-dir(the same as settingINCLUDE_MODELS,EXCLUDE_MODELS,EVAL_PROFILE,OUTPUT_DIRfor that run). - A second speed run is refused while another run is going (
--forcestarts anyway). The lock now carries the host name and a heartbeat, so a run started on another machine over a share is seen as live. Evals-only runs are still only warned. - Model names such as
…-E4B-…(effective parameters) are read as 4B when the name has no plain size.
📊 Dashboard
- Chart.js 4.4.1 is written into every dashboard. It opens without internet access and no longer loads a script from a CDN. Each file grows by about 200 KB.
🔧 Fixes
- One dataset with no real answers switched the whole run from the single-stream s/q estimate to measured latency (see below).
- Answers cached at one concurrency and reused in a run at another: the s/q basis now follows the concurrency the answers were measured at.
- The Home Assistant power poller kept every sample for the whole run and scanned them all per request.
- NVML was shut down while its polling thread could still be reading.
- A run that failed at once was reported as "The run ended". A Full run started from the settings page under a Python without the
datasetspackage died in its first second and the page said nothing. The page now shows a red "The run failed" with the last lines of the log (amber for a run stopped early, green for a finished one), and--checkprintsHugging Face datasets: NOT AVAILABLEwith the install command when an eval needs it. If you use a virtual environment, start--uifrom it: the page runs the benchmark with the Python that started the page. KEY= # commentis read as empty. python-dotenv reads an empty value followed by a comment as the text of the comment (aUI_TOKEN= # ...line became the token). The benchmark and the settings page now read it as empty.--check-envstill flags the line; writeKEY=""or put the comment on its own line.
🔒 Security
- Settings page on the LAN: a
UI_TOKENshorter than 16 characters is refused at start (delete the line and a random one is generated). Ten wrong tokens from one address lock that address out for five minutes. - Dashboards served by the settings page are sandboxed (
Content-Security-Policy: sandbox): their scripts cannot reach the page's cookie or API. .env,.env.bakand.env.tmpare created readable by their owner only (0600), whatever the umask.- The page only reads a run log inside
OUTPUT_DIR(benchmark_*.log,batch_probe_*.log); the path in the lock file is no longer trusted. - Subprocess sandbox as root: model-written code is started as the unprivileged user
nobody. If that is not possible a warning says the code runs as root. - The run prints the sandbox image's ID and reminds you that a tag can change; pin one with
EVAL_CODE_DOCKER_IMAGE=<image>@sha256:<digest>. run_<stamp>.rows.jsonnever stores the value of--api-key.
🔄 Updating
- Copy the files over your install. Your
.env,results/and cache are untouched. python benchmark.py --check-envlists the two new settings.- If your settings page is open to the LAN with a short
UI_TOKEN, it will not start until the token has 16 characters or the line is deleted. - A cron or User Script that tests the exit code: Ctrl+C / Stop now ends with 130, an early stop with 2.
⚙️ Configuration
| Setting | Default | What it does |
|---|---|---|
SERVER_WAIT_SECONDS |
600 |
How long to wait for llama-swap to answer again mid-run before giving up on the failed requests; 0 = off |
NOTIFY_URL |
empty | POST a JSON summary there when a run ends |