Skip to content

v2.3.1: Agent score, a settings page that saves settings and runs the test, numerous improvements (v2.1 → v2.3.1)

Choose a tag to compare

@Maximo101 Maximo101 released this 10 Nov 12:21
· 44 commits to main since this release
098d910

First release since v2.1.0. It rolls up v2.2.0, v2.2.1, v2.3.0 and v2.3.1.

🚀 llama-swap-llm-benchmark v2.2.0 — reuse what you have, check it, and compare across servers

v2.1 introduced the result cache, so a rerun asks only what is missing. v2.2 adds a full retest, a sanity check of cached answers, an --estimate that knows what the cache already holds, and --compare, which puts runs from different servers in one dashboard (for example a model that only fits on Strata next to your llama-swap models). Scores are computed exactly as in v2.1.

✨ Key New Features

🔁 Full retest (EVAL_RESULT_CACHE=refresh)

  • EVAL_RESULT_CACHE now has three modes:
    • true (default): reuse cached answers; only missing or failed questions are asked.
    • refresh: full retest. Every question is asked again, and the new answers replace the cached ones, so the next run can reuse them.
    • false: no cache at all; nothing is reused or stored.
  • Speed tests and the long-context test are always measured again, in every mode.

🩺 Sanity check of cached answers (EVAL_SANITY_PCT)

  • EVAL_SANITY_PCT (0–100, default 0 = off) re-asks that share of the cached answers at random and compares each with the cached one:
    • Code and tool calls must have the same pass/fail; multiple choice and math the same extracted answer and verdict.
    • The new answer is the one scored and stored.
    • Differences are listed in the log ([sanity] <question>: cached ..., now ...), summed per model at the end of the run, and shown in a new Sanity diff dashboard column.
  • EVAL_SANITY_ADAPTIVE=true (default): each model starts at EVAL_SANITY_PCT. Every matching answer halves its rate in whole percents, and below 2% the checks stop for that model for the rest of the run (from 20%: 20 → 10 → 5 → 2 → 0). A different answer puts it back to the full rate. Models that keep answering the same are checked less and less; models that change keep being checked. Each run starts again at the full rate.
  • The questions picked differ per run (the seed is printed), so repeated runs check different questions.

🎛️ Each model's own sampling (EVAL_TEMPERATURE=server, EVAL_SERVER_SAMPLING_MODELS)

  • Until now every eval request carried EVAL_TEMPERATURE (default 0), which overrides the --temp in your llama-swap commands. Some reasoning models, Gemma-family especially, loop under greedy decoding: in a real run Gemma 4 26B and a Gemma 4 fine-tune needed a retry on 32% and 48% of questions, failed the retry budget rule and were dropped from the ranking, although every answer they finished cleanly was right.
  • EVAL_TEMPERATURE=server sends no temperature, so every model uses the sampling in its own llama-swap command, with EVAL_SEED as the sampling seed. This measures each model as you deploy it.
  • EVAL_SERVER_SAMPLING_MODELS=<regex> does this only for the matching models; the rest keep EVAL_TEMPERATURE, and only the matching models are asked again.
  • A new Sampling column shows what each model was evaluated with, e.g. temp 0 or server: temp 1.0, top-k 64, top-p 0.95.

⏱️ --estimate knows what the cache holds

  • It says whether the run will reuse cached answers or is a full re-run (refresh / false).
  • Per model, it shows how many answers it expects to reuse against the questions it expects to ask, and counts time only for the rest, plus any sanity-check re-asks.
  • The one-line estimate at the start of every run says the same.

⚖️ Compare runs in one dashboard (--compare)

  • python benchmark.py --compare combines the newest llama-swap run with the newest Strata run. python benchmark.py --compare A.json B.json ... combines the runs you name; the first is the base.
  • Every model keeps its own measurements. The scores that compare models with each other (Agent score, sweet spot, budget eligibility) are computed again over all models together, so the ranking matches a single run with all of them.
  • Runs whose eval settings differ (seed, temperature, thinking budget, streaming switches) are refused, because they asked different questions or measured differently. --force combines them anyway, and the dashboard notes it.
  • A model missing a dataset or questions shows "not ranked: incomplete evals", as within one run. A model in both runs is shown twice, labelled with its run.
  • The dashboard gets a Run column and a Run filter to show or hide each run's models.
  • Output: results/benchmark_compare_<stamp>.html, copied to benchmark_compare.html.
  • Tip: run both sides with the same eval settings. With EVAL_PROFILE=funnel, a run with a single model goes straight to the final profile, so a lone Strata model answers the same questions as a llama-swap run's finalists.

🧹 Small changes

  • Two .env.example section headings carried stale "(v2.2)" tags for settings that shipped in v2.0; removed.
  • results.json records the cache mode and sanity settings (evaluation.result_cache_mode, sanity_pct, sanity_adaptive) and, per model, sanity_checked and sanity_different. Per-question logs mark re-asked questions with sanity (same / different) and the cached answer in sanity_cached.

⚖️ How old and new results compare

  • Scoring is unchanged. Runs with the same settings give the same scores as v2.1.
  • Cached answers from v2.1 runs stay valid. The cache key is unchanged, so a v2.2 run reuses everything a v2.1.0 run answered.
  • With EVAL_SANITY_PCT > 0, a re-asked question is scored on its new answer. When decoding is not fully deterministic (batched requests, temperature > 0), a score can move by the questions that changed, the same noise a fresh run would show.
  • With own sampling (EVAL_TEMPERATURE=server or EVAL_SERVER_SAMPLING_MODELS), those models are measured with their deployed sampling instead of greedy decoding. Their answers are no longer bit-identical between runs, so small differences mean less. Models left at EVAL_TEMPERATURE are unchanged and keep their cached answers.
  • --compare recomputes the comparative scores over the combined set. A model's Agent score there can differ from its score in its own run, because the best model it is compared with can be different.

🔄 Updating

  1. Pull the new version. There are no new dependencies.
  2. Optional: copy the new settings from .env.example into your .env. The defaults behave like v2.1 (EVAL_RESULT_CACHE=true, EVAL_SANITY_PCT=0).
  3. If you used EVAL_RESULT_CACHE=false to force a full retest, use refresh instead, so the new answers are kept for later runs.
  4. Keep the llama-swap / llama.cpp image unchanged between runs you want to reuse: the llama.cpp build is part of the cache key.

⚙️ Configuration

EVAL_RESULT_CACHE=true           # true | refresh (full retest) | false
EVAL_TEMPERATURE=0               # a number, or server = each model's own llama-swap sampling
EVAL_SERVER_SAMPLING_MODELS=     # regex: only these models use their own sampling
EVAL_SANITY_PCT=0                # % of cached answers re-asked and compared (0-100)
EVAL_SANITY_ADAPTIVE=true        # halve a model's rate after each matching answer

New commands: --compare (with --force).

🚀 llama-swap-llm-benchmark v2.2.1: a settings file you can read at a glance, and a check for your .env

A tidy-up release. .env.example is reorganised so it is clear which settings you must check, which ones you change per run, and which are optional, with the built-in default shown for every one. A new --check-env compares your .env with it. Measurements, scores and the result cache are unchanged.

This prepares v2.3, which will add a settings page that reads .env.example in the same way.

✨ Key New Features

🗂️ .env.example in three parts

  • PART 1 · Your setup: check every line once per machine. Covers the server address, the RAM process pattern, the budget profile, the VRAM sweet spot and Docker model-file paths.
  • PART 2 · Your run: what you change between runs. Covers models, datasets, profile / funnel, concurrency, prefill lengths, cache, sampling, the thinking budget and output files.
  • PART 3 · Optional: everything else, grouped by topic. Every line is commented out and already at its built-in default.
  • One line per setting, in one format:
    • KEY=value # what it does [default: x]: an active line; [default] is what you get if you delete it.
    • # KEY=value # what it does: a commented line; the value shown is the built-in default.
    • # KEY= # what it does: not set by default; the text says what happens then.
  • All 141 settings the code reads are listed (plus the BUDGET_<PROFILE>_<RULE> and EVAL_THINK_BUDGET_<DATASET> families). The old file was missing EVAL_<DATASET>_REVISION for five of the seven datasets.

🩺 python benchmark.py --check-env [file]

Reads files only; it changes nothing and contacts no server. It lists:

  • Unknown settings (ignored by the benchmark), with the closest real name: EVAL_TOOLCAL -> EVAL_TOOLCALL?. Budget rules and per-dataset thinking budgets are checked by name too.
  • Values that won't work as intended:
    • tue or another non-boolean (read as false);
    • a word where a number is expected (the run would stop at start-up);
    • KEY= # comment, which python-dotenv reads as the value # comment.
  • Old names (STRATA_SWAP_URL), keys set twice (the last one wins), and keys your shell also sets (the shell wins).
  • PART 1 / PART 2 settings missing from your .env (usually new in this version), with the default they fall back to.
  • Everything you changed from the built-in defaults.

It exits with 1 if it finds an unknown setting or an unusable value, so it can run before a scheduled run.

🔧 Fixes

  • Documented defaults that didn't match the code (the code was right; the file was wrong):
    • the commented VRAM_SWEET_LOW=82 / HIGH=89.5 / MAX=95 lines suggested defaults of 82 / 89.5 / 95. The built-in values are 80 / 90 / 94.
    • the commented LOGIC_PROMPT showed a "Physics Handbook" prompt; the built-in prompt is the "Biology textbook" one.
  • Built-in .env reader (used only when python-dotenv isn't installed): a key set twice now takes the last value, as python-dotenv does. It used to take the first. Values with an inline comment are read exactly as python-dotenv reads them.

⚖️ How old and new results compare

Nothing that is measured or scored changed, and the result-cache key and GRADER_VERSION are untouched, so cached answers from v2.1.0 / v2.2.0 are reused.

Two values in the new .env.example differ from the old one. They matter only if you build a fresh .env from it:

  • VRAM_LIMIT_GB moved to PART 3 at its default of 16. The old file set 12. It is only used when nvidia-smi cannot report the VRAM total, so it does not affect a normal run.
  • VRAM_SWEET_LOW=82 / VRAM_SWEET_HIGH=89.5 are now active lines in PART 1. Before, they were commented out, so the sweet spot used 80 / 90. A fresh .env built from the new file therefore scores the sweet spot on 82–89.5. Your existing .env keeps whatever it has. This changes the Sweet-spot score only, not the Agent score or any measurement.

🔄 Updating

  1. Copy the changed files over your install.
  2. Run python benchmark.py --check-env.
  3. Then either keep your .env and add any settings it reports as missing, or start from the new layout: cp .env.example .env.new, copy your values across, and rename it. --check-env on the new file should then list only your intended changes.

⚙️ Configuration

  • New command-line option: --check-env [file].
  • No new settings. .env.example now carries a short header that explains the format.

🚀 llama-swap-llm-benchmark v2.3.0: an Agent score built for agents, prefill at depth, a clearer dashboard, and a settings page

The Agent score now ranks models the way an agent uses them: it requires enough context, prefers more, scores tool calling as its own part, and compares long-context retrieval on the same questions for every model. A new speed test measures what an agent waits for on every turn: new tokens on top of a long, cached conversation. The dashboard gets six colour tiers, a light theme, model families and a Help page. A settings page (python benchmark.py --ui) edits .env, keeps favourite setups, and starts or stops runs from the browser.

The Agent score changes in this release. How old and new results compare below shows what that does to rankings, using two real runs.

✨ Key New Features

🤖 Agent score for agentic use

  • Usable context, a minimum and a mild preference. The Agent score now uses the context a model can actually work with, not just the one it was loaded with. The long-context lengths of the run are walked upwards; every length that fits the loaded context must reach AGENT_CTX_PASS_PCT (75%). If all of them pass, the usable context is the loaded context (a 90k model that finds 100% at 64k counts as 90k). If one fails, the usable context is the longest length passed before it (a 128k model that finds 12% at 128k works like a 64k model). A length without a result does not count as passed. A model whose usable context is below AGENT_CTX_MIN (default 64k) gets no Agent score, so a model must pass the 64k test to be ranked. Above it, a mild context factor from AGENT_CTX_CURVE multiplies the score: 128k and up 1.0, 77k 0.97, 64k 0.95, linear in between (90k ≈ 0.98). The three settings are in PART 1, because what an agent needs depends on your setup: a very smart model that only fits with a short context can still be ranked by lowering AGENT_CTX_MIN and adding an anchor. AGENT_CTX_PASS_PCT=0 uses the loaded context; AGENT_CTX_MIN=0 and AGENT_CTX_CURVE=off switch the rest off. New column Usable ctx.

  • Composite quality. The Agent score now combines three parts:

    • evals (every dataset except tool calling) 45%;
    • tool calling 30% (AGENT_TOOL_WEIGHT);
    • long-context retrieval 25% (AGENT_LONGCTX_WEIGHT).

    A part the run doesn't have gives its weight to the evals. Before, tool calling was one dataset among many in the eval average, and long-context retrieval was only blended when every model had it.

  • Long-context retrieval on the same questions for every model. Only the lengths that every ranked model ran count. v2.2 averaged each model over its own lengths, so a 90k model that skipped the hard 128k test could beat a 128k model that passed it only partly. A model with no long-context result at 64k is not ranked (see above), so the others keep the long-context part. Only with AGENT_CTX_PASS_PCT=0 can a ranked model lack long-context; then it is left out for everyone and the dashboard says why.

  • Tool-calling minimum. Below MIN_TOOLCALL_PCT (default 70%), a model gets no Agent score: it can't reliably drive an agent. A model within its margin of error of the minimum is marked ≈ borderline. Tool calling now runs in full (all 32 cases) at every profile and funnel stage, so its margin is the same everywhere.

  • Tie zone from the composite's own noise. A model within one standard error of the best composite counts as a tie at 100. The standard error comes from each part's question count; tool calling has only 32 cases, so it carries more noise than a large eval set. The funnel's confidence gates rank on the same composite.

  • Speed factor. It is now weighted prompt processing 50%, seconds per question 30%, generation 20% (before: 40 / 35 / 25). Prompt processing uses prefill at depth (60%) and the cold agent-prompt TTFT (40%), whichever exist. AGENT_SPEED_WEIGHT stays 0.3: speed can still take off at most 30% and never lifts a clearly weaker model.

  • New columns Composite, Long-ctx (Agent), Usable ctx and Ctx factor, plus a note above the table with the weights the run used.

⏱️ Prefill at depth (PREFILL_DEPTH, default auto)

  • The speed phase keeps a long conversation in llama-server's prompt cache: with auto, per model the largest of 8k / 16k / 32k / 64k that fits the loaded context with the 4k turn: a model loaded at 64k is measured at 32k, every 90k+ model at 64k (a number fixes the depth; 0 switches the test off). It then sends ~4,096 new tokens on top (a tool result) and measures only those new tokens. This is the wait on every turn of a long agent session, which the cold 8k–128k prefill tests don't show.
  • New columns Prefill @Depth t/s and @Depth TTFT s (CSV, JSON context.prefill_depth_*, Markdown). Older history CSVs are migrated.
  • The result is recorded only when the server reused at least 90% of the cached prefix. When it didn't, the run says how much was reused. New column Depth shows the depth each model was measured at.
  • It is part of the Agent score's speed factor: the credit halves at about 1,333 tok/s. Prefill is slower deeper into a context, so the score only compares models measured at the same depth (the one most models had); a model measured at another depth counts its cold agent-prompt TTFT instead, with a note.

🎨 Dashboard: colours, light theme, families, Help

  • Six tiers per metric, worst to best: red with a ⚠ sign, pink, amber, green, blue, purple.

    • Generation tok/s: 30 / 40 / 50 / 80 / 100.
    • Prefill tok/s: 500 / 1,000 / 1,500 / 2,500 / 4,000.
    • Accuracy and scores: 50 / 60 / 70 / 80 / 90.
    • Reversed scales for times, retries and power.

    VRAM keeps its sweet-spot colours. The worst tier always shows ⚠, so colour is never the only signal.

  • Bands or Smooth (flat tiers, or a gradient between them) and Absolute or Relative (tiers spread over this run's own range, so a slow GPU still shows which models are faster). These are toggles at the top right and apply to the table and all charts. COLOUR_STYLE and COLOUR_MODE set the defaults; COLOUR_<METRIC>=a,b,c,d,e sets your own thresholds (e.g. COLOUR_GEN=10,15,20,30,40 for a small GPU).

  • Light and dark theme. Auto follows your system. The light theme has its own tier colours, each with at least 4.5:1 contrast on white.

  • Model families. Chips come from DASH_FAMILIES (Gemma=*gemma-4*;BigBang=*bigbang*) and automatically from model names. A wildcard box takes patterns: *gemma-4* also finds agent-gemma-4-26B….

  • ? Help lists every threshold for the run, explains the toggles, and has a rough speed-by-setup table to read the tiers against your hardware.

  • "How scores work" describes the v2.3 Agent score.

🖥️ Settings page (python benchmark.py --ui)

  • Shows every .env setting by PART and section, with its description, default, .env.example value and allowed choices. Values are checked as you type.
  • Save checks the values, writes .env in the .env.example layout and keeps the old file as .env.bak. Return to defaults asks first and keeps your tokens.
  • Tokens and keys are masked; show fetches one on request.
  • Favourites live in configs/<name>.env. They hold only the settings that differ from .env.example, never secrets. Your current .env is saved as main on the first visit. Loading a favourite into a newer version follows renamed settings and asks you when both sides changed a value.
  • Models lists the server's models with tick boxes for include / exclude, their file size, parameters, trained context and architecture (GGUF header, else the last run), and last run's VRAM, context and Agent score.
  • On opening, the page lists the server's models and runs --check (once per browser session).
  • Estimate (with unsaved settings), Run --check again, Reload dashboard, Compare, Open latest dashboard, ? Help.
  • Run and Stop. Full run, speed only or evals only. The page follows the run's log and picks it back up after a reload. Stop is Ctrl+C for the run, so partial results are written.
  • Port 41010, localhost only by default. See Security below.
  • Editing .env by hand and running from the command line or Unraid User Scripts work exactly as before.

🧭 New defaults and guides

  • EVAL_PROFILE=funnel is the default and the recommended profile (was quick). Every model answers quick; confidence gates drop the clearly weaker ones; the rest go on up to EVAL_FUNNEL_FINAL.
  • EVAL_TEMPERATURE=server and TEMPERATURE=server are the defaults (were 0 and 0.2): no temperature is sent, so every model uses the sampling in its llama-swap config, as you deploy it. Put each model's recommended sampling in its llama-swap command.
  • ESTIMATE_GPU_BW_GBS=auto, now in PART 1 (was 900 in PART 3). It looks up the detected NVIDIA GPU in a table of 22 common cards (RTX 3060 to 5090), else uses 670 GB/s; a number overrides it.
  • Guides on the settings page:
    • EVAL_PROFILE: why the funnel, how it works, the margin of error per test for every profile, and an Estimate button;
    • EVAL_FUNNEL_FINAL: the stages and the finalists' margin for each choice;
    • ESTIMATE_GPU_BW_GBS: the GPU table (click a row to use it);
    • both temperatures.
  • A Batch probe button starts --batch-probe from the page. --batch-probe now writes a log (batch_probe_<stamp>.log) and the run lock, like a run.

🧰 Command line

  • --env FILE: use another complete settings file instead of .env (works with every flag).
  • --validate: check that the settings load; prints OK or the error.
  • --ui: the settings page.
  • A run now marks itself in OUTPUT_DIR/.benchmark_run.json, so the settings page can see and stop it. A second run started on the same GPU prints a warning.

🔧 Fixes

  • llama-swap on another machine: --check, the start of a run with speed tests, and the settings page warn when LLAMA_SWAP_URL points at another machine. The speed test reads GPU, VRAM, power and RAM locally, so it must run on the machine with the GPU; evals-only runs are fine from elsewhere. The page asks before starting a speed run.
  • --reload, --compare and runs no longer fail when the "latest" copy next to benchmark.py can't be written (e.g. root-owned on a share). They warn and keep the dashboard in results/; the settings page opens the newest one. The page's "Open the combined dashboard" link after Compare now works.
  • A model that fails to load is flagged Failed to load (cause) in the log and on the dashboard (badge with the full hint). Causes: out of memory, bad launch flag, model file not found, unsupported model, load timed out, or not reported. Each comes with its fix; out of memory suggests --fit on (with -c / -ngl removed) or a lower -c. It shows llama-swap's own error lines and, when the GGUF is readable, its size, trained context and KV cost.
  • --reload / --compare of an older run. Rescoring a v2.0 JSON could crash on a stale eval_quality; finalize() now clears it first.
  • Docs: interrupted runs. The v2.1 release notes said an interrupted run can't be reloaded. A run stopped with Ctrl+C writes its JSON, CSV and dashboard with the models finished so far, and --reload works on it. Only a hard kill (kill -9, a closed terminal, a power cut) leaves nothing to reload. The v2.1 notes and the README are corrected.
  • GOLDILOCKS_MODE is now checked (linear or log); a typo used to fall back silently.
  • UI_HOST written as a URL (http://192.168.2.200) failed with "Name or service not known". It is now cut down to the address with a note, --check-env and the settings page flag it, and the error names the real cause: not this machine's address, port in use, or port not allowed. The LAN token is saved to .env only once the page has opened.
  • --demo wrote its synthetic results into results/: results.json, a benchmark_<stamp>.json and the latest dashboard. A following --reload rebuilt the demo instead of the last real run, and --estimate and the settings page read the demo files. The demo now writes to results/demo/ and leaves the latest dashboard alone.
  • New requirements-dev.txt (the normal requirements plus pytest) for running the self-tests.

🔒 Security (settings page)

  • Binds 127.0.0.1 unless UI_HOST is set. For the LAN, a token is required: it is generated on the first start, saved as UI_TOKEN, and kept as an HttpOnly, SameSite=Strict cookie after the first visit.
  • Requests whose Host isn't an IP address, localhost or this machine's name are refused (DNS rebinding).
  • Every change needs the page's own header and a matching Origin, so other websites in your browser can't drive it.
  • Only a fixed set of the tool's own commands can run. The page never takes a program or a file path, and runs to compare are picked from its own list.
  • Unsaved settings for Estimate go to a private temporary file (mode 0600) that is deleted straight after.
  • Plain HTTP: on the LAN the token, and any token you reveal, travel unencrypted. Fine on a home network; use a reverse proxy with HTTPS for more, and don't forward the port to the internet.

Defaults agreed for this release: AGENT_CTX_MIN=65536, AGENT_CTX_PASS_PCT=75, AGENT_CTX_CURVE=131072:1.0,78848:0.97,65536:0.95 (agreed 2026-11-10: usable context, mild preference), AGENT_TOOL_WEIGHT=0.30, AGENT_LONGCTX_WEIGHT=0.25 (unchanged), EVAL_PROFILE=funnel (was quick), EVAL_TEMPERATURE=server (was 0), TEMPERATURE=server (was 0.2), ESTIMATE_GPU_BW_GBS=auto (was 900), MIN_TOOLCALL_PCT=70, AGENT_SPEED_WEIGHT=0.3 (unchanged), speed factor 0.5 / 0.3 / 0.2, PREFILL_DEPTH=auto, COLOUR_STYLE=bands, COLOUR_MODE=absolute, UI_PORT=41010, localhost only.

Other display changes: the retry warning (⚠) now shows above 25% retried (was 20%), the red tier of the new retry scale; set COLOUR_RETRY to change it.

🔄 Updating

  1. Copy the changed files over your install. llmbench/ui/, llmbench/agent.py, llmbench/envfile.py, llmbench/gpubw.py, llmbench/reporting/colours.py and configs/ are new.
  2. Run python benchmark.py --check-env. It lists the new PART 1 settings AGENT_CTX_MIN, AGENT_CTX_PASS_PCT and AGENT_CTX_CURVE, and the PART 2 setting PREFILL_DEPTH. Or run python benchmark.py --ui: the page shows the same list, and Save adds them for you (your old file stays as .env.bak).
  3. Decide whether 64k is the right minimum for your agents. If a model has to run at 32k (a large, smart model that only fits that way), set AGENT_CTX_MIN=32768 and add a 32768:… point to AGENT_CTX_CURVE; with long-context tests in the run, it must still pass every length it can hold.
  4. Sampling: your .env keeps whatever EVAL_TEMPERATURE it has. To move to the new default, set EVAL_TEMPERATURE=server once each model's llama-swap command carries its recommended sampling; expect one full re-ask of the evals.
  5. --reload of an older run keeps the scores it stored. --compare of two older runs rescores both with the v2.3 rules (without prefill at depth, which they didn't measure).

⚙️ Configuration

  • New in PART 1: AGENT_CTX_MIN, AGENT_CTX_PASS_PCT, AGENT_CTX_CURVE; ESTIMATE_GPU_BW_GBS moved here from PART 3.
  • Changed defaults: EVAL_PROFILE funnel, EVAL_TEMPERATURE and TEMPERATURE server, ESTIMATE_GPU_BW_GBS auto (see above).
  • New in PART 2: PREFILL_DEPTH.
  • New in PART 3:
    • AGENT_TOOL_WEIGHT, MIN_TOOLCALL_PCT (Agent score);
    • COLOUR_STYLE, COLOUR_MODE, COLOUR_<METRIC> (GEN, PREFILL, ACC, SCORE, TTFT_MS, AGENT_TTFT, DEPTH_TTFT, SQ, RETRY, POWER), DASH_FAMILIES (dashboard view);
    • UI_PORT, UI_HOST, UI_TOKEN (settings page).
  • .env.example lists 153 settings plus three families (141 before). --check-env and the settings page know the allowed values of every setting that has a fixed set.
  • New command-line options: --ui, --env FILE, --validate.
  • No third-party code or data was added. The settings page uses only Python's standard library; the dashboard still loads Chart.js from cdnjs.

🛟 llama-swap-llm-benchmark v2.3.1: runs that survive trouble, --resume, honest timings, a harder settings page

A long run used to have three ways to lose or distort its results when something went wrong: an unexpected error threw away the JSON and CSV, a llama-swap restart cost the models tested in that window their ranking, and one failed dataset quietly changed which latency every model was scored on. This release fixes all three, adds --resume, and tightens the settings page and the code sandbox.

✨ Key New Features

🛟 Nothing measured is lost, and --resume

  • Rows are saved after every model. results/run_<stamp>.rows.json is rewritten after each model of the speed phase and each model of every eval stage. A crash, kill -9 or a power cut loses at most the model being tested.
  • python benchmark.py --resume continues the newest run in OUTPUT_DIR (or --resume 20261110_103848):
    • models whose speed test finished are kept and not benchmarked again;
    • models that failed, or were never reached, are benchmarked;
    • the result cache answers every eval question already asked, so only the missing ones are sent;
    • a long-context result that is complete is kept; a missing one is run.
  • It also repairs a finished run. A finished run keeps its rows file, so --resume re-tests only what went wrong in it: a model that failed to load, lost its long-context test, or has incomplete evals.
  • The resumed run gets its own stamp and files; artifacts.resumed_from in the JSON names the run it continued. Runs made before v2.3.1 have no rows file and cannot be resumed (their eval answers are still in the result cache).
  • An unexpected internal error no longer discards the run. In the speed phase the model is marked error and the next one runs. In the eval phase everything measured so far is written (status interrupted).
  • SIGTERM is handled like Ctrl+C (Unraid "abort script", docker stop, systemd): partial results are written.
  • Exit codes: 0 complete, 2 stopped early by an error with partial results written, 130 stopped by Ctrl+C / Stop / SIGTERM, 1 could not run.

🔌 llama-swap goes away mid-run

  • SERVER_WAIT_SECONDS (default 600; 0 = off). When requests fail because the server is not there (refused, dropped, 5xx), the run waits for it to list its models again, reloads the model, and:
    • asks the failed questions, and the ones never asked because the dataset stopped, once more;
    • repeats a long-context test that failed on a connection error;
    • benchmarks a model again whose speed test failed that way.
  • Timeouts are the model's own behaviour and are not asked again. Re-asked questions are counted in eval_reasked_n and marked "reasked": true in eval_details_*.jsonl.

⏱️ Timings measured over the right window

  • A retry is no longer inside the measurement. Latency, energy, J/token and GPU watts of a speed request are taken over the attempt that answered, not over a failed attempt plus its backoff. speed_retries counts them.
  • Durations use a clock that cannot step. Every interval and telemetry window uses a monotonic clock; an NTP correction during a long run no longer shifts them.
  • "TTFT ms" is now "Prompt ms". The column was always llama-server's prompt-processing time, not what a client waits. The evals now record what the client sees through llama-swap, per dataset in the JSON: ttft (median seconds to the first token), ttfa (to the first visible answer token, after the thinking) and ttfa95. Display only: no score uses them.
  • Spread between runs. tps_samples keeps every speed run and TPS spread % shows how far apart they were.
  • How tokens/s was obtained (tps_basis): server from llama-server's timings, wall when a backend reports none (that figure includes prompt processing).
  • Change since the last run. TPS vs last % compares with the previous run of the same model file at the same context and offload; changes of 5% or more are printed with the llama.cpp builds (build_info).
  • A deferred prefill figure (taken from the long-context test) is recorded only when the prompt was cold; very short requests no longer read a telemetry sample from before they started.

📟 Following a run

  • results/run_status.json: phase, model i of N, dataset, questions done, started / updated, host, pid. The settings page shows it above the log; a script or a Home Assistant sensor can read it.
  • NOTIFY_URL: a short JSON summary is POSTed there when a run ends (status, duration, models, failed models, top three). No settings, tokens or answers.
  • New flags for one-off runs: --models, --exclude, --profile, --output-dir (the same as setting INCLUDE_MODELS, EXCLUDE_MODELS, EVAL_PROFILE, OUTPUT_DIR for that run).
  • A second speed run is refused while another run is going (--force starts anyway). The lock now carries the host name and a heartbeat, so a run started on another machine over a share is seen as live. Evals-only runs are still only warned.
  • Model names such as …-E4B-… (effective parameters) are read as 4B when the name has no plain size.

📊 Dashboard

  • Chart.js 4.4.1 is written into every dashboard. It opens without internet access and no longer loads a script from a CDN. Each file grows by about 200 KB.

🔧 Fixes

  • One dataset with no real answers switched the whole run from the single-stream s/q estimate to measured latency (see below).
  • Answers cached at one concurrency and reused in a run at another: the s/q basis now follows the concurrency the answers were measured at.
  • The Home Assistant power poller kept every sample for the whole run and scanned them all per request.
  • NVML was shut down while its polling thread could still be reading.
  • A run that failed at once was reported as "The run ended". A Full run started from the settings page under a Python without the datasets package died in its first second and the page said nothing. The page now shows a red "The run failed" with the last lines of the log (amber for a run stopped early, green for a finished one), and --check prints Hugging Face datasets: NOT AVAILABLE with the install command when an eval needs it. If you use a virtual environment, start --ui from it: the page runs the benchmark with the Python that started the page.
  • KEY= # comment is read as empty. python-dotenv reads an empty value followed by a comment as the text of the comment (a UI_TOKEN= # ... line became the token). The benchmark and the settings page now read it as empty. --check-env still flags the line; write KEY="" or put the comment on its own line.

🔒 Security

  • Settings page on the LAN: a UI_TOKEN shorter than 16 characters is refused at start (delete the line and a random one is generated). Ten wrong tokens from one address lock that address out for five minutes.
  • Dashboards served by the settings page are sandboxed (Content-Security-Policy: sandbox): their scripts cannot reach the page's cookie or API.
  • .env, .env.bak and .env.tmp are created readable by their owner only (0600), whatever the umask.
  • The page only reads a run log inside OUTPUT_DIR (benchmark_*.log, batch_probe_*.log); the path in the lock file is no longer trusted.
  • Subprocess sandbox as root: model-written code is started as the unprivileged user nobody. If that is not possible a warning says the code runs as root.
  • The run prints the sandbox image's ID and reminds you that a tag can change; pin one with EVAL_CODE_DOCKER_IMAGE=<image>@sha256:<digest>.
  • run_<stamp>.rows.json never stores the value of --api-key.

🔄 Updating

  1. Copy the files over your install. Your .env, results/ and cache are untouched.
  2. python benchmark.py --check-env lists the two new settings.
  3. If your settings page is open to the LAN with a short UI_TOKEN, it will not start until the token has 16 characters or the line is deleted.
  4. A cron or User Script that tests the exit code: Ctrl+C / Stop now ends with 130, an early stop with 2.

⚙️ Configuration

Setting Default What it does
SERVER_WAIT_SECONDS 600 How long to wait for llama-swap to answer again mid-run before giving up on the failed requests; 0 = off
NOTIFY_URL empty POST a JSON summary there when a run ends