Skip to content

Release - v2.0.0

Choose a tag to compare

@Maximo101 Maximo101 released this 07 Nov 13:24
· 56 commits to main since this release
3a5cfc1

🚀 Release - v2.0.0

We are excited to release v2.0.0 of the llama-swap-llm-benchmark suite! v1.5 taught the tool to grade model intelligence. v2.0 makes those grades trustworthy and decisive, and measures models the way you actually deploy them: at your real context size, with your real KV cache settings, and against your machine's own VRAM budget.

d1

✨ Key New Features & Capabilities

🧠 Smarter, more reliable evaluations

  • Math evals: GSM8K and MATH-500 join GPQA Diamond, MMLU-Pro, HumanEval, MBPP and LiveCodeBench. Answers are normalised (fractions, \frac, units, thousands separators) and matched numerically.
  • Tool-calling eval (BFCL-style): 32 built-in cases (no download) send native OpenAI tools requests in four categories: simple, multiple, parallel and "should not call". Arguments are graded field by field. Calls written as plain text instead of native tool_calls are caught, which usually means a missing --jinja or a chat template without tool support. Off by default (EVAL_TOOLCALL=true).
  • Long-context retrieval: a multi-key needle test at 64k and 128k tokens, with decoys. The benchmark reads each model's KV cache type and flags (or skips) results below LONGCTX_KV_MIN (default q8_0).
  • Streamed evals with loop abort: a reasoning model stuck in a repetition loop is stopped within seconds (Fail: Degradation) instead of hitting a 10-minute timeout. Thinking is capped per question (Fail: Think cap), and a stopped reply whose visible answer is already complete is still graded.
  • Margin of error shown with every score: the dashboard shows ± points next to the test profile. test and quick profiles carry a "rough guide only, re-test your finalists" banner.
  • Retries tracked and penalised: questions that needed the No-Thinking retry are counted, shown next to the scores, and lower the Agent score. More than 20% is flagged ⚠, because it usually points to a template or thinking setting rather than a weak model.
  • Concurrent evals: EVAL_CONCURRENCY keeps several questions in flight for servers running --parallel N.

🔍 Knows what your server actually loaded

  • Launch flags recorded per model: KV cache type (-ctk/-ctv), flash attention, -ub/-b, slots, mmproj, --jinja and CPU offload, read from the llama-server command line.
  • GGUF header reader: file size, bits per weight, total and active parameters, expert count, and attention type (GQA / MLA / sliding window / hybrid SSM) come from the model file itself, not the filename. A MoE without "A3B" in its name is now correctly reported as MoE.
  • KV cost and "Max ctx @85%": KV cache size per token and the largest context that fits in 85% of VRAM, for every model.

📏 Tested at your deployed context

  • Minimum context rule: models loaded below MIN_CTX are flagged and kept off the podium, or skipped entirely (MIN_CTX_ACTION=refuse).
  • Prefill at real prompt lengths: prefill speed at 8k / 32k / 64k tokens. Offloaded MoEs that look fast at 1.4k often slow down sharply here.
  • Agent time to first token: TTFT for a ~20k-token agent-style prompt, cold and on the next turn with the prefix cached.

🏆 Fairer rankings

  • Rebuilt Agent score (0–100, quality first): accuracy decides, and speed can only separate models of similar quality. Prefill speed and agent TTFT now count most within the speed part. VRAM is no longer part of the Agent score.
  • Per-machine budget profiles: BUDGET_PROFILE=server or main (or your own), with rules for minimum context, VRAM reserve (e.g. for Immich or the desktop), minimum KV type, retries and more. The dashboard shows a pass/fail column per rule, and failing models stay off the podium.
  • VRAM sweet spot: Agent score × VRAM fit finds the most capable model that still leaves sensible headroom. Under-using the GPU is now only a small penalty.

📊 Dashboard & outputs

  • New columns, charts and filters for everything above, plus click-to-highlight: click a model name anywhere and it is highlighted across every chart and table.
  • Structured JSON export: results.json (always the latest run, written atomically) and benchmark_<timestamp>.json, with null for missing values instead of NaN.
  • Markdown report with a new Deployment fit section.

🔒 Safety & quality

  • Hardened code sandbox: Docker runs as a non-root user with no network, a read-only filesystem and resource limits. A self-test runs at start-up and in --check. Programs that exit before their tests run no longer count as passes.
  • NVML telemetry in-process via nvidia-ml-py, with automatic fallback to nvidia-smi.
  • Refactored into the llmbench/ package with a 90-test self-test suite (python -m pytest selftests/, no GPU or server needed).
  • Credits & licences: a new README section credits every dataset's authors and licence, and documents the GPQA authors' request that questions are never published online.

⚠️ Behaviour changes

  • Scores are not comparable with v1.5. The Agent score, VRAM fit and eval average are all calculated differently, and enabling GSM8K, MATH or tool calling changes the eval average.
  • Bug fix: a model whose evals all failed used to receive a score on a different scale and could rank first. It now gets no Agent score and is labelled "no eval score".
  • The history CSV gains new columns. It is migrated automatically, and a .bak copy is kept.
  • Keep EVAL_CONCURRENCY=1 for runs you want to compare with older results.

📦 Updating to v2.0.0

# Pull the latest changes
git pull origin main

# Install new dependencies (adds nvidia-ml-py and aiohttp; both optional at runtime)
pip install -r requirements.txt

# Update your configuration (back up your old .env first)
cp .env .env.v1.5.bak
cp .env.example .env

# Confirm telemetry, launch flags and the GGUF reader on your machine
python benchmark.py --check

🛠️ Configuration

Review the new sections in .env.example:

  • PREFILL_CTX_TESTS, AGENT_PROMPT_TOKENS, MIN_CTX (deployed context)
  • EVAL_TOOLCALL, EVAL_LONGCTX, LONGCTX_* (new evals)
  • EVAL_STREAM, EVAL_THINK_BUDGET (streaming)
  • BUDGET_PROFILE (per-machine rules)
  • GGUF_PATH_MAP (if llama-swap runs in Docker and the benchmark isn't run as root)

Refer to the README.md for complete details.

🧪 Known limitations

  • Reading launch flags and GGUF files from a llama-server inside Docker needs root, or GGUF_PATH_MAP. Run --check to confirm.
  • test and quick profiles are for shortlisting only. Use standard or thorough before picking a model.