Skip to content

v2.4.1: `--diff-runs`, what changed between runs

Latest

Choose a tag to compare

@Maximo101 Maximo101 released this 20 Nov 06:36
aea1cc9

🧰 Local-LLM-Benchmark v2.4.1: --diff-runs, what changed between runs

A report that follows the same models across finished runs and says which changes are real and which are noise. Nothing is benchmarked, and no score changes.

✨ What changes

🔍 --diff-runs

  • python benchmark.py --diff-runs compares the two newest finished runs; --diff-runs 4 the four newest; --diff-runs A B [C ...] these runs (a stamp, a benchmark_<stamp>.json / rescored_<stamp>.json, or a folder with results.json). The first run is the baseline.
  • Per model: rank and Agent score in every run (with a trend line from three runs), the change in Agent quality, each dataset's accuracy, long context per length, generation, prefill, prefill at 64k and at depth, agent TTFT, seconds per question, load time, VRAM peak, GPU power and energy per token. Launch changes are listed (context, quant, KV type, --n-cpu-moe, ngl, batch sizes, flash attention, llama.cpp build, sampling), and so are the run settings that differ.
  • Noise first. Every change from the baseline to the latest run is marked ↑ better, ↓ worse or ≈ within noise:
    • accuracy is paired when both runs' eval_details_<stamp>.jsonl are found (exact McNemar test on the questions that flipped; the result cache and seed give the same questions), otherwise an unpaired 95% Agresti-Caffo interval. A details file recorded under the server's path is also looked for next to the JSON, so a report built on the main PC over SMB still pairs;
    • tokens/s and seconds: at least 5%, or twice the larger run-to-run spread of the timed samples; power and energy 5%; VRAM 0.1 GB; Agent score and quality 0.5 points;
    • rank agreement between consecutive runs as Kendall's τ-b.
  • Honest about comparability. Runs that asked or graded the questions differently (the checks --compare refuses on) get a "Not like-for-like" banner instead of a refusal. Runs scored by different versions are noted, and --diff-rescore scores every run with this version first (in memory; no file is written but the report). The report says that the Agent score and quality are relative to each run's best finalist.
  • Output: results/run_diff_<stamp>.html (self-contained, no CDN, a Content-Security-Policy that blocks any network access; sortable tables, a model filter, an "only changes beyond noise" switch; light and dark), results/run_diff_<stamp>.json with every number, and a terminal summary with the biggest movers.
  • The report reads verdicts and aggregates only. No question, reply or reference answer is read into it, so it is safe with GPQA enabled; it still lives in the git-ignored results/ folder.
  • --forget-run leaves diff reports alone (they belong to no single run).

🧪 Tests

  • selftests/run_factory.py writes synthetic finished runs with the real JSON writer, plus matching per-question details, with planted changes.
  • selftests/test_v241_rundiff.py (21 tests) covers:
    • the three statistical tests against known values;
    • that the planted changes are found and the 2-question wobble is not;
    • the unpaired fallback;
    • a run diffed against itself reports nothing;
    • three runs;
    • settings and version mismatches, --diff-rescore, and that no run file is touched;
    • run selection;
    • HTML escaping of model names, no question text in the report, the page script parsing under Node, and the table shape;
    • the command line;
    • an end-to-end diff of two evals-only runs against the mock llama-swap server.
  • 360 selftests pass, 1 skipped.

How old and new results compare

No scoring, grading, prompt or cache-key change: every run scores exactly as with v2.4.0. GRADER_VERSION is unchanged and the result cache stays valid.

Credits

New methods credited in the README: McNemar (1947), Agresti and Caffo (2000), Kendall (1938, 1945). No dataset, library or third-party code was added.