You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
v2.3.7: fairer long prompts, a steadier long-context test, cheaper cache changes
This commit was created on GitHub.com and signed with GitHub’s verified signature.
🧰 Local-LLM-Benchmark v2.3.7: fairer long prompts, a steadier long-context test, cheaper cache changes
✨ What changes
📏 Long prompts are sized without the chat template's own tokens
The benchmark builds its long prompts (prefill at 64k/90k/128k, prefill at depth, the 20k agent prompt, the long-context document) from a measured tokens per word. v2.3.6 measured it as prompt tokens ÷ words of one 1,200-word prompt, so the template's fixed tokens were counted as words.
Most templates add 10-20 tokens and the error was about 1%. Cyber-Tiel-Coder's template adds ~180 tokens to every request (a default system prompt), so it read 1.33 tokens per word instead of ~1.18 and every long prompt came out about 12% short: its "128k" long-context test was 111k tokens (others ~125k), its 90k test 80k, its agent prompt 18.1k instead of 20k.
Now the figure is the slope between the short tech prompt and the prefill prompt, so the template's tokens cancel out. The log says so when a template adds 64 tokens or more ([info] the chat template adds ~180 tokens to every prompt), and the JSON stores it as prompt_overhead_tokens.
🧷 The long-context document stays the same between runs
The seeded document's length, and so where its codes sit, followed each run's measured tokens per word (1.172 one run, 1.177 the next), so most models got a slightly different document every run. The figure is now rounded up to a 0.02 step before the document is built: the same document while the measurement wobbles, and never longer than before (at most ~1.7% shorter).
It does not remove llama.cpp's own run-to-run differences: on 2026-11-19 Bolt-E4B dropped one of 8 codes at 64k with an identical document. For a steadier long-context score now, set LONGCTX_TRIALS=4 (16 codes per length instead of 8). The extra questions reuse the cached document, so they cost generation time only.
♻️ Cached answers survive a change of thinking budget
The thinking budget and max_tokens are part of each question's cache key, so changing EVAL_THINK_BUDGET (or the context, which moves the auto budget) meant asking everything again.
Now an answer that came out of one attempt that ended on its own (no thinking cap, timeout, loop or length cut) is reused under another budget when it used no more thinking tokens and no more tokens than the new limits. Neither limit was reached, so it could not have changed. Capped, retried or cut answers are asked again.
The result line shows it: [96 from cache, 12 asked under another thinking budget].
Only answers stored by v2.3.7 or later carry the extra key (akey in the cache file); older answers keep working as before.
Why it matters: on the 2026-11-16 fresh run, 111 of 7,386 questions (1.5%) hit the 32k thinking cap or the 600 s timeout. They took 19% of the question time and 25 of them ended correct. Only 1.2% of correct answers thought for more than 16k tokens. A lower budget is now a cheap experiment instead of a full re-run.
⚠️ A llama.cpp update is announced before it empties the cache
The llama.cpp build stays part of every model's fingerprint (a new build can change answers), so an update means every question is asked again.
--estimate reads the current build from a model llama-swap already has loaded (nothing is loaded for it), compares it with each model's last results and leaves those models' cached answers out of the estimate: llama.cpp build changed (b12316 -> b12407): the result cache will NOT be reused for 22 of 22 model(s). With no model loaded it cannot tell and says nothing.
The run checks again after the speed test, when every model's build is known, and prints a [warn] line before the evals start.
🔧 Small things
Tool calling: a model that calls a tool although a required value is missing (the missing cases) is now told called a tool although a required value was not given (it should ask the user); it used to get the irrelevance wording "needs none". Grading is unchanged. Answers already in the cache keep their old text until asked again.
The "in flight: 3 (capped: KV pool ... -kvu unknown)" line was printed before every dataset of every stage (16 times per model). It is now printed once per model, as In flight: 3 per dataset (...), and again only if the number changes.
🔁 How old and new results compare
Scoring code is unchanged. Rescoring the 2026-11-19 run gives the stored scores exactly. Stored results, --reload and --compare are unaffected.
New runs measure slightly different prompts:
Models with a plain template: long prompts about 1% longer, closer to the target lengths (most landed 1-3% short).
The long-context document changes once for every model (new rounding), so long-context results of the first v2.3.7 run are not like-for-like with v2.3.6 runs. They are never cached anyway.
Result cache: nothing is invalidated. Question keys are the same as v2.3.6; new answers just carry one extra key.
Credits
No dataset, benchmark method, library or other third-party work is added or changed. The README credits section is unchanged.