Repository navigation
v2.3.6: tool calling grows with the profile, finalists-only accuracy charts
🧰 Local-LLM-Benchmark v2.3.6: tool calling grows with the profile, finalists-only accuracy charts
Two changes from the first full v2.3.5 baseline on the server (2026-11-15, 22 models, funnel to moderate, resumed twice after UPS shutdowns), plus the runner fix from the v2.3.5 review.
✨ Key New Features
🛠️ Tool calling: 96 cases, more as the profile grows
- The built-in tool-calling set grows from 32 to 96 cases. A stage now asks max(32, its rows) of them: 32 at
quickandbasic, 43 atmoderate, all 96 fromstandardup. The cases are asked in a fixed order, so every stage is a prefix and the first 32 are always the v2.3 core set. MIN_TOOLCALL_PCTis still decided on those 32 at the first funnel stage (and checked again on the final count when the run ends).- Two new categories, after the ideas of BFCL V3 (in a much simpler single-step form):
- missing: a tool fits but the user left out a required value (no e-mail address, no room, no repo). The right answer is to ask, not to call with a made-up value.
- chained: the conversation already holds the model's earlier call and the tool's result; the next call must use that result (convert the share price it just got, read the newest of the listed files, set the thermostat only because it is 11 °C), or no call when the result already answers the question.
- Harder cases in the old categories: close tools side by side (forecast vs current weather, exchange rate vs conversion, turn off vs turn on, search files vs list a directory), three calls in one reply, mixed tools in parallel, dates relative to a given "today", numbers in words ("two and a half thousand", "a quarter of full volume").
- New tool definitions (forecast, lights off, locks, exchange rate, file search/write, shell command, Docker restart/logs, restaurant booking, GitHub issue, volume, calendar lookup, order status, meetings) are only offered in the new cases.
multiple-10is replaced bymultiple-10b. The old case ("How do you say 'thank you' in German?" with a translate tool offered) was answered directly by 10 of 22 models. Answering a well-known fact yourself is exactly what the irrelevance cases reward, so the case measured style, not tool use. The new one asks for the translation tool explicitly.- The funnel gate counts tool calling's error while the final stage will still ask a model more tool cases (it used to be complete at the first stage). Bounds get slightly wider: the safe side.
📊 Accuracy charts show the finalists only
- With the funnel, a model that stopped early answered only that stage's questions (11 per dataset at
quick) while the finalists answered 43, so flint's and LFM2.5 Q4_K_XL's 11/11 sat above finalists' 41/43. Such models are now left out of the dataset accuracy / pass-rate / tool-calling charts, the Quality vs speed chart and the quant table's eval average. - A line above the charts says how many stopped and where; Show them brings them back faded, with
[quick]after the name and the reason in the tooltip. - Long-context retrieval is asked in full at the first stage, so its chart keeps every model. The table and the Agent leaderboard are unchanged. Display only: no score changes.
🔁 How old and new results compare
- Existing results do not change. The 22 models of the 2026-11-15 run rescored under v2.3.6 give the same Agent scores, quality and rank states as stored (
--reloadkeeps stored scores anyway). - The gate: replayed on the 2026-11-15 run with its own answers: the same 18 of 22 models pass
quickand 18 of 18 passbasic. The one screened-out model (Empero-Qwen3.8-4B) moves from 0.94 to 0.18 points below the bar. - New runs: tool-calling percentages are not like-for-like with earlier runs. Finalists of a
moderatefunnel answer 43 tool cases instead of 32, 11 of them harder.--comparenotes the different question counts. How much the new cases move the ranking can only be seen after a run. multiple-10b, simulated on the 2026-11-15 run (counting it as passed for the 10 models that answeredmultiple-10directly): every other model keeps its relative order, but Empero-Qwen3.8-9B drops from #2 (89.4) to #11 (85.2) and Nemotron-3.5-Lightning from 85.6 to 81.6. Neither model's own answers change. The best composite rises, and both fall just outside the tie zone. That is the honest result of removing a case that penalised direct answers, and it also shows how compressed the field is: one tool case moves a model nine places.- Result cache: 31 of the 32 core cases keep their cache keys. A first v2.3.6 run asks
multiple-10bof every model and the 11 new cases of everymoderatefinalist. Tool cases are short (about 130 tokens), so this is roughly a minute per model.
⚠️ Worth knowing
- The missing and chained cases follow the system prompt the tool always sends ("call it with exactly the arguments the user gave"). A model that fills a required value with a guess fails them. That is deliberate for agent use.
- Chained cases send
assistantmessages withtool_callsandtoolmessages. llama.cpp with--jinjashould render these for any template with tool support (not checked on every model here). If a model scores 0 on every chained case, look at those requests ineval_details_*.jsonlbefore blaming the model. - The defaults are unchanged:
EVAL_FUNNEL_FINALdefaults tostandard, so a default funnel run asks all 96 tool cases of its finalists.
Fixes
evals/runner.pyheld its second half twice (the pause work merged onto a pre-v2.3.3 runner). Runs used the older gate, so hard-rule models were logged asdropand stored as screened out. Fixed. Gate decisions and scores were not affected.selftests/test_v236_runner.pyguards it (from the v2.3.5 review).
Credits
- The tool-calling categories and AST-style matching follow the Berkeley Function Calling Leaderboard (Patil, Mao, Yan, Ji, Suresh, Stoica, Gonzalez, ICML 2025, PMLR 267:48371-48392; Apache 2.0). The new categories follow BFCL V3's Missing Parameters and multi-turn ideas. No BFCL data is used: all 96 cases were written for this tool.