Skip to content

v2.3.6: tool calling grows with the profile, finalists-only accuracy charts

Choose a tag to compare

@Maximo101 Maximo101 released this 15 Nov 10:20
· 6 commits to main since this release
51a5819

🧰 Local-LLM-Benchmark v2.3.6: tool calling grows with the profile, finalists-only accuracy charts

Two changes from the first full v2.3.5 baseline on the server (2026-11-15, 22 models, funnel to moderate, resumed twice after UPS shutdowns), plus the runner fix from the v2.3.5 review.

✨ Key New Features

🛠️ Tool calling: 96 cases, more as the profile grows

  • The built-in tool-calling set grows from 32 to 96 cases. A stage now asks max(32, its rows) of them: 32 at quick and basic, 43 at moderate, all 96 from standard up. The cases are asked in a fixed order, so every stage is a prefix and the first 32 are always the v2.3 core set.
  • MIN_TOOLCALL_PCT is still decided on those 32 at the first funnel stage (and checked again on the final count when the run ends).
  • Two new categories, after the ideas of BFCL V3 (in a much simpler single-step form):
    • missing: a tool fits but the user left out a required value (no e-mail address, no room, no repo). The right answer is to ask, not to call with a made-up value.
    • chained: the conversation already holds the model's earlier call and the tool's result; the next call must use that result (convert the share price it just got, read the newest of the listed files, set the thermostat only because it is 11 °C), or no call when the result already answers the question.
  • Harder cases in the old categories: close tools side by side (forecast vs current weather, exchange rate vs conversion, turn off vs turn on, search files vs list a directory), three calls in one reply, mixed tools in parallel, dates relative to a given "today", numbers in words ("two and a half thousand", "a quarter of full volume").
  • New tool definitions (forecast, lights off, locks, exchange rate, file search/write, shell command, Docker restart/logs, restaurant booking, GitHub issue, volume, calendar lookup, order status, meetings) are only offered in the new cases.
  • multiple-10 is replaced by multiple-10b. The old case ("How do you say 'thank you' in German?" with a translate tool offered) was answered directly by 10 of 22 models. Answering a well-known fact yourself is exactly what the irrelevance cases reward, so the case measured style, not tool use. The new one asks for the translation tool explicitly.
  • The funnel gate counts tool calling's error while the final stage will still ask a model more tool cases (it used to be complete at the first stage). Bounds get slightly wider: the safe side.

📊 Accuracy charts show the finalists only

  • With the funnel, a model that stopped early answered only that stage's questions (11 per dataset at quick) while the finalists answered 43, so flint's and LFM2.5 Q4_K_XL's 11/11 sat above finalists' 41/43. Such models are now left out of the dataset accuracy / pass-rate / tool-calling charts, the Quality vs speed chart and the quant table's eval average.
  • A line above the charts says how many stopped and where; Show them brings them back faded, with [quick] after the name and the reason in the tooltip.
  • Long-context retrieval is asked in full at the first stage, so its chart keeps every model. The table and the Agent leaderboard are unchanged. Display only: no score changes.

🔁 How old and new results compare

  • Existing results do not change. The 22 models of the 2026-11-15 run rescored under v2.3.6 give the same Agent scores, quality and rank states as stored (--reload keeps stored scores anyway).
  • The gate: replayed on the 2026-11-15 run with its own answers: the same 18 of 22 models pass quick and 18 of 18 pass basic. The one screened-out model (Empero-Qwen3.8-4B) moves from 0.94 to 0.18 points below the bar.
  • New runs: tool-calling percentages are not like-for-like with earlier runs. Finalists of a moderate funnel answer 43 tool cases instead of 32, 11 of them harder. --compare notes the different question counts. How much the new cases move the ranking can only be seen after a run.
  • multiple-10b, simulated on the 2026-11-15 run (counting it as passed for the 10 models that answered multiple-10 directly): every other model keeps its relative order, but Empero-Qwen3.8-9B drops from #2 (89.4) to #11 (85.2) and Nemotron-3.5-Lightning from 85.6 to 81.6. Neither model's own answers change. The best composite rises, and both fall just outside the tie zone. That is the honest result of removing a case that penalised direct answers, and it also shows how compressed the field is: one tool case moves a model nine places.
  • Result cache: 31 of the 32 core cases keep their cache keys. A first v2.3.6 run asks multiple-10b of every model and the 11 new cases of every moderate finalist. Tool cases are short (about 130 tokens), so this is roughly a minute per model.

⚠️ Worth knowing

  • The missing and chained cases follow the system prompt the tool always sends ("call it with exactly the arguments the user gave"). A model that fills a required value with a guess fails them. That is deliberate for agent use.
  • Chained cases send assistant messages with tool_calls and tool messages. llama.cpp with --jinja should render these for any template with tool support (not checked on every model here). If a model scores 0 on every chained case, look at those requests in eval_details_*.jsonl before blaming the model.
  • The defaults are unchanged: EVAL_FUNNEL_FINAL defaults to standard, so a default funnel run asks all 96 tool cases of its finalists.

Fixes

  • evals/runner.py held its second half twice (the pause work merged onto a pre-v2.3.3 runner). Runs used the older gate, so hard-rule models were logged as drop and stored as screened out. Fixed. Gate decisions and scores were not affected. selftests/test_v236_runner.py guards it (from the v2.3.5 review).

Credits

  • The tool-calling categories and AST-style matching follow the Berkeley Function Calling Leaderboard (Patil, Mao, Yan, Ji, Suresh, Stoica, Gonzalez, ICML 2025, PMLR 267:48371-48392; Apache 2.0). The new categories follow BFCL V3's Missing Parameters and multi-turn ideas. No BFCL data is used: all 96 cases were written for this tool.