Skip to content

v2.3.3: the funnel says what happened to every model

Choose a tag to compare

@Maximo101 Maximo101 released this 12 Nov 23:05
· 22 commits to main since this release
96b153d

🪟 llama-swap-llm-benchmark v2.3.3: the funnel says what happened to every model

Real runs (2026-11-12, two PCs, 33 models) showed the funnel pruned 1 model of 26, and that one (flint: 100% on the evals, 0% at 128k) was labelled "screened out" although it could never have had an Agent score. v2.3.3 separates rules from comparisons.

✨ Key New Features

🚦 Three explicit states: Ranked, Screened out, No Agent score

  • Every evaluated model ends in exactly one state: Ranked (has an Agent score), No Agent score (fails a hard rule) or Screened out (the funnel gate dropped it as clearly below the leaders). rank_state / rank_reason are in results.json (scores group), shown as a badge on the model name, a Rank filter (with counts), the leaderboard note, the legend and the markdown report. If an older result is both, No Agent score wins.
  • Hard rules are decided outside the funnel: loaded or usable context below AGENT_CTX_MIN, long-context below AGENT_CTX_PASS_PCT at every length that counts, tool calling below MIN_TOOLCALL_PCT. They are plain thresholds on the model's own numbers: no estimate, no margin of error, never compared with other models, and a failing model no longer lifts the leaders' bar.
  • EVAL_HARD_FAIL=stop|continue (new, default stop): a model that fails a hard rule at a gate is not asked further stages. continue keeps asking, so its later-stage scores exist.

🔬 A funnel that prunes

  • The gate's bounds use the reducible error (the evals part; tool calling and long-context are asked in full at the first stage and do not shrink) plus EVAL_FUNNEL_MARGIN points (new, default 3). Rule: score + z × SE_evals + margin ≥ score_k − z × SE_evals_k for the EVAL_LEADERS-th model. This replaces the EVAL_FUNNEL_CONF=0.85 stopgap.
  • Decimal gaps: the log shows, per model and gate, KEEP / SCREENED OUT / NO AGENT SCORE, score ± the error the gate used (and the error of all parts) and the signed gap to the leaders' bar to two decimals. evaluation.funnel.gates in the JSON carries state, se_red, ub, lb, gap; the run header records margin and hard_fail.

🧠 Thinking budget: auto is the baseline

EVAL_THINK_BUDGET=auto was already the code default and is the documented norm. If your .env still has EVAL_THINK_BUDGET=16384, delete the line: the first run then re-asks the questions whose budget changed (it is part of the result-cache key), and --compare refuses to mix runs with different budgets.

🔁 How old and new results compare

  • Agent scores of an existing result set do not change (python benchmark.py --reload on the 2026-11-12 runs gives the same score_agent for every ranked model). Only labels change: on the server run flint and LFM2.5-24B-UD-Q3_K_XL now show No Agent score (long-context 50% / 62% at 64k, below the 75% pass mark) instead of flint's "screened out"; 24 models stay Ranked.
  • New runs can screen out more models (that is the point), but only models clearly below the leaders, and a model's stage scores are unchanged. No weaker model can rank higher because of this change: a model that is dropped is never ranked, and hard-rule models never enter the bar.
  • One run set (v2.3.1) is the only evidence for the margin: nothing within 3 composite points of the final 5th-best was pruned at any gate in a retrospective, and 9 models would have been. Treat it as indicative.

⚠️ Worth knowing

  • AGENT_CTX_MIN defaults to 64k, so a model that passes 64k but fails 128k keeps an Agent score (lower context factor). AGENT_CTX_MIN=131072 makes 128k mandatory but would remove 13 of 26 models of the server run (loaded at 64k-90k). The default is not changed.
  • The long-context borderline problem (8 retrievals per length, so 75% is a cliff between 6/8 and 5/8) is not addressed here.
  • The deeper MMLU-Pro tie-breaker stage is planned for v2.4.
  • New settings: EVAL_FUNNEL_MARGIN, EVAL_HARD_FAIL (in .env.example).

🙏 Credits

No dataset, benchmark method or library was added or changed.