Repository navigation
v2.3.3: the funnel says what happened to every model
🪟 llama-swap-llm-benchmark v2.3.3: the funnel says what happened to every model
Real runs (2026-11-12, two PCs, 33 models) showed the funnel pruned 1 model of 26, and that one (flint: 100% on the evals, 0% at 128k) was labelled "screened out" although it could never have had an Agent score. v2.3.3 separates rules from comparisons.
✨ Key New Features
🚦 Three explicit states: Ranked, Screened out, No Agent score
- Every evaluated model ends in exactly one state: Ranked (has an Agent score), No Agent score (fails a hard rule) or Screened out (the funnel gate dropped it as clearly below the leaders).
rank_state/rank_reasonare inresults.json(scoresgroup), shown as a badge on the model name, a Rank filter (with counts), the leaderboard note, the legend and the markdown report. If an older result is both, No Agent score wins. - Hard rules are decided outside the funnel: loaded or usable context below
AGENT_CTX_MIN, long-context belowAGENT_CTX_PASS_PCTat every length that counts, tool calling belowMIN_TOOLCALL_PCT. They are plain thresholds on the model's own numbers: no estimate, no margin of error, never compared with other models, and a failing model no longer lifts the leaders' bar. EVAL_HARD_FAIL=stop|continue(new, defaultstop): a model that fails a hard rule at a gate is not asked further stages.continuekeeps asking, so its later-stage scores exist.
🔬 A funnel that prunes
- The gate's bounds use the reducible error (the evals part; tool calling and long-context are asked in full at the first stage and do not shrink) plus
EVAL_FUNNEL_MARGINpoints (new, default 3). Rule:score + z × SE_evals + margin ≥ score_k − z × SE_evals_kfor theEVAL_LEADERS-th model. This replaces theEVAL_FUNNEL_CONF=0.85stopgap. - Decimal gaps: the log shows, per model and gate,
KEEP / SCREENED OUT / NO AGENT SCORE, score ± the error the gate used (and the error of all parts) and the signed gap to the leaders' bar to two decimals.evaluation.funnel.gatesin the JSON carriesstate,se_red,ub,lb,gap; the run header recordsmarginandhard_fail.
🧠 Thinking budget: auto is the baseline
EVAL_THINK_BUDGET=auto was already the code default and is the documented norm. If your .env still has EVAL_THINK_BUDGET=16384, delete the line: the first run then re-asks the questions whose budget changed (it is part of the result-cache key), and --compare refuses to mix runs with different budgets.
🔁 How old and new results compare
- Agent scores of an existing result set do not change (
python benchmark.py --reloadon the 2026-11-12 runs gives the samescore_agentfor every ranked model). Only labels change: on the server run flint and LFM2.5-24B-UD-Q3_K_XL now show No Agent score (long-context 50% / 62% at 64k, below the 75% pass mark) instead of flint's "screened out"; 24 models stay Ranked. - New runs can screen out more models (that is the point), but only models clearly below the leaders, and a model's stage scores are unchanged. No weaker model can rank higher because of this change: a model that is dropped is never ranked, and hard-rule models never enter the bar.
- One run set (v2.3.1) is the only evidence for the margin: nothing within 3 composite points of the final 5th-best was pruned at any gate in a retrospective, and 9 models would have been. Treat it as indicative.
⚠️ Worth knowing
AGENT_CTX_MINdefaults to 64k, so a model that passes 64k but fails 128k keeps an Agent score (lower context factor).AGENT_CTX_MIN=131072makes 128k mandatory but would remove 13 of 26 models of the server run (loaded at 64k-90k). The default is not changed.- The long-context borderline problem (8 retrievals per length, so 75% is a cliff between 6/8 and 5/8) is not addressed here.
- The deeper MMLU-Pro tie-breaker stage is planned for v2.4.
- New settings:
EVAL_FUNNEL_MARGIN,EVAL_HARD_FAIL(in.env.example).
🙏 Credits
No dataset, benchmark method or library was added or changed.