Repository navigation
v2.4.0: a fair Agent score across runs, `--rescore`, steadier long context
π§° Local-LLM-Benchmark v2.4.0: a fair Agent score across runs, --rescore, steadier long context
β¨ What changes
βοΈ Long context: fixed lengths per model (AGENT_LONGCTX_BASIS)
- The problem. The long-context part of the Agent score used only the lengths every ranked model ran.
- Now: each model is judged on the lengths in
AGENT_LONGCTX_BASIS(default64k,90k) that its loaded context fits. A 64k model is judged on 64k; a 90k or 128k model on 64k and 90k. The context factor (AGENT_CTX_CURVE) still carries the preference for longer context, and the usable-context rule is unchanged. - The tie-zone bar is still the best finalist, but only models judged on every basis length can set it. Otherwise a 64k model with a perfect, easier 64k test could raise the bar for everyone.
- 128k is not in the default basis. A 128k model that does badly at 128k already loses through its usable context (it counts as a 90k model). Set
AGENT_LONGCTX_BASIS=allto count every tested length, orcommonfor the v2.3 rule.
π― Long context: more questions near the pass mark (LONGCTX_BORDERLINE_TRIALS)
- With 8 codes per length, 75% is the step between 6/8 and 5/8: one code decides whether a model's usable context is 64k or 90k.
- Now a result within one code of
AGENT_CTX_PASS_PCT(8 codes at 75%: 5/8 or 6/8) is askedLONGCTX_BORDERLINE_TRIALSmore questions (default 2, i.e. 8 more codes) before it stands. The extra questions reuse the cached document, so they cost generation time only. - The codes for the extra questions are always in the document, so it is the same document whether or not they are asked. The log notes when it happens (
64k: 5/8 is within one code of 75%, asked 8 more). The per-length count is stored aslongctx_<length>_nand used for the error.
π --rescore
python benchmark.py --rescore [JSON]scores a finished run again with this version's rules. It keeps every measurement and answer and prints old vs new rank, Agent score and quality per model.- It writes
rescored_<run>.jsonand its dashboard. The run's own files,results.jsonand the latest dashboard are left alone. - A scoring setting the run did not record (
AGENT_LONGCTX_BASISfor runs before v2.4) comes from today's.envand is named in the output.
π§Ή Housekeeping
--forget-run STAMPremoves one run's files fromOUTPUT_DIRand its rows in the history CSV (.bakkept). Ifresults.jsonwas that run, the newest other run takes its place. Without--forceit only lists what would go. The result cache is kept.--comparenow also refuses runs graded differently: strict tool calls, code-test timeout, LiveCodeBench tests per problem and date, extra request fields, long-context thinking. Runs too old to record a setting are not refused for it. Different long-context trials, keys or borderline trials are noted.--estimate:- The funnel's pass share was a fixed 50%. With no earlier funnel run it assumes 85%.
- A
--resumerun no longer counts the speed test (and the long-context test) of models the rows file already holds.
- "GLM shown as Dense" is struck from the plan: both November runs label it MoE (read from the GGUF header).
π How old and new results compare
-
Stored results do not change.
--reloadkeeps stored scores;--rescoreshows the new ones.Model Rank old β new Agent score old β new Why Groot 1 β 1 93.80 β 94.47 its 90k counts Qwen3.6-MTP 10 β 9 86.46 β 87.08 its 90k counts (100%) Qwen3.6 Q4_K_M 9 β 10 86.83 β 86.83 unchanged Ornith-9B / Qwen3.8-27B 12 / 11 β 11 / 12 83.22 β 83.84 / 83.73 flint 13 β 14 82.55 β 76.63 its 90k is 87.5% Wicked-Stuff 15 β 15 77.32 β 71.47 its 90k is 75% Empero-Qwen3.8-9B 18 β 16 64.45 β 71.16 its 90k is 100% The other 10 same rank +0 to +0.6 Every change comes from the model's own 90k answers, now counted. With this, the 16 Nov and 19 Nov rankings agree on the order of the models in both runs.
-
New runs:
- Long-context results near the pass mark get more codes, so a borderline model's usable context is decided on 16 codes instead of 8.
- The long-context document holds the extra codes, so v2.4 long-context results are not like-for-like with v2.3.x runs. They are never cached anyway.
β οΈ Worth knowing
-
New settings:
AGENT_LONGCTX_BASIS(default64k,90k;common= v2.3;all)LONGCTX_BORDERLINE_TRIALS(default 2; 0 = off)
Both are in
.env.examplePART 3. -
--compareof a v2.4 run with an older run uses the v2.4 rule for both: each model's own lengths.
Credits
- No dataset, benchmark method, library or other third-party work is added or changed. The README credits section is unchanged.