Skip to content

v2.4.0: a fair Agent score across runs, `--rescore`, steadier long context

Choose a tag to compare

@Maximo101 Maximo101 released this 20 Nov 05:20
· 1 commit to main since this release
b6a5e13

🧰 Local-LLM-Benchmark v2.4.0: a fair Agent score across runs, --rescore, steadier long context

✨ What changes

βš–οΈ Long context: fixed lengths per model (AGENT_LONGCTX_BASIS)

  • The problem. The long-context part of the Agent score used only the lengths every ranked model ran.
  • Now: each model is judged on the lengths in AGENT_LONGCTX_BASIS (default 64k,90k) that its loaded context fits. A 64k model is judged on 64k; a 90k or 128k model on 64k and 90k. The context factor (AGENT_CTX_CURVE) still carries the preference for longer context, and the usable-context rule is unchanged.
  • The tie-zone bar is still the best finalist, but only models judged on every basis length can set it. Otherwise a 64k model with a perfect, easier 64k test could raise the bar for everyone.
  • 128k is not in the default basis. A 128k model that does badly at 128k already loses through its usable context (it counts as a 90k model). Set AGENT_LONGCTX_BASIS=all to count every tested length, or common for the v2.3 rule.

🎯 Long context: more questions near the pass mark (LONGCTX_BORDERLINE_TRIALS)

  • With 8 codes per length, 75% is the step between 6/8 and 5/8: one code decides whether a model's usable context is 64k or 90k.
  • Now a result within one code of AGENT_CTX_PASS_PCT (8 codes at 75%: 5/8 or 6/8) is asked LONGCTX_BORDERLINE_TRIALS more questions (default 2, i.e. 8 more codes) before it stands. The extra questions reuse the cached document, so they cost generation time only.
  • The codes for the extra questions are always in the document, so it is the same document whether or not they are asked. The log notes when it happens (64k: 5/8 is within one code of 75%, asked 8 more). The per-length count is stored as longctx_<length>_n and used for the error.

πŸ” --rescore

  • python benchmark.py --rescore [JSON] scores a finished run again with this version's rules. It keeps every measurement and answer and prints old vs new rank, Agent score and quality per model.
  • It writes rescored_<run>.json and its dashboard. The run's own files, results.json and the latest dashboard are left alone.
  • A scoring setting the run did not record (AGENT_LONGCTX_BASIS for runs before v2.4) comes from today's .env and is named in the output.

🧹 Housekeeping

  • --forget-run STAMP removes one run's files from OUTPUT_DIR and its rows in the history CSV (.bak kept). If results.json was that run, the newest other run takes its place. Without --force it only lists what would go. The result cache is kept.
  • --compare now also refuses runs graded differently: strict tool calls, code-test timeout, LiveCodeBench tests per problem and date, extra request fields, long-context thinking. Runs too old to record a setting are not refused for it. Different long-context trials, keys or borderline trials are noted.
  • --estimate:
    • The funnel's pass share was a fixed 50%. With no earlier funnel run it assumes 85%.
    • A --resume run no longer counts the speed test (and the long-context test) of models the rows file already holds.
  • "GLM shown as Dense" is struck from the plan: both November runs label it MoE (read from the GGUF header).

πŸ” How old and new results compare

  • Stored results do not change. --reload keeps stored scores; --rescore shows the new ones.

    Model Rank old β†’ new Agent score old β†’ new Why
    Groot 1 β†’ 1 93.80 β†’ 94.47 its 90k counts
    Qwen3.6-MTP 10 β†’ 9 86.46 β†’ 87.08 its 90k counts (100%)
    Qwen3.6 Q4_K_M 9 β†’ 10 86.83 β†’ 86.83 unchanged
    Ornith-9B / Qwen3.8-27B 12 / 11 β†’ 11 / 12 83.22 β†’ 83.84 / 83.73
    flint 13 β†’ 14 82.55 β†’ 76.63 its 90k is 87.5%
    Wicked-Stuff 15 β†’ 15 77.32 β†’ 71.47 its 90k is 75%
    Empero-Qwen3.8-9B 18 β†’ 16 64.45 β†’ 71.16 its 90k is 100%
    The other 10 same rank +0 to +0.6

    Every change comes from the model's own 90k answers, now counted. With this, the 16 Nov and 19 Nov rankings agree on the order of the models in both runs.

  • New runs:

    • Long-context results near the pass mark get more codes, so a borderline model's usable context is decided on 16 codes instead of 8.
    • The long-context document holds the extra codes, so v2.4 long-context results are not like-for-like with v2.3.x runs. They are never cached anyway.

⚠️ Worth knowing

  • New settings:

    • AGENT_LONGCTX_BASIS (default 64k,90k; common = v2.3; all)
    • LONGCTX_BORDERLINE_TRIALS (default 2; 0 = off)

    Both are in .env.example PART 3.

  • --compare of a v2.4 run with an older run uses the v2.4 rule for both: each model's own lengths.

Credits

  • No dataset, benchmark method, library or other third-party work is added or changed. The README credits section is unchanged.