Repository navigation
v2.3.5: settings named for their role, a 90k long-context length, safer defaults
🪟 llama-swap-llm-benchmark v2.3.5: settings named for their role, a 90k long-context length, safer defaults
A tidy-up release before a fresh baseline run: the endpoint settings no longer carry product names, long-context gets the 90k step the 27B models are loaded at, and the dashboard shows VRAM pressure the way you read it. Nothing about how a model is measured or graded changes, so existing Agent scores are identical (rescored: 26 server + 7 main-PC models from the 2026-11-12 runs).
✨ Key New Features
🏷️ Settings renamed (old names keep working)
LLAMA_SWAP_URL/LLAMA_SWAP_API_KEYare nowSERVER_URL/SERVER_API_KEY(any OpenAI-compatible server works, llama-swap is one of them).STRATA_URL(and the olderSTRATA_SWAP_URL),STRATA_API_KEY,STRATA_MODEL,STRATA_PARAMS_B,STRATA_ACTIVE_B,STRATA_QUANT,STRATA_CTXare nowSECONDARY_*.- An old name in
.env, a favourite or the shell still works and prints one line saying what it is called now;--check-envlists it; the settings page renames it when you save. If both names are set, the new one wins. - Not renamed on purpose: names inside result files (
strata_api_key,strata_details, columns) so older results reload and compare;--probe-strata; the words "llama-swap" and "Strata" in text.
📏 Long-context at 90k
LONGCTX_LENGTHSdefaults to 65536,92160,131072 (newlongctx_90k_pctcolumn; the history CSV is rewritten under the new header, a.bakis kept). A length larger than the model's loaded context is skipped as before, so a model loaded at-c 90112(88k) is not asked the 92160 length.- The tooltip on every context-size setting now reads: "Suggested context sizes: 4096 (4k) ... 1048576 (1M) | Enter the whole number (bracket value are just for help - do not use them in the fields)".
🛡️ Defaults
MAX_GPU_TEMP=85(was off). The run pauses at 85 C and carries on 10 C lower. With the server on another machine, or no readable GPU sensor, the existing "not enforced" warning is printed.MAX_GPU_TEMP=0turns it off.EVAL_SANITY_PCTstays 0 (off). The settings page now has a guide under it: what it does, and how the rate halves per model while answers match (20 -> 10 -> 5 -> 2 -> 0) and resets on a difference.
🎨 Dashboard
- VRAM bar above
VRAM_SWEET_HIGH: red, darkening evenly to black at 100% (no pink step). The hazard sign still starts atVRAM_SWEET_MAX. - Ctx and Max ctx tooltips give the exact size:
90k (92160 tokens).
🔁 How old and new results compare
- Agent scores, usable context and rank states of existing result sets are identical under v2.3.3 and v2.3.5 (checked on the 33 models of the 2026-11-12 runs; they hold no 90k results).
- New runs with a 90k result: a model loaded at 92160 or more that passes 90k but fails 128k now has usable context 92160 (context factor about 0.978) instead of 64k (0.95), a small rise it has earned. A model at 92160 or more with no 90k result counts as "not passed" and drops to 64k, so
--compareof an old run against a new one can penalise it; re-run the old one. The composite stays on 64k while any ranked model is loaded at 64k or 90112. - The result cache is not keyed on the tool version, so v2.3.5 reuses a matching v2.3.x cache. For a clean baseline move
results/cache/aside (or setEVAL_RESULT_CACHE=refreshfor the first run).
⚠️ Worth knowing
- Load the "90k" models at
-c 92160, not 90112, if you want them in the 90k test and the same dashboard label. - The long-context borderline cliff (8 retrievals per length) and the v2.4 MMLU-Pro tie-breaker are unchanged from v2.3.3.
- No new dataset, library or third-party work: the README credits section is unchanged.
Fixes / Security
- None in this release (no security-relevant change).