Repository navigation
v2.3.2 - Pause Function and adjusted thinking cap
Pause/resume, thermal guard, auto thinking budget (work on v2.3.2, 2026-11-12)
Pause / resume
- Same-process pause controlled by
run_control.jsonin OUTPUT_DIR (polled ~1 s). Works on Windows and for a run on another machine sharing OUTPUT_DIR. CLI:--pause,--resume(no arg + live run); settings page: Pause / Resume buttons;/api/pause,/api/resume. - Temporary state:
run_<stamp>.pause.json+ the existingrun_<stamp>.rows.json. Both deleted when the run ends or resumes.--resume <run>still recovers a process lost while paused. - Manual pause: current model's speed phase finishes; evals stop taking new questions, in-flight ones finish and are graded. The batch probe cannot be paused.
- Thermal:
MAX_GPU_TEMP(0 = off, else 40-110),GPU_TEMP_RESUME_DELTA(default 10),GPU_TEMP_MAX_RESTARTS(default 3). Own GPU monitor thread at 500 ms (never inside token generation); trip is latched; resume after 3 consecutive cool samples. Not enforced, with a loud warning, when the server is remote or no temperature can be read. - Fresh restart:
PauseRestart(BaseException)raised at checkpoints only insideCONTROL.restartable(). Speed test restarts wholesale; long-context restores its row snapshot; an interrupted evalDatasetRunis discarded and re-run (answered questions come from the result cache). After GPU_TEMP_MAX_RESTARTS restarts of one line,PauseGaveUpends the run like Ctrl+C (exit 130, resumable). - Time paused is recorded in
phases.paused_s(it is also inside speed_s / eval_s).
Thinking budget
EVAL_THINK_BUDGET=autois now the default:EVAL_THINK_BUDGET_PCT(25) % of the model's own context, raised toEVAL_THINK_BUDGET_MIN(4096) and held toEVAL_THINK_BUDGET_MAX_PCT(60) % of the context (ceiling wins on a tiny context), rounded down to a multiple of 1024. Unknown context: 16384. A number or 0 still fixes it. Validation: PCT 5-90, MIN >= 256, MAX_PCT 10-90, PCT <= MAX_PCT.- Logged per model:
thinking budget 32,768 tokens (25% of 131,072 ctx)(or "floor ..." / "ceiling ..."). Exported in results.json / dashboard meta (think_budget,think_budget_pct,_min,_max_pct);--comparerefuses runs that differ. - Settings page: hover tip on the four keys lists the resulting budget for 4k..1M contexts from the current field values (JS twin of
auto_think_budget, tested under Node). - Consequences: the budget is in the result-cache key, so models whose budget changed are re-asked (
EVAL_THINK_BUDGET=16384keeps old cache and comparability). A bigger budget fits fewer requests in the shared KV pool: in-flight drops (4 -> 3 at 128k in the mock). Scores of previously capped models may rise; not measured. - Evidence (user's 2026-11-12 server log): many models hit the 16,384 cap on ~5-15% of questions; most recovered by the No-Thinking retry (time cost), the LFM2.5 builds show "0 recovered, N counted as wrong".
Rebrand (display strings only)
"Local LLM Benchmark Suite" in the dashboard, settings page, markdown report, argparse description, .env.example header, __init__. Left for the owner to decide: GitHub repo / Project name, REPO_URL and clone URLs, tool.name in the JSON and the notify body, the markdown front-matter tag, historical release notes, regenerated demo dashboards.
Open items
- Not verified: real GPU / NVML thermal behaviour, a real llama-server, Docker, Hugging Face.
- Suggested commits:
feat: pause/resume and GPU thermal guard,feat: auto thinking budget with floor and ceiling,fix: rebrand display strings, anddocs:(README) separately. - No new third-party dataset, method or library: no credits/licence change.