v0.8.0: scheduler tests

Scheduler tests (#28): a new mode, --scheduler all (or a comma list of scenarios), exercises the serving engine's scheduler from the outside and gives every scenario a verdict: PASS, FAIL, STALLED, WEDGED or SKIPPED. The exit code is 2 when a scenario fails.

Scenarios:

• arrivals: decode streams while long prompts arrive periodically. Reports decode tok/s during and outside the prefills, token gaps and arrival TTFT.
• burst: more long prompts at once than running slots / prefill lanes. Every request must finish.
• agents: token-ID agent conversations that outgrow the KV pool, reported per minute (running, waiting, KV, preemptions, prefix hits).
• short-under-long: short-request TTFT alone and while long prompts prefill.
• overload: a capacity probe, open-loop arrivals at 2x capacity, then a drain and recovery.
• abort-storm: cancels during prefill and decode. The server must return to idle and keep serving.
• soak-mix: long mixed traffic with prompts of 1k-160k tokens; images are optional.

During every scenario /metrics is sampled (vLLM and SGLang). A watchdog flags stalls: requests in flight but nothing moving. Scenarios size themselves from the server's context length, KV pool and running-request limit. --sched-ab-fairness off,0.4 A/Bs vLLM's prefill compute share live on one server through /prefill_fairness (VLLM_SERVER_DEV_MODE=1), with the arms interleaved over rounds. See the README section "Scheduler Tests".

Tested against a simulated engine (17 new tests). Validation on GLM-5.3-Flash TP2 is still pending.

v0.7.9: mixed prefill + decode TTFT

Mixed prefill + decode (#27): a new phase, --mixed-prefill-contexts 32k,128k (--mixed-only runs only this phase), measures what a long prompt costs when it arrives while other requests decode.

For each context it records:

• the TTFT of a unique prompt on an idle server;
• the TTFT of another unique prompt that arrives after --mixed-decode-streams (8) background streams are decoding;
• the background decode tok/s before, during and after that prefill, counted from continuous usage stats, so MTP chunks count fully.

Contexts that exceed the model length, the KV budget or the server's running-request limit are skipped with the reason. Results go to a new mixed_prefill section of --output and to a Phase 4 table.

Verified live on GLM-5.3-Flash TP2/DCP2: 64K prompt, TTFT 4.74 s alone and 9.09 s under 6 decoding streams (1.92x).

v0.7.8: lil-bench 1.4 DGX Spark (GB10) and servers across several machines

lil-bench 1.4 (#26): GB10/integrated GPUs record unified memory (host MemTotal), integrated/unified_memory flags and no PCIe fields (1.3 crashed in the inventory on GB10); a single GPU skips p2pmark with the single-GPU reason; servers whose tensor-parallel group spans several machines record server.nodes (nnodes, node rank, headless) and lil-bench refuses to run on a headless worker rank. Verified on a 4x DGX Spark test cluster (GLM-5.3-Flash TP2 across two boxes).

v0.7.7: tool-eval-bench with the leaderboard settings

tool-eval-bench: tool-calling quality with the leaderboard settings

• New mode --tool-eval runs
  tool-eval-bench (MIT)
  against the server the bench targets, with the community leaderboard
  settings: --hardmode --seed 42 --temperature 0 --parallel 4 --max-turns 30 --timeout 600, backend label vllm. --host/--port become --base-url http://<host>:<port>/, --model (or the first served model) its --model,
  and --api-key travels in its environment. --tool-eval-seed,
  --tool-eval-temperature, --tool-eval-parallel, --tool-eval-max-turns,
  --tool-eval-timeout and --[no-]tool-eval-hardmode change single settings;
  --tool-eval-args appends other tool-eval-bench options, such as
  --scenarios for a short subset. A run with changed settings is reported as
  custom, not as leaderboard settings.
• tool-eval-bench stays an external program. The first run installs the pinned
  commit 570951a7 (version 2.7.1.dev14+g570951a77, 69 + 23 Hard Mode
  scenarios) from GitHub into its own venv under
  ~/.cache/llm_decode_bench/tool-eval-bench/<commit>/ (or
  LLM_BENCH_CACHE_DIR); the environment that runs the bench, in the serving
  images the server's venv, is never modified. --tool-eval-ref /
  LLM_BENCH_TOOL_EVAL_REF select another commit, tag or branch,
  LLM_BENCH_TOOL_EVAL_REPO a mirror, and --tool-eval-bin an existing
  install. Without network access the run stops with an error that says so.
• The report shows one line per finished scenario, then the score and rating,
  pass/partial/fail counts, pass rate, completion rate, safety warnings, the
  per-category table, the Hard Mode capabilities and the scenarios that did
  not pass. --output holds the summary (tool_eval) and tool-eval-bench's
  JSON as written (tool_eval_raw); <output>.tool-eval/ keeps its
  run.json, progress events, Markdown report and SQLite database. A failed
  run still writes --output with tool_eval.error and exits with code 1.
• Validated on GLM-5.3-Flash NVFP4 TP4 (4× RTX PRO 6000 Blackwell): 91/100
  median over 12 runs (86–93), about 85 s per run. --tool-eval and the
  leaderboard command run directly give the same config fingerprint, and their
  scores differ less than the run-to-run noise of --parallel 4.

v0.7.5: lil-bench 1.3.2 KV capacity of hybrid models

KV capacity comes from vLLM's own kv_cache_size_tokens (group- and DCP-aware); hybrid models such as Qwen3.8-Flash-Next no longer show many times their real KV capacity. See CHANGELOG.md.

v0.7.4: lil-bench 1.3.1 p2pmark reuse and PCIe link fixes

p2pmark runs safely next to the loaded model and is reused on later runs; no false PCIe recovery warnings; clearer max-num-seqs skips. See CHANGELOG.md.

v0.7.3: lil-bench 1.3 PCIe ACS

Records PCIe ACS of the GPU paths (needs docker exec --privileged). See CHANGELOG.md.

v0.7.2: lil-bench 1.2 image integrity

lil-bench flags containers whose code differs from their image build. See CHANGELOG.md.

v0.7.1: lil-bench 1.1 telemetry

lil-bench 1.1 adds three recordings: vLLM counters and host CPU every second, PCIe traffic per GPU (nvidia-smi dmon), and PCIe error counters. See CHANGELOG.md.

v0.7.0: lil-bench standardized container benchmark

See CHANGELOG.md: the new lil-bench standardized container benchmark with upload to docker.local-inference-lab.ai, plus LLM_BENCH_EVENT_FILE and LLM_BENCH_NO_UPDATE_CHECK in llm_decode_bench.py.