Repository navigation
v0.13.2
A patch release: a model that serves input-scoring tasks on SGLang reserves
memory for their logits, so the engine no longer runs out of memory under full
load. The plan lowers mem_fraction_static and sets a chunked_prefill_size,
reported in --dry-run under serving.<group>.engines.<model>.headroom;
infer.overrides wins. No identity or on-disk format changes: meta.json
identity stays schema 4 and 0.13.1 result directories resume.
Fixed
- An engine that serves input scoring reserves logit headroom. A task that scores its input (
TaskRequirements.input_scoring: the PPL and CLP tasks) makes SGLang compute full-vocabulary logits for every prompt token of a prefill batch, and a recipe sized for generation left too little memory for them: under full load the engine died and every later request failed. An SGLang plan serving such a task now capschunked_prefill_sizeatmin(10% of GPU memory ÷ row, longest scored prompt, 8192), rounded to 1024, whererow = vocab_size × (dtype bytes + 12), and lowersmem_fraction_staticby that reserve's share of GPU memory (Qwen3-30B-A3B-Base on H100 80GB: 3072 tokens, 0.87 becomes 0.79).vocab_sizeandtorch_dtypecome fromconfig.json, GPU memory fromSIEVAL_TARGET_GPUornvidia-smi; when either is unknown, or on a vLLM plan, the plan is unchanged and the reason says why. A pinnedinfer.overridesvalue is kept, but a pinnedchunked_prefill_sizewhose reserve exceeds the 10% budget is refused unlessmem_fraction_staticis pinned too. The reserve changes the launch only, so it is recorded ininfer_plans.yamlandruns[].serving, never in the inference identity, and a resume does not take it twice (#243, closes #242).