- decider/serve_vllm.py: the /v1/systemone readout on vLLM 0.29.0 for large stock or chat-layout
checkpoints. Rows from decider.serve.prepare (same token ids, chat template, thinking off, same answer
slot); one generated token per row with logprob_token_ids = the option letters; softmax(logprob / T)
over the letters equals softmax(logit / T). Temperatures from decider.temperature. Rows over a shared
prefix of at least one cache block run the first row alone so the others hit vLLM's prefix cache.
Request limits and capacity messages as in decider.serve (HTTP 413 / 503). Rows run in one TaskGroup;
a failed row, a client disconnect (HTTP 499) or a cancelled handler cancels them, and the admission is
released only after they have stopped, also when the handler is cancelled again during that cleanup.
The warm-up fits the model length.
- decider/vllm_worker.py: vLLM worker extension; turns cuDNN SDPA off in the worker and raises vLLM's
128-id logprob_token_ids cap to 256, so questions with up to 255 options are one request (wider
questions are split into slices when a worker keeps 128).
- decider/prompt.py with_layout, decider/serve.py: DECIDER_LAYOUT=chat|plain replaces the layout of
decider_config.json, so a stock checkpoint can be read in the chat layout.
- scripts/stage_release.py copies serve_vllm.py and vllm_worker.py; tests/test_serve_vllm.py (13 tests,
CPU, skip without vLLM).
- README.md, docs/SERVING.md section 8, docs/CHANGELOG.md: the vLLM server, its separate environment
(vLLM 0.29.0 needs numpy 2), and measurements with Qwen/Qwen3.6-27B at T 1.943 on one B300.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>