Skip to content

v1.5.0

@Mapika Mapika tagged this 25 Sep 08:17
- decider/serve_vllm.py: the /v1/systemone readout on vLLM 0.29.0 for large stock or chat-layout
  checkpoints. Rows from decider.serve.prepare (same token ids, chat template, thinking off, same answer
  slot); one generated token per row with logprob_token_ids = the option letters; softmax(logprob / T)
  over the letters equals softmax(logit / T). Temperatures from decider.temperature. Rows over a shared
  prefix of at least one cache block run the first row alone so the others hit vLLM's prefix cache.
  Request limits and capacity messages as in decider.serve (HTTP 413 / 503). Rows run in one TaskGroup;
  a failed row, a client disconnect (HTTP 499) or a cancelled handler cancels them, and the admission is
  released only after they have stopped, also when the handler is cancelled again during that cleanup.
  The warm-up fits the model length.
- decider/vllm_worker.py: vLLM worker extension; turns cuDNN SDPA off in the worker and raises vLLM's
  128-id logprob_token_ids cap to 256, so questions with up to 255 options are one request (wider
  questions are split into slices when a worker keeps 128).
- decider/prompt.py with_layout, decider/serve.py: DECIDER_LAYOUT=chat|plain replaces the layout of
  decider_config.json, so a stock checkpoint can be read in the chat layout.
- scripts/stage_release.py copies serve_vllm.py and vllm_worker.py; tests/test_serve_vllm.py (13 tests,
  CPU, skip without vLLM).
- README.md, docs/SERVING.md section 8, docs/CHANGELOG.md: the vLLM server, its separate environment
  (vLLM 0.29.0 needs numpy 2), and measurements with Qwen/Qwen3.6-27B at T 1.943 on one B300.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Assets 2
Loading