vllm-rlt v0.2.0 — Core runtime refactor

vllm-rlt v0.2.0 — Core runtime refactor

This release centers on the underlying runtime refactor: unified Stage/Loop
execution, explicit resource ownership, and clearer boundaries between scheduling,
execution and result application. Scheduler-owned progress and shared execution
contracts provide the foundation for recurrent-model batching and optional
execution paths.

It is a versioned research and evaluation milestone for recurrent-model inference.
This is not a production-stability, broad hardware, or API-compatibility guarantee.
Start with default synchronous execution and qualify optional paths for your workload.

Release source: c08b9d79834bb8b4ff283b15bae2c14c559f6b52.

Three highlights

1. Core runtime refactor. Unified Stage/Loop execution makes Stage the deployment/resource boundary and loop progress visible to scheduling. Scheduler owns request progress and result application, with shared execution contracts rather than separate runtime paths for each deployment.
2. WCPB on the prefill side. Opt-in wavefront chunked prefill schedules ready prompt ranges across chunks and recurrent depths under per-depth KV dependencies. Compatible ranges can share a recurrent batch; it is not merely splitting one prompt into sequential chunks. It requires LAST_EXITED KV and compatible refill/mixed scheduling. WCPB design.
3. Token pipelining on the decode side. Experimental self-speculation advances tokens at different recurrent depths in the same recurrent batch. The asynchronous path keeps token/hidden/acceptance state on the GPU and can submit two steps before CPU delivery; compatible CUDA Graph and preemption paths are supported. Greedy sampling, fixed target depth and LAST_EXITED KV are required; block mode remains the default. Decode pipeline.

Prefill and decode now have complementary pipelined execution paths on the shared runtime. In supported same-host PD deployments, the prefill worker can use WCPB and the decode worker can use speculative pipelining, connected by the existing KV-transfer/admission contract. P/D execution pipelining and P/D disaggregation are different concepts: enabling one does not automatically enable the other. This is not a claim that every model/backend/feature combination is qualified, that all stages overlap continuously, or that a fresh end-to-end GPU acceptance campaign passed for this tag. Mixed recurrent batches do not include speculative work; Huginn does not support these PD/WCPB/speculative combinations.

Additional changes include native Huginn and bounded Ouro-Thinking integration, plus optional native CUDA SiLU-and-Mul for Ouro. Checkpoint support does not qualify every backend or feature combination; see the support matrix. Fixed draft length remains explicit; adaptive draft-length policy is not included. The native operator is not yet integrated into Nanbeige or Huginn.

This release includes the async preemption token-restoration fix #109 and speculative request-failure isolation/cleanup fix #115. Their historical v0.1.0 warnings must not be mistaken for unchanged current defects.

Install and migrate

Requires Python 3.10+ and PyTorch 2.5+. Model weights are separate.

git clone --branch v0.2.0 --depth 1 https://github.com/InferMatrixLab/vllm-rlt.git
cd vllm-rlt
uv pip install -e .
vllm-rlt-serve --help

The attached py3-none-any wheel is Python-only: it uses the Torch activation fallback, not a precompiled CUDA extension. The source distribution includes native C++/CUDA sources. To build the optional extension, use compatible CUDA-enabled PyTorch, the CUDA toolkit's nvcc, a C++ compiler and --no-build-isolation; keep your allocated GPU visible. Native binaries depend on Torch/CUDA/target architecture. See the first-request guide.

LLM now starts a spawned stage worker. Construct it under an if __name__ == "__main__": guard and use its context manager. LLM.engine is a StageClient, not direct access to worker-local caches. See migration notes.

Validation and limits

• Final-source CPU regression: 911 passed, 584 skipped. Skips are not passes; CUDA and optional-asset/dependency coverage is not qualified by this CPU run.
• Ruff lint/format, wheel and source-distribution builds, dependency consistency and all three installed CLI help entrypoints passed. Outside the source checkout, the installed wheel reports 0.2.0 and passes tiny randomly initialized Ouro synchronous CPU generation, spawned-worker shutdown and KV cleanup. This is not real-checkpoint accuracy evidence.
• Local validation used Linux, Python 3.12.14 and PyTorch 2.15.0+cpu. Final-commit GitHub CPU CI provides its separate environment and logs.
• No fresh release-tag GPU/model-quality/performance campaign was run. #160 reports bounded B300 native-kernel/build validation on the same implementation tree before these version/docs-only changes; raw artifacts remain contributor-local. This does not qualify other GPUs, FlashAttention combinations, autograd or torch.compile.
• Existing serving benchmarks are diagnostic. Full greedy output equivalence has not passed across all configurations; do not interpret them as equal-quality speedups or production acceptance. Adaptive exit can change quality; historical model measurements are not fresh tag measurements.
• Huginn remains synchronous fixed-depth LAST_EXITED, with recurrent-core/coda decode graphs; mixed/WCPB staged prefill, async/speculation and PD are excluded. Other optional combinations have their own restrictions.
• The HTTP interface exposes completions, not /v1/chat/completions, and has no built-in authentication/TLS. Use trusted/local access or an appropriately secured gateway.
• Unmerged proposals, including training integration, FP8 KV, FlashLoop and LoopCD, are not release features.

Artifacts: Python wheel, source distribution and SHA256SUMS. This is a GitHub release; no PyPI publication is implied. Source licensing remains Apache-2.0; see LICENSE and NOTICE.

Full source comparison