v0.1.0: Experimental release

vllm-rlt v0.1.0: Experimental release

The first tagged, experimental pre-release of vllm-rlt, a standalone inference and serving engine for recurrent language models. It implements loop-level continuous batching: requests at different recurrent depths can share a batch while progressing toward their next token. vLLM does not need to be installed.

For research and evaluation; not production-ready. Start with the default synchronous path. Known correctness and failure-recovery issues affect async execution with preemption and async self-speculation; #109 and #115 are not included. See the detailed limitations below before enabling advanced options.

Release source: ea3f74c5cd9c280086195d996a4f9fbcf82bc04b.

Highlights

Supported models

• ByteDance/Ouro-1.4B: native checkpoint loading, fixed-depth inference, and adaptive decoding with the model's learned early-exit gate. The first-request guide pins checkpoint revision 574fa66cb8bf5abdc979642d01cf2b79b16bfab1.
• Nanbeige/Nanbeige4.2-3B: native model-config dispatch and safetensors loading for its fixed two-loop recurrent architecture (#63). Published speed figures below use Ouro-1.4B.

Measured speed

Synchronous self-speculation: 236.45 committed decode tokens/s, versus 154.23 tokens/s for native fixed-depth async decoding (~1.53×). Experimental async self-speculation reached 243.00 tokens/s (~1.58× the same baseline; +2.8% over synchronous speculation), with the unresolved #115 failure-handling limitation.

These are historical medians of five trials on one NVIDIA H20-3e, Ouro-1.4B BF16, Triton 3.6.0, eager execution and LAST_EXITED KV: 64-token prompt / 32-token output, concurrency 8, fixed target depth 4, draft depth 2, K=2. Committed-decode timing excludes prefill, the first output token and HTTP/client overhead. Ratios are calculated from reported medians.

Source: #66's matched BF16 measurements, measured runtime d977817d4e174cbc2fcc9aa7ef5893c35da2ec89. All 80 measured trials across four configurations and four workload cells, plus warmups, matched the native fixed-depth baseline's token IDs. These are workload-specific engine measurements, not fresh v0.1.0 GPU results or a peak-capacity/production guarantee.

Main features

• Loop-level continuous batching: mix requests at different recurrent depths, with refill/no-refill scheduling, chunked full-depth prefill, dynamic arrivals, cancellation and per-request loop controls.
• Depth-aware paged KV caching: LAST_EXITED and SHARED layouts, automatic CUDA cache sizing, plus opt-in prefix reuse, incremental allocation, priority scheduling and CPU-snapshot preemption.
• Configurable GPU execution: Torch/Triton/optional FlashAttention backends and capability-based auto selection; opt-in asynchronous scheduling, reusable buffers, decode-core CUDA Graphs, and fixed-depth synchronous/asynchronous self-speculation.
• Single-host prefill/decode disaggregation: separate GPU worker pools with NIXL KV transfer and overlap between chunked prefill and transfer.
• Python, CLI and HTTP interfaces: LLM/SamplingParams, installed commands, /health, /v1/models, and OpenAI-compatible /v1/completions with JSON/SSE, greedy decoding and seeded top-k/top-p sampling.
• Profiling and evaluation: runtime profiling plus runtime and GSM8K benchmark entrypoints, with source-qualified historical baselines.

Advanced paths are opt-in. In particular, async execution with preemption can produce incorrect output (#109); supported combinations and the remaining experimental limitations are detailed below.

Install and start

Requires Python 3.10+ and PyTorch 2.5+. For GPU inference, use Linux, a BF16-capable NVIDIA GPU, and a compatible CUDA-enabled PyTorch installation. Start with the default synchronous execution settings and the Triton serving backend.

git clone --branch v0.1.0 --depth 1 https://github.com/ThinkFlowLab/vllm-rlt.git
cd vllm-rlt
uv pip install -e .
vllm-rlt-serve --help

Run this in an activated environment. The base install includes text, HTTP serving, and Linux Triton dependencies. Follow the first-request guide for the pinned Ouro checkpoint download, GPU selection, server startup, and a real completion. Model weights are downloaded separately. CPU use requires the Torch backend.

The release includes a Python wheel, a source distribution, and SHA-256 checksums. This is a GitHub release; no PyPI publication is implied.

Validation and compatibility

Checks on the exact release commit, using Linux x86-64, Python 3.12.14 and PyTorch 2.14.1+cpu:

• 480 tests passed, 164 skipped. Skips: 135 GPU tests requiring a reserved device, 26 optional lm_eval tests, and three Nanbeige tests requiring unavailable official assets.
• Ruff lint and formatting checks passed for all tracked Python files.
• Wheel and source-distribution builds passed. The installed wheel reports 0.1.0; dependency consistency and all three CLI help entrypoints passed. A tiny randomly initialized Ouro model passed default-synchronous CPU generation and KV cleanup from the installed wheel outside the source checkout.

These checks do not qualify GPU kernels, real-checkpoint accuracy, or production capacity for this exact tag. The repository has no automated Actions/check-run release gate.

Historical GPU evidence is available separately:

• The #66 validation report records source d977817d4e174cbc2fcc9aa7ef5893c35da2ec89 on one H20-3e, Python 3.10.20, PyTorch 2.11.0+cu128, CUDA 12.8 and Triton 3.6.0. Its real Ouro-1.4B BF16, greedy fixed-depth d=2, D=4, eager LAST_EXITED tests passed 26 async-module cases; 80 measured trials and warmups matched the fixed-depth native-async token IDs. This is scoped historical evidence, not a guarantee for error paths or other configurations.
• #30 and #31 contain the B300 Ouro-1.4B BF16 runtime and four-GPU serving baselines. Their workload, timing boundaries, exit-policy differences and shared-host caveats apply; no new speedup or broad hardware qualification is claimed for this release.

Known limitations

• Use synchronous execution as the initial-release baseline. Asynchronous scheduling, speculation, CUDA Graphs, disaggregation and other advanced feature combinations remain opt-in and require workload-specific validation.
• Do not combine ordinary async execution with preemption in v0.1.0. Resuming into a reassigned device slot can consume stale tokens and produce incorrect outputs. #109 tracks the unmerged fix. The merged #113 scheduling correction alone does not resolve it.
• Async self-speculation has unresolved failure handling. Certain result-collection or per-request detokenization failures can affect the whole serving worker or retain speculative resources. #115 tracks the unmerged fix. Exclude this path from reliable-serving deployments of v0.1.0.
• Adaptive exit changes computation and can change output quality. Delayed/lookahead exit modes are experimental; their quality has not been established by the fixed-depth evidence. See #56.
• The HTTP frontend has no authentication or TLS. Keep it on a trusted/local interface or put an appropriate secured gateway in front of it. It implements completions, not /v1/chat/completions; stop-string matching, beam search, prompt-list batching and multiple completions per request are not supported.
• PD is single-host, multi-GPU with complete model replicas and optional NIXL. Speculative decoding with PD is unsupported. Nanbeige support must not be read as a new real-checkpoint GPU or task-quality qualification.
• This is an experimental 0.x pre-release, not a production-readiness or stability guarantee. APIs and configuration contracts may evolve.

Migration and acknowledgments

This is the first tagged release, so there is no previous release migration. Users following main should check out v0.1.0 to pin this snapshot.

Thanks to all contributors whose work is included in this release, and to the Ouro, Nanbeige, Continuous Depth Batching and vLLM projects for their model work and ideas. Source is licensed under Apache-2.0; see LICENSE and NOTICE.