v0.1.5.post2 PD Disaggregation: vLLM Prefill + TileRT Decode

🔀 TileRT v0.1.5.post2 – PD Disaggregation: vLLM Prefill + TileRT Decode

We're excited to announce TileRT v0.1.5.post2, the first GitHub release on the v0.1.5 line, which introduces PD (prefill–decode) disaggregation — vLLM prefill + TileRT decode, behind a single OpenAI-compatible endpoint. Supported on GLM-5/5.1 and DeepSeek-V3.2.

With this release, TileRT plugs into vLLM as a decode engine through vLLM's standard V1 KVConnector interface — no vLLM fork or patch required. Latency-critical requests are routed to the TileRT decode pool; other traffic can stay on native vLLM decode, all behind the same OpenAI surface.

✨ Key Highlights

🔌 PD Disaggregation, in the box

Everything ships inside the tilert wheel under tilert.pd_vllm — the prefill-side connector plugin, the decode server, and the OpenAI-compatible router:

• Stock vLLM prefill — the TileRT connector loads via vLLM's standard kv_connector_module_path; the prefill node runs an unmodified vLLM build.
• TileRT decode — attention state is handed off over RDMA (NIXL or Mooncake transports), with fp8 KV cache on both ends.
• OpenAI-compatible router — clients keep using /v1/chat/completions; the router runs prefill on vLLM (first token), hands off to TileRT decode, and streams the completion back.
• Two topologies — a dedicated vLLM-prefill → TileRT-decode pipeline, or a shared prefill pool feeding TileRT and native vLLM decode side by side under vLLM's MultiConnector.
• MTP-aware — the prefill populates the draft-layer KV that decode-side speculation resumes from.

See the Disaggregated Serving guide in the README for launch commands and configuration.

🛠 Router & Client-Compatibility Fixes (post2)

On top of v0.1.5, this release fixes four tilert.pd_vllm issues that affected OpenAI-compatible clients (see #55):

• Prefill request hygiene — the prefill request no longer forwards stream_options or max_completion_tokens from the client body, which previously caused vLLM 400s for streaming clients and made the prefill node generate the full answer instead of one token.
• ignore_eos support — the flag is now forwarded end-to-end and honoured by the decode engine, enabling fixed-length generation.
• Streaming usage — usage now arrives in its own trailing chunk with choices: [], the shape vLLM and the OpenAI API emit, so standard clients read token counts correctly.
• Relaxed dependency pins — transformers and tokenizers moved from == to >=, so tilert installs cleanly alongside the newer transformers a vLLM prefill environment needs.

📦 Install

pip install tilert==0.1.5.post2

sha256 of the wheel: e65b876ccfc1a419b0047a6d6b395f619ea35c15194ad1892f171c78476fe407

🤝 Join the Community

TileRT is developed in the open, and user feedback plays a key role in shaping its evolution ❤️

• ⭐ Star the repo to show your interest and support
• 🐞 Open issues to report bugs, share feedback, or request features
• 💬 Start discussions around use cases, performance observations, and integration experiences

Let's keep pushing the limits of low-latency inference together with TileRT 🚀