[Roadmap]: PD Disaggregation with `NixlConnector` Roadmap

[NickLucche]

Description

This RFC tracks the current state and planned improvements for Prefill-Decode (P/D) Disaggregation using the NixlConnector, which enables high-performance KV cache transfer between prefill and decode instances using the NIXL library.

Currently Supported Features

Core Infrastructure

• NIXL Integration - Core P/D disaggregation framework ([P/D] NIXL Integration #17751)

Async KV Cache Transfers

• Fully asynchronous KV cache transfers
  • [Bugfix][Async][Connector] avoid vllm-side double free during async scheduling + request abort + async KV cache transfer #33377 - Bugfix for async scheduling + request abort + async KV transfer
  • [Core] Simplify async KV output aggregation #28327 - Simplify async KV output aggregation
  • [KV offload] Offloading connector async scheduling support #27648 - Async scheduling support
  • [BugFix] scheduler: Fix resuming of preempted requests after async load #31583 - Fix resuming preempted requests after async load

Multi-Transport Backend Support

• Multi-transport backend support - UCX (default), LIBFABRIC, and other NIXL plugins
  • ROCm support through RIXL library
  • Support for OOT NIXL backends via kv_connector_extra_config (Document NixlConnector backend selection via kv_connector_extra_config #33552)

Tensor Parallelism

• Homogeneous Tensor Parallelism - P and D instances with matching TP sizes ([P/D] NIXL Integration #17751)
• Heterogeneous Tensor Parallelism - Support for different TP sizes between P and D
  • [P/D] Heterogeneous TP #18833 - Base heterogeneous D TP > P TP support
  • [Nixl] Heterogeneous TP support FlashInfer #20189 - Heterogeneous TP for FlashInfer
  • [NIXL] Support P tensor-parallel-size > D tensor-parallel-size #27274 - P TP > D TP support (including MLA use-case)

MLA

• Add support for MLA caches with different latent dim (Deepseek v3.2 Indexer)
  • [NIXL] Add support for MLA caches with different latent dim #25902

CPU Host Buffer Transfers

• CPU host buffer transfers - Support for platforms without direct NIXL GPU-GPU transfer (D2H->H2D), for TPU, XPU and more.
  • [P/D] Support CPU Transfer in NixlConnector #18293 - Base CPU transfer support
  • [Cuda2CPU][P/D] Add cuda2cpu support in NixlConnector #24690 - CUDA to CPU memory transfers
  • add cpu option for p/d in nixl_connector #28356 - Pure CPU environment support

Heterogeneous Configurations
The following also partially enable Hybrid hardware deployment among other use-cases.

• Support kernel_block_size != block_size (logical <> physical block_size mismatch)
  • OffloadingConnector: Support kernel_block_size != block_size #30692 -
• Heterogeneous block sizes - Different block sizes between P and D instances (cc @xuechendi )
  • [NIXL] heterogeneous block_size support #26759 - Heterogeneous block_size support
  • [NIXL] refine decoder side post process for heterogeneous BlockSize and kv_layout #30275 - Decoder-side post-processing for heterogeneous BlockSize
• Heterogeneous KV layout (experimental) - HND to NHD permutation via enable_permute_local_kv
  • [KVConnector][Core] Support cross-layer KV blocks #27743 - Cross-layer KV blocks support
  • [NIXL] refine decoder side post process for heterogeneous BlockSize and kv_layout #30275 - Heterogeneous layout handling

Reliability & Observability

• Compatibility hash validation - Automatic P/D configuration compatibility checking
  • [NIXL] Add compatibility checking to NIXL KV connector handshake #29503 - Compatibility checking in NIXL handshake
  • [MISC]: change NIXL compatibility hash logging level to debug #30182 - Debug logging level for compatibility hash
• Transfer failure handling - Block invalidation and kv_load_failure_policy (fail/recompute)
  • [NIXL][Bugfix] Failure logging overhaul + early metadata free on failure #32031 - Failure logging overhaul + early metadata free
  • [bugfix] avoid NIXL_ERR_REMOTE_DISCONNECT in nixl_connector when Prefill dies #28120 - Avoid NIXL_ERR_REMOTE_DISCONNECT on prefill failure
  • [Docs] Nixl Usage recommend fail kv_load_failure_policy #32198 - Document fail kv_load_failure_policy
  • [NIXL] Add remote_request_id to kv_transfer_params #29665 - Add remote_request_id for better tracking
• NIXL telemetry and metrics - Transfer duration, throughput, failure counters (Prometheus)
  • [NIXL][Misc] Expose metrics from NIXL for logging to CLI #25388 - Expose NIXL metrics for CLI logging
  • [P/D][Nixl] Introduce KVTransferMetrics and aggregation strategy #22188 - KVTransferMetrics aggregation strategy
  • [Nixl][Bugfix] Track nixl_num_kv_expired_reqs metric in Prometheus #32340 - Track nixl_num_kv_expired_reqs in Prometheus
  • [KVConnector] Add KV events to KV Connectors #28309 - KV events infrastructure
• Request timeout/expiration - Automatic KV block release on P side via VLLM_NIXL_ABORT_REQUEST_TIMEOUT
  • [PD][Nixl] Remote consumer READ timeout for clearing request blocks  #20139 - Remote consumer READ timeout for clearing blocks
  • [Nixl][Bugfix] Track nixl_num_kv_expired_reqs metric in Prometheus #32340 - Expired requests metric tracking
• Lease TTL renewals for improved freeing of kv blocks on P - [Nixl][PD] Lease renewal TTL KV blocks on P #41383 (supersedes VLLM_NIXL_ABORT_REQUEST_TIMEOUT)

Deployment Configurations Guides & Docs

• Multi-instance deployments - Multiple P and D instances across hosts
  • [Disagg] Support large batch size in proxy server and update NixlConnector doc for DP #28782 - Proxy server improvements for high concurrency
  • [test/doc] make NixlConnector example more clear #24249 - Clearer deployment examples
• Data Parallel support - DP deployments with per-rank side channel ports
  • [Disagg] Support large batch size in proxy server and update NixlConnector doc for DP #28782 - DP deployment documentation and proxy improvements
• Bidirectional KV Transfer - [Docs][PD][NIXL] Bidirectional kv-cache transfer #43097
• KV Cache block Lease mechanism - [Docs][PD][NIXL] Lease extension mechanism for blocks on P #43099

Spec Decoding

• Speculative decoding integration - P/D disaggregation with speculative decoding +CI @ZhanqiuHu
  • Fix for layer-by-layer transfer (not NIXL) [Spec Decode][KV Connector] Fix KV transfer in PD + speculative decoding #35158
  • [Test] Add PD disagg + SD acceptance tests #35760
• HMA enabled Hybrid (SW) SD integration - example Kimi K2.5 Eagle3 see https://vllm-dev.slack.com/archives/C07RFT2UT16/p1773434620383319
  • Follows us from [Core][KVConnector] Support HMA+NixlConnector #35758
• Hybrid SSM + SD
  • Qwen3.5+MTP - [CI] Add MTP + PD disagg test for Qwen3.5 #42677
• Clarify Spec Decoding compatibility matrix
• Fix PD async KV load lookahead handling for spec decode - [P/D][Bugfix] Fix PD async KV load lookahead handling for MTP spec decode #46694

SSM

• NIXL+Hybrid Memory Allocator - [Core][KVConnector] Support HMA+NixlConnector #35758
• SSM (Mamba) support - [PD][Nixl] Add support for hybrid SSM-FA models  #34727
• TP>1 async scheduling issue - [Bug]: PD disaggregation for SSM models requires --no-async-scheduling when TP>1 #37285
• Fix Mamba P/D workflow to avoid recomputing last token twice (both on P and D) - [SSM/Mamba] Follow-up: N-1 prefill for P/D disaggregation #37310
• Heterogeneous TP support for hybrid SSM models - [NIXL][Mamba][3/N] Heterogeneous TP: 3-read conv state transfer #37635
• Mamba1 support - [NIXL][Mamba] Add Mamba1 support to NIXL P/D disaggregation #45019

Work in Progress

• Avoid reading expired blocks in bidirectional turn-2 read
• [NIXL] Avoid reading expired blocks in bidirectional turn-2 read #47021
• Enhanced error diagnostics - Structured logging with failure context for easier debugging
• Enable drain scaledown mode for single process deployments - [Frontend] Enable drain shutdown mode for non-DP deployments #32420
• Speculative decoding integration - P/D disaggregation with speculative decoding +CI @ZhanqiuHu
• CPU Offloading verification and CI tests - [CI] Add Nixl+OffloadingConnector e2e integration tests #39200
• PD+CPU Offloading+Spec Decoding verification and CI tests
• DecodeContextParallel support - feat(nixl,dcp): Supports MLA DCP for PD disaggregation with nixl connector #38433
• Bi-directional KV transfers with Nixl connector - [P/D] Prefill compute optimizations with bi-directional KV cache transfers between P and D nodes #32553
• GDN model support (Qwen3.5+) - PD disagg with NIXL Connector: GDN support (Qwen3.5) #41869
• Mamba prefix caching mode align/all - [PD][Nixl] Mamba prefix caching mode support  #42554
• Pipeline parallelism support - P/D disaggregation with pipeline parallelism [Feature]: Support NixlConnector with Pipeline Parallelism for disaggregated serving #40674 [KV Connector][2/N][NIXL] Pipeline-parallel support for PD-disaggregated serving with NIXL connector #43366
• NixlConnector Refactor:
  • [KVConnector][NIXL] Organize NIXL connector into its own directory #39354
  • nixl refactor [2/N]: unify TpKVTopology + HeteroTPTransferConfig into TransferTopology #39529

Planned

• Nixl + HMA support request failure handling
• Fix SpecDecoding asymmetric num_speculative_tokens UX - [Bugfix][DFlash]allocate the proper number of lookahead slots #43733 (review) (will likely require roles to be defined at config time)
• Documentation improvements - Clarify PD feature matrix in docs with examples
• Multi-backend model support - Models with multiple attention backends (mostly validation of HMA feature coverage)
• Hybrid hardware deployment - Supported in the measure tested by @xuechendi and team. Another AMD-Nvidia use-case reported https://uccl-project.github.io/posts/uccl-ep-full/. This is un-tested in CI and we should clarify capabilities and limitations.
• FP8 kv cache support (attention-dependent for now, depending on how scales are stored) - Issue requesting support [Bug]: FP8 KV cache corrupts output in Qwen3.5-397B-NVFP4 Disagg serving #42179
• nvfp4 kv cache support

Backlog

• HTTP-based handshake endpoint - Replace ZMQ side channel with HTTP for better observability
• Transfer Failure handling for HMA
• More efficient h2d copy_blocks operations for HMA groups
• Heterogenous block size (blcok_size_ration > 1) HMA support

NixlPushModeConnector

• Support "Push/WRITE" P->D mode - [KV Connector]: Support KV push from Prefill to Decode node using Nixl KV Connector #35264
• PP support for long context serving [KVConnector][NIXL] Support pipeline-parallel prefill in push mode #45880

RFCs

• Bi-directional KV transfers with Nixl connector - [RFC]: [P/D] Prefill compute optimizations with bi-directional KV cache transfers between P and D nodes #32733
• Remove Per-Block KV Transfer Error Handling - [RFC]: Remove Per-Block KV Transfer Error Handling #35780
• kv_both role deprecation - [RFC]: Deprecate kv_both for NIXLConnector and Enforce Explicit P/D Roles #43807

Known Issues/Bugs:

• Gemma3 heterogeneous TP accuracy drop - [Bug]: Gemma-3 specific heterogeneous TP failures with PD disagg #37333
• TP-dependent block sizes break HMA heterogeneous TP - [Bug] _align_hybrid_block_size produces TP-dependent block sizes, currently unsupported when local and remote kernel block size mismatch #41037
• Bidirectional KV transfer produces incorrect results when reasoning traces are stripped between turns - [Bug][PD] Bidirectional KV transfer produces incorrect results when reasoning traces are stripped between turns #43094
• PD+SD Prefix-cache trimming drops wrong block when P has extra lookahead block - [Bug]: [PD + SpecDec] Prefix-cache trimming drops wrong block when P has extra lookahead block #43996

Bug Fixes

• Fix NIXL handshake failures not honoring kv_load_failure_policy - [Disagg] Fix NIXL handshake failures not honoring kv_load_failure_po… #33745
• Fix multi-node TP (TP>8) - [Bugfix][PD] Fix multi-node TP (TP>8) #39907
• Fix CG dispatch for mamba PD to FULL_DECODE - [Bugfix] mamba: run single-token extends as decodes #42430

Related Projects

• Encoder-Prefill-Decode Disaggregation: [Core] Encoder separation for Encode-Prefill-Decode Disaggregation #25233
• Mooncake Transfer Engine: [P/D] Introduce Mooncake Transfer Engine as kv_connector #24718, [P/D] Refactor mooncake connector sender thread using async coroutines #31573

cc @robertgshaw2-redhat @tlrmchlsmth @markmc @njhill @orozery

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[tlrmchlsmth]

@NickLucche and @orozery

I think we need to track MTP + KV Connector support here and in #33689.
I'm still running down current status but I think there's an issue because the draft model forward passes are done outside of the with maybe_get_kv_connector_output() context, so I don't think the draft layers' KV caches are being handled properly.

IIUC there may be a race conditions in some cases and a complete failure to save or load layers in other. Still digging to understand further

[snadampal]

Hi @NickLucche , can you please add the "bi-directional KV transfers with Nixl connector (#32733)" feature as well to the work in progress items? thank you!

[NickLucche]

Thanks for bringing this up @snadampal and apologies on the delay in getting back to you on your RFC so far.

[ZhanqiuHu]

FYI: filed known issue #41037 — _align_hybrid_block_size produces TP-dependent block sizes, which breaks HMA heterogeneous TP NIXL transfers when block_size_ratio != 1.

[zsn6493]

I opened a related RFC here: #43470

This is not about replacing PD disaggregation. The goal is to discuss whether PD deployments should become more adaptive by identifying the dominant bottleneck (prefill, decode, or decode-side KV) and expanding the most effective side first.