[Roadmap]: SGLang Distributed KVCache System For Agentic Workload

[hzh0425]

Motivation

As agentic workloads drive rapid growth in KV cache storage and transfer volumes, the existing PD disaggregation + HiCache stack is hitting significant bottlenecks. Meanwhile, HiCache's compatibility with hybrid models and other evolving features remains limited. There is an urgent need for a robust, high-performance distributed KV cache system capable of sustaining the heavy demands of large-scale agentic scenarios.

The core objective is to achieve end-to-end integration of PD incremental transfer, HiCache / HiSparse, pipeline parallelism, and Eagle — validated at scale on DeepSeek DSA (V32、GLM5) and Linear model (Qwen35) families.

Q3 Roadmap

• Agentic KV Cache Scheduling

  • Support session-aware RadixTree and HiCache (PR #29173)
  • Build a KV cache orchestrator across the router and engine to detect agentic workflows—including sessions, sub-agents, and tool calls—and provide scheduling hints such as PREFETCH, DEMOTE, and PIN; integrate with the router roadmap for global KV cache scheduling (Issue #25760)
  • Support direct L3 cache mode to enable a larger globally shared memory pool [Unified Cache][Draft POC]: Add optional external-cache linker mode to Unified Radix Cache #35687
  • Build a CPU-only KV cache simulator that replays representative agentic workloads without GPUs (Issue #21891)
  • Support KVCache Sequence Split Mode to distribute VRAM pressure evenly. KV Cache Shard with Sequence Split #30501
  • Support Unified-KVCache Layout Unified KV Cache Layout in L3 #33651

• UnifiedRadixTree Enhancements

  • Decouple the tree core and implement it in Rust to accelerate hot paths such as matching and eviction (PR #29901) 、Rust Tree Core with Full/SWA/Mamba Components #32710
  • Integrate with KVCache-Canary to enable debugging capabilities for KV cache correctness. [HiCache] Add canary sentinel verification for unified tree MHA/SWA #32476

• PD Disaggregation Integration

  • Support an L2 DecodeRadixTree to increase transfer batch sizes and reduce GPU memory usage
  • Support SWA/DsV4/Mamba/DSpark with decode unified radix treee
  • Explore a proof of concept for Prefill-as-a-Service

• HiSparse Production Readiness

  • Improve the reliability and production readiness of HiSparse (Issue #28874)

Q2 Roadmap

The roadmap below is still incomplete and will be expanded with more content related to Agent/SubAgent KVCache.

• HiCache: Support Hybrid Model (Linear & DSA) @hzh0425

  • Hybrid Cache Controller & Support Linear Hybrid Model [HiCache][HybridModel]: Support mamba state offloading & HybridCacheController #20457
  • Mooncake backend support for hybrid model [HiCache & HybridModel] mooncake backend support DSA & mamba model #21259 @huangtingwei9988
  • Support MLA + Mamba Hybrid model(KeLing) [HiCache & Hybrid model] Support mla + mamba hybrid model #22957
  • 3FS Backend support for hybrid model [HiCache & HybridModel] 3FS backend support DSA & mamba model #23241 @kangyifei
  • HiCache support for Unified Hybrid Radix Cache [Roadmap] Unified Hybrid Radix Cache Refactor #20415
  • Support DeepSeekV4 [UnifiedTree]: Support HiCache For DeepSeek_V4 #24691
  • Support MiMO-V2 feat: Support HiCache for MiMo-V2 models (1/N) #27378
  • Dedup MLA KV cache in host memory across TP ranks [HiCache] Dedup MLA KV cache in host memory across TP ranks #26691

• PD-Decode Support HiCache @hzh0425 @huangtingwei9988

  • Support Decode Side RadixTree @ishandhanani
  • Support Decode Side HybridRadixTree (SWA/Mamba) ([P/D disagg] Decode-side radix cache for SWA hybrid models (unified radix tree) #27770)
  • Storage Prefetch Interface — Expose a prefetch API that allows newly arriving decode requests to query storage hits and initiate Storage→Host→Device prefetch directly
  • Support L2 RadixTree to increase cache hit rate
  • Storage supports the concept of groups, enabling unified visibility and eviction for groups composed of multiple keys within a page (e.g., key and value, split by rank, etc.). @huangtingwei9988
  • Heterogeneous TP Support For PD With HiCache @huangtingwei9988

• PD Disaggregation：Idea provided by @ShangmingCai @hzh0425

  • Incremental KV Transfer — Decode queries storage for cached prefix tokens and reports the hit count to Prefill via protocol extension; Prefill only transfers the non-cached (incremental) portion
  • PD Host TransferMode — Prefill transfers the KV cache directly to Decode's DRAM, bypassing the GPU as an intermediary；Helps increase the decode-side batch size in long-context scenarios. [PD]: Add support for HiSparse to directly transfer the cache from Prefill to Decode DRAM. #21591
  • Support for large decode batches: During decoding, use the L2 and L3 prefix portions cached on the host (hit from prior computations) along with the incremental part transferred from the prefill phase; then load everything back onto the GPU to perform subsequent inference steps.

• Agent/Rollout KVCache Management @hzh0425 @stmatengss @ishandhanani

  • [RFC] Agent-Aware KV Cache Phase 1 for Agentic Workloads #24656
  • [Agentic Inference] Programmatic KV Cache for Agentic Workloads #27574

• HiCache Compatibility with Other Features

  • PP × HiCache Consistency Fix — Resolve compatibility issues between PP and HiCache by synchronizing Rank 0's cache events across PP ranks via event numbering, ensuring RadixTree state consistency @hzh0425 @whybeyoung [PP + HiCache] HiCache Consistency Fix Plan #22607
  • MTP × HiCache [HiCache] feat: add draft KV cache backing for L2/L3 #21125 [HiCache] Support packed and sidecar draft caches for MTP/EAGLE/DSpark #30393
  • EP & DP × HiCache
  • CP × HiCache [HiCache] Add synchronization for context parallelism #20460
  • Support CP Layer-Wise Split With HiCache: [Feat][GLM5.2] Add DSA Cache Layer Split under Prefill CP #29421

Related resources

No response

[adityavaid]

Just a novice a question.
Are you looking in direction of Plugging LMCache for transport/offload/remote-tier mechanics where it already shines (since SGLang already supports LMCache) ?

Or this is going to be a ground up effort from scratch.

[dongyibo]

hello~, is there any plan to support 「pd-disagg + pp + mtp 」

[hzh0425]

hello~, is there any plan to support 「pd-disagg + pp + mtp 」

cc @ShangmingCai

[ShangmingCai]

hello~, is there any plan to support 「pd-disagg + pp + mtp 」

@dongyibo Hello, it is supported already, but only for prefill pp + decode mtp (because prefill instance configured with mtp server args is not supported since pp's compatibility with mtp is not ready yet).

[qimcis]

I'd love to pick up work on hybrid model offloading support + support L2 radixtree, if these aren't already in progress; After I complete my work on #21860 #21862

[dongyibo]

hello~, is there any plan to support 「pd-disagg + pp + mtp 」

@dongyibo Hello, it is supported already, but only for prefill pp + decode mtp (because prefill instance configured with mtp server args is not supported since pp's compatibility with mtp is not ready yet).

@ShangmingCai hello, based on your description, does this mean that under pd-disagg, the prefill nodes can use PP, while the decode nodes can skip PP but enable MTP?
It's known that the draft model performs its first extend operation after prefill. If so, the prefill nodes would skip this operation—would that cause any issues?

[hzh0425]

I'd love to pick up work on hybrid model offloading support + support L2 radixtree, if these aren't already in progress; After I complete my work on #21860 #21862

Thanks, we're already working on it.