[Roadmap-HiCache]: Unified Storage Framework for SGLang Across Diverse Scenarios

[hzh0425]

Motivation

In our previous work, we introduced the HiCache framework, which provides Hierarchical storage capabilities for KVCache. https://lmsys.org/blog/2025-09-10-sglang-hicache/

SGLang currently has multiple storage implementations scattered across different modules. Several new features require similar storage capabilities:

• EPD Embedding Cache: Embedding cache with offload/prefetch (K-V) Encoder Global Cache Manager #16137
• RL Token Topk Info: Storage-based transfer to avoid slow ZMQ (K-V)
• MultiModal Streaming vlm rope info，Which can occupy several GB VRAM Support UVA Storage Framework #17763
• Elastic EP Weight Offloading： [4/N] (Elastic EP) Back up Expert Weights in DRAM #17374
• Sparse KVCache: Token-level offload/prefetch (SparseKVCacheManager)
• PD Decode Offload: Request-level KV cache offloading; (DecodeKVCacheOffloadManager) [PD-HiCache]: Support Async Offloading KVCache In Decode Side #10192
• ….. other features

Problem: Each module reimplements similar storage logic (async I/O, queues, eviction, host memory management, prefetch), leading to code duplication and inconsistent interfaces.

Goal: Provide a unified, extensible storage module with:

• Generic data transfer pipelines (Device ↔ Host ↔ Storage)
• Pluggable host memory pools for different data types
• Reusable storage backends (File, Mooncake, 3FS)
• Clean separation between generic infrastructure and business logic

Overview Design

Roadmap

• General CacheControllerV2: Abstract DeviceHostPipeline and StoragePipeline as reusable components @hzh0425

• Storage Backend Support new key-value interface @stmatengss @pansicheng

  • Mooncake Backend + KVCM
  • 3FS Backend

• EPD EmbeddingCache integration, Implement Embedding HostPool， EmbeddingCacheManager migrated to new architecture @liusy58

• DecodeKVCacheOffloadManager、SparseKVCacheManager integration @huangtingwei9988

• MultiModal Streaming vlm rope info @yuan-luo

• RL Cache integration @yizhang2077

• Elastic EP Weight Offloading: Dynamic weight management @ympcMark

• DepSeek Engram https://github.com/deepseek-ai/Engram/

[Edward-lyz]

Hello, I'd like to ask about your latest refactoring progress, especially the support for SparseKVCacheManager, which is very useful for cache optimization when deploying sglang with large-scale sparse LLMs.

[liubiyongge]

Hi @hzh0425 , I see that 'DeepSeek Engram' is currently listed in the Roadmap without a designated owner. I'm very interested in implementing this, especially the [CPU-side optimization/kernel implementation] part. Is there anyone already working on this? If not, I'd love to take this task

[magicYang1573]

Hi @liubiyongge !
We are already working on integrating Engram into the low-tier memory, including CPU-DRAM, as well as some remote memory pools such as Mooncake RDMA pool and CXL memory pool. We will share our initial version soon. We’d be thrilled to have you join the next phase of development!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.