Motivation
With the introduction of HiCache, SGLang now leverages multi-level memory hierarchies to maximize the benefits of KV cache reuse. As HiCache continues to mature, our focus is expanding toward broader model support, including hybrid LLMs such as Qwen3-Next (integrating linear and full attention), as well as sparse LLMs like DeepSeek-V3.2-Exp and GPT-OSS (whose sliding-window attention can be viewed as a simple form of sparsity).
Ongoing work
You’re welcome to join the discussion in #kv-cache-store 😊
Motivation
With the introduction of HiCache, SGLang now leverages multi-level memory hierarchies to maximize the benefits of KV cache reuse. As HiCache continues to mature, our focus is expanding toward broader model support, including hybrid LLMs such as Qwen3-Next (integrating linear and full attention), as well as sparse LLMs like DeepSeek-V3.2-Exp and GPT-OSS (whose sliding-window attention can be viewed as a simple form of sparsity).
Ongoing work
Sparse LLM support: Enable selective KV cache loading to GPU memory, reducing memory footprint and allowing larger batch sizes.[Sparse & HICache]: Enables hierarchical sparse KV cache management and scheduling for DeepSeek V32. #14619
Refactored memory subsystem: Refactoring the
mem-cachecomponent to establish a more robust Radix Tree and memory pool foundation. [Feature] Memory Cache System Refactoring Road Map (Mem Cache V2) #12587Linear attention caching: Develop new query and storage mechanisms for KV caches in linear attention layers, building upon the refactored
mem-cache.You’re welcome to join the discussion in #kv-cache-store 😊