Skip to content

[Feature] Enable MambaRadixCache (prefix caching) for KDA (KimiLinearForCausalLM) #26575

Description

@yyq0210

Summary

KDA (Kimi Delta Attention) backend currently has
support_mamba_cache=False, which means KimiLinearForCausalLM
models cannot benefit from SSM state prefix caching via
MambaRadixCache. This PR enables it by:

  1. Setting support_mamba_cache=True in server_args for
    KimiLinearForCausalLM
  2. Adding the missing _track_mamba_state_decode,
    _track_mamba_state_extend, and init_forward_metadata
    implementations in KDAAttnBackend
  3. Returning the intermediate h tensor from chunk_kda_fwd
    (already computed but previously discarded)

Motivation

With MambaRadixCache enabled, requests sharing a common prefix can
reuse cached SSM states instead of recomputing from scratch. In
testing with Kimi-Linear-48B-A3B-Instruct (TP=2,
--mamba-scheduler-strategy extra_buffer):

  • Cold request: 401 prompt tokens, 0 cached
  • Subsequent requests with shared prefix: 384 tokens cached, only
    16-17 new tokens need computation

This provides significant TTFT reduction for workloads with shared
system prompts or repeated prefixes.

Why it wasn't enabled before

KDA's forward_decode and forward_extend were missing the state
tracking calls that GDN already has. Additionally,
chunk_kda_fwd discarded the h tensor (intermediate SSM states
at chunk boundaries) via del h, making extend-time tracking
impossible.

The implementation is straightforward since:

  • The base class MambaAttnBackendBase already provides
    _init_track_conv_indices, _init_track_ssm_indices,
    _track_mamba_state_decode, and _track_mamba_state_extend
  • chunk_gated_delta_rule_fwd_h (shared with GDN) already
    computes h — KDA just needs to stop deleting it
  • No Triton kernel changes required

Key difference from GDN

KDA stores conv states with transposed layout: (num_slots, conv_width, qkv_dim) vs GDN's (num_slots, qkv_dim, conv_width).
The implementation accounts for this via:

  • Swapping conv_states_shape last two dims so
    _init_track_conv_indices sees correct conv_state_len
  • Using .permute(1, 2, 0) instead of .transpose(0, 1) when
    writing tracked conv states

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions