Summary
KDA (Kimi Delta Attention) backend currently has
support_mamba_cache=False, which means KimiLinearForCausalLM
models cannot benefit from SSM state prefix caching via
MambaRadixCache. This PR enables it by:
- Setting
support_mamba_cache=True in server_args for
KimiLinearForCausalLM
- Adding the missing
_track_mamba_state_decode,
_track_mamba_state_extend, and init_forward_metadata
implementations in KDAAttnBackend
- Returning the intermediate
h tensor from chunk_kda_fwd
(already computed but previously discarded)
Motivation
With MambaRadixCache enabled, requests sharing a common prefix can
reuse cached SSM states instead of recomputing from scratch. In
testing with Kimi-Linear-48B-A3B-Instruct (TP=2,
--mamba-scheduler-strategy extra_buffer):
- Cold request: 401 prompt tokens, 0 cached
- Subsequent requests with shared prefix: 384 tokens cached, only
16-17 new tokens need computation
This provides significant TTFT reduction for workloads with shared
system prompts or repeated prefixes.
Why it wasn't enabled before
KDA's forward_decode and forward_extend were missing the state
tracking calls that GDN already has. Additionally,
chunk_kda_fwd discarded the h tensor (intermediate SSM states
at chunk boundaries) via del h, making extend-time tracking
impossible.
The implementation is straightforward since:
- The base class
MambaAttnBackendBase already provides
_init_track_conv_indices, _init_track_ssm_indices,
_track_mamba_state_decode, and _track_mamba_state_extend
chunk_gated_delta_rule_fwd_h (shared with GDN) already
computes h — KDA just needs to stop deleting it
- No Triton kernel changes required
Key difference from GDN
KDA stores conv states with transposed layout: (num_slots, conv_width, qkv_dim) vs GDN's (num_slots, qkv_dim, conv_width).
The implementation accounts for this via:
- Swapping
conv_states_shape last two dims so
_init_track_conv_indices sees correct conv_state_len
- Using
.permute(1, 2, 0) instead of .transpose(0, 1) when
writing tracked conv states
Summary
KDA (Kimi Delta Attention) backend currently has
support_mamba_cache=False, which means KimiLinearForCausalLMmodels cannot benefit from SSM state prefix caching via
MambaRadixCache. This PR enables it by:
support_mamba_cache=Truein server_args forKimiLinearForCausalLM
_track_mamba_state_decode,_track_mamba_state_extend, andinit_forward_metadataimplementations in KDAAttnBackend
htensor fromchunk_kda_fwd(already computed but previously discarded)
Motivation
With MambaRadixCache enabled, requests sharing a common prefix can
reuse cached SSM states instead of recomputing from scratch. In
testing with Kimi-Linear-48B-A3B-Instruct (TP=2,
--mamba-scheduler-strategy extra_buffer):16-17 new tokens need computation
This provides significant TTFT reduction for workloads with shared
system prompts or repeated prefixes.
Why it wasn't enabled before
KDA's
forward_decodeandforward_extendwere missing the statetracking calls that GDN already has. Additionally,
chunk_kda_fwddiscarded thehtensor (intermediate SSM statesat chunk boundaries) via
del h, making extend-time trackingimpossible.
The implementation is straightforward since:
MambaAttnBackendBasealready provides_init_track_conv_indices,_init_track_ssm_indices,_track_mamba_state_decode, and_track_mamba_state_extendchunk_gated_delta_rule_fwd_h(shared with GDN) alreadycomputes
h— KDA just needs to stop deleting itKey difference from GDN
KDA stores conv states with transposed layout:
(num_slots, conv_width, qkv_dim)vs GDN's(num_slots, qkv_dim, conv_width).The implementation accounts for this via:
conv_states_shapelast two dims so_init_track_conv_indicessees correctconv_state_len.permute(1, 2, 0)instead of.transpose(0, 1)whenwriting tracked conv states