[Bug]: DeepSeek-V4 MTP speculative decoding over-truncates prefix cache hit from 32K to 16K

[LL1024LL]

Your current environment

The output of `python collect_env.py`

Please paste the output of:

wget https://raw.githubusercontent.com/vllm-project/vllm-ascend/main/collect_env.py
python collect_env.py

Image used in our test:
ascend/vllm-ascend:deepseekv4-a3

🐛 Describe the bug

When serving DeepSeek-V4-Flash with prefix caching enabled, enabling MTP speculative decoding significantly reduces the reported local prefix cache hit length.

For the same repeated long prompt, without MTP the prefix cache can hit 32768 tokens, but with MTP enabled it drops to 16384 tokens.

This causes a large TTFT regression for long-context repeated prompts.

The behavior seems related to the interaction between:

1. DeepSeek-V4 hybrid KV cache layout
2. HybridKVCacheCoordinator.lcm_block_size, which appears to be 16384 tokens in this case
3. EAGLE/MTP handling that drops the last matched block
4. The following LCM alignment logic, which rounds the cache hit down to the previous 16K boundary

In our case, a potential 32768-token cache hit becomes slightly smaller after the EAGLE/MTP last-block drop, and is then aligned down to 16384.

Reproduction

Serve DeepSeek-V4-Flash with prefix caching and MTP enabled:

vllm serve <model_path> \
  --enable-prefix-caching \
  --max_model_len 1024000 \
  --max-num-batched-tokens 8192 \
  --served-model-name DeepSeek-V4-Flash-Agent \
  --gpu-memory-utilization 0.9 \
  --api-server-count 1 \
  --max-num-seqs 16 \
  --data-parallel-size 2 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --tokenizer-mode deepseek_v4 \
  --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice \
  --reasoning-parser deepseek_v4 \
  --safetensors-load-strategy 'prefetch' \
  --quantization ascend \
  --speculative-config '{"num_speculative_tokens": 1,"method": "mtp"}' \
  --port 9000 \
  --block-size 16 \
  --enable-prompt-tokens-details \
  --enable-force-include-usage \
  --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
  --async-scheduling \
  --additional-config '{"ascend_compilation_config":{"enable_npugraph_ex":true,"enable_static_kernel":false},"enable_cpu_binding": "true","multistream_overlap_shared_expert":false,"multistream_dsa_preprocess":false}'

Then send the same long prompt twice. In our test, the prompt length is around 46240 tokens.

Observed usage:

{
  "prompt_tokens": 46240,
  "completion_tokens": 500,
  "total_tokens": 46740,
  "prompt_tokens_details": {
    "cached_tokens": 16384
  }
}

We also tested:

• --max-num-batched-tokens=16384: still only 16384 cached tokens with MTP enabled
• --max-num-batched-tokens=65536: OOM in our deployment
• --block-size=16: does not improve the effective cached token granularity
• Same request sent multiple times: still 16384 cached tokens with MTP enabled

When MTP and async scheduling are disabled, the same repeated 46K prompt can reach:

"prompt_tokens_details": {
  "cached_tokens": 32768
}

After re-enabling MTP, it drops back to:

"prompt_tokens_details": {
  "cached_tokens": 16384
}

So the main trigger appears to be MTP speculative decoding.

Expected behavior

For a repeated prompt of around 46240 tokens, given DeepSeek-V4's current 16K-aligned hybrid prefix cache behavior, the expected local prefix cache hit should be:

floor(46240 / 16384) * 16384 = 32768

Enabling MTP may need to recompute a small tail region for hidden states, but it should ideally not reduce a 32768-token prefix cache hit all the way down to 16384.

Expected behavior:

• Without MTP: cached_tokens = 32768
• With MTP: cached_tokens should remain close to 32768, or at least not lose a full 16K segment

Actual behavior

With MTP enabled:

cached_tokens = 16384

Without MTP:

cached_tokens = 32768

This means enabling MTP causes an extra 16384 tokens of prompt recomputation.

For long prompts and moderate completion lengths, this can make MTP slower overall despite improving decode speed.

Root cause hypothesis

Based on source code inspection, the issue seems to be caused by the interaction between MTP/EAGLE prefix-cache handling and DeepSeek-V4 hybrid KV cache alignment.

Relevant code paths:

1. vLLM-Ascend scheduler reports local prefix cache result

In vLLM-Ascend scheduler, cached tokens are obtained from:

self.kv_cache_manager.get_computed_blocks(request)

and then written into prefill stats, which becomes:

usage.prompt_tokens_details.cached_tokens

So this is local prefix cache behavior, not only external KV transfer / Mooncake behavior.

2. DeepSeek-V4 uses HybridKVCacheCoordinator

In upstream vLLM:

vllm/v1/core/kv_cache_coordinator.py

HybridKVCacheCoordinator computes:

self.lcm_block_size = lcm(*block_sizes)

and cache hit length is aligned to this LCM because partial block cache hit is not supported.

For DeepSeek-V4 in our deployment, the effective LCM alignment appears to be 16384 tokens.

3. MTP/EAGLE marks the DeepSeek-V4 MTP group

In:

vllm/v1/core/kv_cache_utils.py

DeepSeek-V4's MTP attention layer is marked as an EAGLE group:

# DeepseekV4's MTP attention layer is always the last layer,
# and we flag whichever group contains it.
last_layer = next(reversed(kv_cache_spec))
for group in kv_cache_groups:
    if last_layer in group.layer_names:
        group.is_eagle_group = True
        break

4. EAGLE/MTP drops the last matched block

In:

vllm/v1/core/single_type_kv_cache_manager.py

FullAttentionManager.find_longest_cache_hit() contains logic similar to:

if use_eagle and computed_blocks[0]:
    # Need to drop the last matched block if eagle is enabled.
    for computed in computed_blocks:
        computed.pop()

Then it applies alignment:

while (
    block_size != alignment_tokens
    and len(computed_blocks[0]) * block_size % alignment_tokens != 0
):
    for computed in computed_blocks:
        computed.pop()

This means:

1. Without MTP:

   • Longest hit can be 32768 tokens

2. With MTP:

   • One matched block is popped
   • Hit length becomes slightly less than 32768
   • Then LCM alignment rounds it down to the previous 16384 boundary
   • Final hit becomes 16384

So a small MTP/EAGLE recomputation requirement is amplified into losing a full 16K prefix-cache segment.

Impact

This significantly hurts TTFT for long prompts with high prefix reuse.

Example:

prompt_tokens = 46240

without MTP:
cached_tokens = 32768
tokens to recompute = 46240 - 32768 = 13472

with MTP:
cached_tokens = 16384
tokens to recompute = 46240 - 16384 = 29856

Enabling MTP causes about 16384 extra prompt tokens to be recomputed.

For workloads with long prompts and moderate output length, the extra prefill cost can outweigh MTP's decode speed benefit.

Suggested fix direction

Possible directions:

1. Decouple MTP/EAGLE hidden-state recomputation from the global prefix-cache hit length.

   MTP may need to recompute the last block or tail hidden states, but the main model's prefix cache hit should not necessarily be truncated by a full DeepSeek-V4 hybrid LCM segment.

2. Make the EAGLE/MTP drop logic alignment-aware.

   If dropping one block would cause the hit length to cross a large LCM boundary, consider preserving the main cache hit and only recomputing the required MTP tail.

3. Support per-group computed token lengths or partial hybrid prefix cache hits.

   The current HybridKVCacheCoordinator uses one global LCM-aligned hit length for all groups. This is conservative and causes large granularity loss for DeepSeek-V4.

4. Add DeepSeek-V4 specific handling for MTP + hybrid KV cache.

   Since DeepSeek-V4 has a special hybrid KV layout and an MTP attention layer, it may need special handling to avoid over-truncating prefix cache hits.

Related issue

This may be related to the DeepSeek-V4 hybrid KV cache discussion in:

• [RFC]: AscendStore KV Cache Pool supports DeepseekV4 #9166

However, this issue is specifically about local prefix caching with MTP enabled, not only AscendStore / external KV Pool transfer.

[ZT-AIA]

@lilinsiman Plz see this mtp+prefix chached issue

[lilinsiman]

Please send the error log.

[Jubileus-ryz]

https://github.com/vllm-project/vllm/blob/main/vllm/v1/core/kv_cache_manager.py#L219
I noticed that this already ensures the last KV cache block is recomputed. However, could this lead to more than one final KV cache block being recomputed?