Restore fine-grained hybrid prefix-cache reuse under EAGLE/MTP

[randomvariable]

Objective

Verify and, if necessary, fix fine-grained prefix-cache reuse under EAGLE/MTP for hybrid attention plus Mamba models.

Target workloads

• Qwen hybrid attention/Mamba or Gated-DeltaNet models using EAGLE/MTP
• Multi-turn prompts with partial prefix-cache hits

References

• [2/N][Core] support partial prefix cache hit for hybrid model vllm-project/vllm#46384
• https://github.com/eugr/spark-vllm-docker/tree/main/mods/fix-eagle-fine-prefix

Context

The fork already has prefix_match_unit, finer hybrid block hashing, and scheduler handling for a Mamba partial-tail boundary. The external patch may therefore be obsolete, partially overlapping, or expose one remaining predecessor-hash case. Reproduce before porting.

Proposed work

• Build a focused multi-turn EAGLE/MTP regression test where a context-load probe occupies the final hash unit.
• Measure FullAttention and Mamba computed-token rewind boundaries.
• Compare current fork behavior with the external predecessor-hash retention heuristic.
• Apply only the missing bounded metadata change, if a regression remains.

Acceptance criteria

• Focused tests specify token geometry and assert exact FullAttention match, EAGLE rewind, and Mamba replay boundaries.
• No stale/reused Mamba state crosses an invalid boundary.
• Metadata remains bounded per prompt and cache geometry is unchanged.
• Report cached follow-up TTFT medians over a defined prompt pair and repetition count versus unpatched behavior; do not gate adoption on an unspecified percentage.

Target hardware

• CUDA sm_121a (GB10 / DGX Spark)
• ROCm gfx1151 (Strix Halo) — both targets run EAGLE/MTP and hybrid attention

[randomvariable]

Current-model applicability

Primary current architecture candidate: Qwen3.6 Gated DeltaNet/global GQA with native MTP. Nemotron 3 Ultra is a secondary candidate because it combines Mamba-2/attention and MTP, but only after current vLLM hybrid-cache and drafter support is established.

Use “Gated DeltaNet,” not generic Qwen/Mamba terminology. Dense or attention-only speculative targets such as Mistral Medium 3.5 EAGLE and Gemma 4 external MTP lack this issue's recurrent hybrid-cache condition. Kimi K3, LFM2.5, Inkling, and Nemotron Nano expose different recurrent/convolutional state contracts and are not drop-in coverage without dedicated adapters.

[randomvariable]

Additional hybrid speculative candidate

Step-3.7-Flash combines mixed full/SWA attention with three published NEXTN layers, making it a concrete current architecture candidate for fine-grained speculative prefix reuse. Model/processor loading, drafter wiring, cache ABI, and graph support remain separate gates.

[randomvariable]

Status: partial.

7c813934b adds EAGLE hybrid partial-prefix regression coverage in tests/v1/core/prefix_cache/test_partial_prefix_cache_hits.py, including the group-block bound. It does not prove exact rewind safety for Mamba/recurrent state, bound metadata growth, or measure cached-follow-up TTFT. Next gate: add an exact-rewind stale-state oracle plus bounded-metadata assertion, then report TTFT for a cached follow-up against a non-reused baseline.

[randomvariable]

Implemented focused hybrid Mamba prefix-rewind regression coverage in ae8e571 (test: cover hybrid Mamba prefix rewind).

Validation: 19 Mamba utility tests pass. Coverage targets prefix rewind behavior.

Remaining qualification gaps: fused GPU/EAGLE-MTP integration, bounded repeated-reuse metadata, and cached-follow-up TTFT. Keeping issue open until these are covered.

[randomvariable]

Test coverage landed in ef1b25e for hybrid MTP prefix-cache reuse. Ruff, py_compile, and pytest collection passed. Native Qwen3.5 run with HF_HUB_OFFLINE=1 is blocked before model load because Qwen/Qwen3.5-0.8B-Base is not fully cached. Native MiMo MTP passed separately. Remaining gap: native Qwen3.5 TTFT evidence requires model availability.