**Body:**
```markdown
## Summary
When running Qwen3.6-27B-MTP (a model with Sliding Window Attention / hybrid recurrent memory) through llama.cpp with MTP speculative decoding enabled, the draft acceptance rate drops significantly (~35-37%) compared to standard attention models, despite MTP being designed for higher throughput. This appears to be caused by KV cache invalidation triggered by SWA/hybrid memory mechanisms, forcing full prompt re-processing and degrading MTP prediction accuracy.
## Environment
- **Model**: Qwen3.6-27B-MTP-GGUF (Q4_K_M quantization)
- **Backend**: llama.cpp Vulkan build (build-vulkan)
- **GPU**: AMD RX 7900 XTX (24GB VRAM)
- **CPU**: Intel LGA1200 platform
- **OS**: Windows 11
- **llama.cpp version**: Latest main branch (MTP recently merged)
## Observed Behavior
From server logs, the MTP draft acceptance rate is consistently low:
Task 310: draft acceptance rate = 0.36667 (44 accepted / 120 generated)
Task 354: draft acceptance rate = 0.83333 (20 accepted / 24 generated) <-- short context, works fine
Task 366: draft acceptance rate = 0.35360 (157 accepted / 444 generated)
Cumulative statistics show ~45% of generated draft tokens are discarded:
#gen drafts = 1446, #acc drafts = 799, #gen tokens = 4338, #acc tokens = 2397
The critical warning appears before low-acceptance tasks:
W slot update_slots: forcing full prompt re-processing due to lack of cache data
(likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
## Root Cause Analysis (Hypothesis)
1. Qwen3.6 uses Sliding Window Attention (SWA) / hybrid recurrent memory architecture
2. When the attention window slides, previously cached KV pairs become invalid
3. llama.cpp detects this and forces full prompt re-processing instead of partial cache reuse
4. Full re-processing disrupts MTP's prediction state, causing draft tokens to diverge from what the main model would generate
5. Result: ~65% of draft tokens are rejected and discarded, wasting compute
## Expected Behavior
MTP draft acceptance rate should be closer to 70-90% for models with compatible attention mechanisms, similar to what's observed with standard full-attention models. The cache invalidation logic for SWA models should preserve enough context for MTP to make accurate predictions.
## Suggested Investigation Areas
1. **Checkpoint restoration for SWA**: When `restored context checkpoint` is triggered, does the SWA window state get properly preserved? The log shows `pos_min = 16550, pos_max = 16550` being restored but then 8 tokens later it fails.
2. **MTP state synchronization**: After cache invalidation and full re-processing, is the MTP draft model's internal state resynchronized with the main model? If not, predictions will naturally diverge.
3. **Cache granularity for hybrid models**: Could the checkpoint strategy be adapted for SWA models to preserve more granular cache states, reducing the frequency of full re-processing events?
4. **Alternative: MTP-aware SWA handling**: When SWA triggers cache invalidation, could MTP be temporarily disabled or use a fallback strategy (e.g., fewer draft tokens) instead of generating predictions that will almost certainly be rejected?
## Impact
- **Performance**: ~65% of MTP compute is wasted on rejected drafts
- **Latency**: Full prompt re-processing adds 300-3000ms overhead per request
- **Throughput**: Despite MTP, effective tokens/second is only ~35 t/s instead of the potential 60-80+ t/s with proper cache utilization
## Reproduction Steps
1. Deploy Qwen3.6-27B-MTP-GGUF via llama.cpp server
2. Enable MTP speculative decoding (default behavior)
3. Send a multi-turn conversation with growing context
4. Observe server logs for `forcing full prompt re-processing` warnings and low `draft acceptance rate` values
## Additional Notes
This issue is particularly relevant as more models adopt SWA/hybrid memory architectures for efficiency. Qwen3.6 is just the first one I've tested — this will likely affect other hybrid-attention MTP models too.
Happy to provide more logs or test specific patches if helpful.
Title:
[Issue] Low MTP Draft Acceptance Rate with SWA/Hybrid Memory Models (Qwen3.6)