Bug: MTP + Vision (multimodal) causes slot position corruption and OOM

[Scorp1o117]

Bug: MTP + Vision (multimodal) causes slot position corruption and OOM

Referencing PR #22673 (MTP Support)

Description

When using MTP (--spec-type mtp) together with vision/multimodal input, the server crashes with memory exhaustion due to infinite find_slot loops.

Steps to Reproduce

1. Build with MTP support (PR llama + spec: MTP Support  #22673)
2. Start server with: --spec-type mtp --spec-draft-n-max 3
3. Send an image for analysis with a multimodal model (e.g., Qwen3.6-35B-A3B + mmproj)
4. Server crashes with OOM

Error Log

srv  process_chun: processing image...
encoding image slice...
image slice encoded in 2780 ms
decoding image batch 1/2, n_tokens_batch = 2048
init: embeddings required but some input tokens were not marked as outputs -> overriding
find_slot: non-consecutive token position 33 after 32 for sequence 0 with 512 new tokens
find_slot: non-consecutive token position 33 after 33 for sequence 0 with 512 new tokens
(find_slot repeats indefinitely, consuming all VRAM and RAM)

Root Cause Analysis

The MTP hook (handle_mtp_for_ubatch) in process_ubatch fires after every ubatch computation. When processing vision embeddings, the token layout is non-consecutive (image tokens + text tokens with gaps). MTP's slot management doesn't account for this, causing:

1. Position validation failures (non-consecutive token position)
2. Infinite retry loop (512 new tokens each iteration)
3. Memory exhaustion

Expected Behavior

MTP speculative decoding should either:

• Skip draft generation for encoder/embedding batches
• Handle non-consecutive token positions from vision encoding gracefully

Environment

• GPU: NVIDIA RTX 5070 Ti (Blackwell architecture)
• CUDA: 13.2
• Model: Qwen3.6-35B-A3B with mmproj-BF16 vision projector
• Build branch: PippBauda/llama.cpp-turboquant-mtp (based on PR llama + spec: MTP Support  #22673)

Note

Works perfectly without --spec-type mtp. Vision/multimodal processing alone has no issues.

[calvarado2004]

I submitted a PR with a fix for this issue, MTP and image works fine now!

[Scorp1o117]

Confirmed working! The fix resolves the OOM crash when using --spec-type mtp with multimodal input. Cherry-picked the change from your PR (am17an#5) onto my turboquant-mtp fork — MTP + vision now works correctly.

Thanks for the quick fix @calvarado2004 🎉