Speculative decoding with draft model: extreme slowdown on Vulkan (UMA iGPU) when both models on same device

[ElSnacko]

Speculative decoding with draft model: extreme slowdown on Vulkan (UMA iGPU) when both models on same device

Describe the bug

When using draft-model speculative decoding on the Vulkan backend with both models loaded on the same GPU, the draft model's per-token generation time increases by ~100,000x compared to running standalone. This makes speculative decoding completely ineffective — the target model with draft is no faster (or slower) than without.

The same hardware runs the draft model at 33-35 t/s when it's the only model loaded. When the target model is also loaded on the same Vulkan device, draft generation slows to effectively zero.

Hardware

Component	Detail
GPU	AMD Radeon 780M (gfx1103, RDNA3 iGPU, unified memory)
RAM	128 GB DDR5 (100 GB shared for GPU via bios_size)
Driver	Mesa RADV (open-source Vulkan)
Backend	Vulkan only (ROCm unsupported on gfx1103)
OS	Ubuntu 22.04, Docker

This is a unified memory architecture (UMA) — the GPU and CPU share the same physical RAM. There is no discrete VRAM.

Models

Role	Model	Quant	Size	Layers on GPU
Target	gemma-4-31B-it	Q6_K	23.5 GB	41/41 on Vulkan0
Draft	gemma-4-E2B-it	Q5_K_M	3.1 GB	26/26 on Vulkan0 (ngl=99)

Both models from the same family (same tokenizer, same architecture). Total GPU memory: ~30.3 GB model buffers on Vulkan0 + ~2.3 GB on Vulkan_Host. Well within the 100 GB shared RAM budget.

Build

llama-b9114-bin-ubuntu-vulkan-x64 (pre-built release from GitHub)

Steps to Reproduce

1. Draft model standalone (no target) — works great

llama-server \
  -m gemma-4-E2B-it-Q5_K_M.gguf \
  -ngl 99 -fa 1 -c 8192 \
  --host 0.0.0.0 --port 8083

Benchmarked via chat completions API (256 output tokens):

• 33-35 t/s across multiple prompts (code, freeform, JSON, summary)

2. Target model alone (no draft) — expected baseline

llama-server \
  -m gemma-4-31B-it-Q6_K.gguf \
  -ngl 99 -fa 1 -c 8192 \
  --host 0.0.0.0 --port 8080

• 3-7 t/s (dense 31B on DDR5 bandwidth — expected)

3. Target + draft model (speculative decoding) — broken

llama-server \
  -m gemma-4-31B-it-Q6_K.gguf \
  --spec-draft-model gemma-4-E2B-it-Q5_K_M.gguf \
  --spec-draft-ngl 99 \
  -ngl 99 -fa 1 -c 8192 \
  --host 0.0.0.0 --port 8080

• 3.2-4.9 t/s — no improvement over baseline, sometimes worse
• dur(g) for draft model evaluation: ~43,000-73,000 ms (should be ~30ms at 35 t/s)
• This is a ~100,000x slowdown in draft model generation

4. CPU draft fallback (ngl=0) — also broken

Same as #3 but with --spec-draft-ngl 0:

• 1.8 t/s — even worse than GPU draft
• Draft model running on CPU is somehow slower when the target model is on GPU

Expected Behavior

With a 35 t/s draft model and even a modest acceptance rate (50-75%), speculative decoding should yield significant speedup over the 3 t/s baseline. The math:

• Target alone: ~3 t/s
• Draft: 35 t/s, generating 5-16 draft tokens per verification
• Expected with 50% acceptance: ~6-12 t/s (2-4x speedup)

Instead, the draft model runs at ~0.014 t/s when the target model is loaded, making spec decoding counterproductive.

Analysis

Not a memory issue

• Total model buffers: ~30.3 GB on Vulkan0, well within 100 GB budget
• No OOM, no swapping, no mlock failures
• Both models load successfully and offload all layers to GPU

Not a context size issue

• Both models use 8192 context (explicitly set, verified no 256K leak)
• Draft model uses the same context size standalone vs paired

Likely root cause: Vulkan queue serialization

On UMA with Mesa/RADV, there appears to be a single Vulkan compute queue. When two models' compute graphs are submitted:

1. Draft model graph submission must wait for target model graph to complete
2. Target model verification must wait for draft model graph to complete
3. Each context switch between the two models' Vulkan pipelines flushes L2 cache and potentially invalidates the compute state

This serialization makes the draft model's forward pass wait for the entire target model forward pass to complete, destroying the pipelining that makes speculative decoding effective.

The fact that CPU draft (ngl=0) is also slow (1.8 t/s) when the target is on GPU suggests the bottleneck is in the synchronization/barrier between the two models, not in compute throughput.

Related issues

• Optimize Vulkan buffer transfers on UMA (Unified Memory Architecture) devices #22462 (closed/abandoned) — Vulkan UMA buffer transfer optimization. Attempted to remove unnecessary staging buffers for UMA, abandoned due to correctness issues. Suggests UMA Vulkan path has known overhead.
• Eval bug: NextN (PR #22673) Vulkan path produces ~0.7% accept rate + garbled output on RDNA3+RADV, while HIP/ROCm works correctly (81%) #23088 (closed) — MTP/NextN speculative decoding produces garbage output on Vulkan+RADV (correctness, not performance). Shows Vulkan speculative decoding path has known issues.
• llama: allow partial seq_rm for GDN models for speculative decoding #22400 (draft) — Partial seq_rm for speculative decoding not implemented for Vulkan. This is a dependency for MTP on Vulkan.
• llama + spec: MTP Support  #22673 (open) — MTP support PR. Author acknowledged Vulkan is not yet supported.

Proposed Investigation

1. Does the Vulkan backend support concurrent submission of multiple compute graphs, or does it serialize?
2. Would using separate Vulkan queues for draft vs target models help?
3. Is there unnecessary synchronization between draft and target model evaluations?
4. Does the UMA staging buffer overhead (Optimize Vulkan buffer transfers on UMA (Unified Memory Architecture) devices #22462) compound with the dual-model scenario?

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.