[Bug]: MTP speculative decoding produces corrupted output at concurrency >= 4 (V1 engine)

[dkremez]

Your current environment

The output of python collect_env.py

==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0
Clang version                : Could not collect
CMake version                : version 3.22.1
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.9.1+cu129
Is debug build               : False
CUDA used to build PyTorch   : 12.9
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.12 (main, Oct 10 2025, 08:52:57) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-6.1.159-181.297.amzn2023.x86_64-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.9.86
CUDA_MODULE_LOADING set to   : 
GPU models and configuration : 
GPU 0: NVIDIA H200
GPU 1: NVIDIA H200
GPU 2: NVIDIA H200
GPU 3: NVIDIA H200

Nvidia driver version        : 580.126.09
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           46 bits physical, 48 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  192
On-line CPU(s) list:                     0-191
Vendor ID:                               GenuineIntel
Model name:                              Intel(R) Xeon(R) Platinum 8488C
CPU family:                              6
Model:                                   143
Thread(s) per core:                      2
Core(s) per socket:                      48
Socket(s):                               2
Stepping:                                8
BogoMIPS:                                4800.00
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd ida arat avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid cldemote movdiri movdir64b md_clear serialize amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
Hypervisor vendor:                       KVM
Virtualization type:                     full
L1d cache:                               4.5 MiB (96 instances)
L1i cache:                               3 MiB (96 instances)
L2 cache:                                192 MiB (96 instances)
L3 cache:                                210 MiB (2 instances)
NUMA node(s):                            2
NUMA node0 CPU(s):                       0-47,96-143
NUMA node1 CPU(s):                       48-95,144-191
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.1
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.9.1.4
[pip3] nvidia-cuda-cupti-cu12==12.9.79
[pip3] nvidia-cuda-nvrtc-cu12==12.9.86
[pip3] nvidia-cuda-runtime-cu12==12.9.79
[pip3] nvidia-cudnn-cu12==9.10.2.21
[pip3] nvidia-cudnn-frontend==1.18.0
[pip3] nvidia-cufft-cu12==11.4.1.4
[pip3] nvidia-cufile-cu12==1.14.1.1
[pip3] nvidia-curand-cu12==10.3.10.19
[pip3] nvidia-cusolver-cu12==11.7.5.82
[pip3] nvidia-cusparse-cu12==12.5.10.65
[pip3] nvidia-cusparselt-cu12==0.7.1
[pip3] nvidia-cutlass-dsl==4.3.5
[pip3] nvidia-ml-py==13.590.48
[pip3] nvidia-nccl-cu12==2.27.5
[pip3] nvidia-nvjitlink-cu12==12.9.86
[pip3] nvidia-nvshmem-cu12==3.3.20
[pip3] nvidia-nvtx-cu12==12.9.79
[pip3] pyzmq==27.1.0
[pip3] torch==2.9.1+cu129
[pip3] torchaudio==2.9.1+cu129
[pip3] torchvision==0.24.1+cu129
[pip3] transformers==4.57.6
[pip3] triton==3.5.1
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.15.1
vLLM Build Flags:
  CUDA Archs: 7.0 7.5 8.0 8.9 9.0 10.0 12.0; ROCm: Disabled
GPU Topology:
  	GPU0	GPU1	GPU2	GPU3	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NV18	NV18	NV18	0-47,96-143	0		N/A
GPU1	NV18	 X 	NV18	NV18	0-47,96-143	0		N/A
GPU2	NV18	NV18	 X 	NV18	0-47,96-143	0		N/A
GPU3	NV18	NV18	NV18	 X 	0-47,96-143	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=GPU-f01ae316-426e-0268-5ab7-17088fc82336,GPU-14a2aa9f-6e9c-f0cc-2d81-8a0395a707a6,GPU-84d5c4e8-aee3-9063-a74a-4b7762d3aebc,GPU-442240d6-9a81-c0e8-0978-1d2a4863f307
NVIDIA_REQUIRE_CUDA=cuda>=12.9 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566 brand=unknown,driver>=570,driver<571 brand=grid,driver>=570,driver<571 brand=tesla,driver>=570,driver<571 brand=nvidia,driver>=570,driver<571 brand=quadro,driver>=570,driver<571 brand=quadrortx,driver>=570,driver<571 brand=nvidiartx,driver>=570,driver<571 brand=vapps,driver>=570,driver<571 brand=vpc,driver>=570,driver<571 brand=vcs,driver>=570,driver<571 brand=vws,driver>=570,driver<571 brand=cloudgaming,driver>=570,driver<571
TORCH_CUDA_ARCH_LIST=7.0 7.5 8.0 8.9 9.0 10.0 12.0
NVIDIA_DRIVER_CAPABILITIES=compute,utility
VLLM_USAGE_SOURCE=production-docker-image
CUDA_VERSION=12.9.1
LD_LIBRARY_PATH=/opt/amazon/ofi-nccl/lib:/opt/amazon/openmpi/lib:/opt/amazon/efa/lib:/usr/local/cuda/lib64:/usr/local/nvidia/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib64
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1

🐛 Describe the bug

Model

Large MoE model (80B total / 3B active params, Qwen3-based architecture)
Qwen3 Next 80B instruct

How would you like to use vllm

Serving via the OpenAI-compatible API with MTP speculative decoding.

Description

When serving a model with MTP speculative decoding (num_speculative_tokens=2) and 4+ concurrent requests land in the decode batch simultaneously, a CUDA illegal memory access corrupts the model's internal state. This produces completely garbage output — not verbose-but-valid text, but corrupted token sequences that loop until max_tokens is hit.

The configured stop token is never matched because the output is unintelligible noise. This is 100% reproducible.

Speculative decoding config

{"method": "mtp", "num_speculative_tokens": 2}

Normal output (concurrency=1, MTP working correctly)

The model produces a coherent, well-structured ~93-token answer.
finish_reason=stop, stop token matched correctly.

Corrupted output (concurrency=4, bug triggered)

Example 1: Garbage then single-token repetition loop

<!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! ** 	** <<

 the,<!** the d- the **< the<
<...<...<<!
...	**
 2<...1 **... n the the<  the ****<** the<...<... the, < A<...n<<<,2 thes< the s**2 **22.2
	202222 about 2022222222022222120202 202222222222220222
  I also
    (I)
  Some
    (I)
        I
    I
  I
    I
        This
        I
        I
        I
        [... "I" repeated ~500 more times until max_tokens ...]

• 2000 tokens, finish_reason=length, stop_reason=None

Example 2: Garbage then year-number loop

<!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! **
 the **1\1 **
 **1<1,https#** ** n0.s . 2 .1 .整洁.s s> suggested> (black/now.com **>!<this>etc/ast Bill **
# 1997-1997-1996-1996-1996-1996-1996-1996-#1997-1996-1996-1996-1996-1996-1996-
1996-1996-1996-1996-1996-1996-1996-1996-1996-1996-1996-1996-1996-1996-1996-
[... "-1996" repeated ~300 more times until max_tokens ...]

• 2000 tokens, finish_reason=length, stop_reason=None

Example 3: Garbage then flag-like pattern loop

<!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

...**d <package <mg> - 1 -n,ñ,ñ(,` -x - -  ft  - - -x -1 -x -x -_ -x -x -_ -x
-_ -y -1 -_ -y n -1 -x -11 -_ -y -1 -_ -y n -1 -_ -y n -1 -_ -y -1 -_ -y -1
[... "-_ -y -1" repeated ~400 more times until max_tokens ...]

• 2000 tokens, finish_reason=length, stop_reason=None

Common corruption pattern

All corrupted outputs follow the same three-phase structure:

1. Garbage header — starts with <!!!!!!!!... (corrupted first token), followed by random fragments
2. Transition noise — brief incoherent text with random words, symbols, numbers, occasionally recognizable fragments
3. Repetition loop — locks into a short repeating token pattern (e.g. I, -1996, -2019, -_ -y -1) and generates this until max_tokens is hit

The stop token never appears because the output is completely corrupted — the model is not generating valid text.

Benchmark statistics

We ran a load test with varying concurrency levels on the same model and dataset:

Metric	c=1 (healthy)	c=2 (healthy)	c=4 (broken)
Mean output tokens	253	261	1883
P50 output tokens	250	255	2000
P95 output tokens	382	382	2000
Max output tokens	453	551	2000
Requests hitting max_tokens	0%	0%	88%

For comparison, SGLang with MTP on the exact same model and dataset at c=4: mean=253, max=583, 0% hitting max_tokens. The issue is specific to vLLM.

Root cause analysis

The crash originates in the V1 speculative decoding path when the decode batch contains 4+ sequences. We observe a CUDA illegal memory access in gpu_model_runner.py (_update_states_after_model_execute -> .cpu()) during the sample_tokens phase. With streaming enabled, the corruption manifests as garbage output rather than a hard crash.

Related issues

• [Bug]: [V1][SpecDec] RuntimeError: CUDA error: an illegal memory access was encountered #13673 — similar CUDA illegal memory access was fixed for ngram speculative decoding, but the fix does not cover the MTP path.

Reproduction

# Minimal reproduction: send 4+ concurrent requests to a vLLM server
# with MTP speculative decoding enabled (num_speculative_tokens=2)

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url="http://<vllm-server>/v1", api_key="unused")

async def send_request():
    response = await client.completions.create(
        model="<model-name>",
        prompt="<any reasonable prompt>",
        max_tokens=2000,
        stop=["</stop_token>"],
        extra_body={"skip_special_tokens": False},
    )
    print(f"tokens={response.usage.completion_tokens}, "
          f"finish={response.choices[0].finish_reason}")
    # At c>=4, expect finish_reason=length and ~2000 tokens of garbage

async def main():
    # c=1-3: all requests produce clean output
    # c=4+: 50-100% of requests produce corrupted garbage
    tasks = [send_request() for _ in range(4)]
    await asyncio.gather(*tasks)

asyncio.run(main())

100% reproducible. At c=1-3, all requests produce clean output. At c>=4, 50-100% of requests produce corrupted output.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[Saad-Mallebhari]

@dkremez
The corruption pattern (garbage header → repetition loop → max_tokens) is consistent with KV cache state getting corrupted mid-batch when _update_states_after_model_execute triggers a .cpu() transfer while 4+ sequences are in the decode batch simultaneously. The MTP path likely has an off-by-one or incorrect tensor slicing when num_speculative_tokens=2 and batch size crosses the threshold.

A few things that might help narrow it down while waiting for a fix:

1. Try num_speculative_tokens=1 , if corruption disappears, the bug is in the multi-token speculation logic specifically
2. Try --disable-async-output-proc , if that helps, the issue may be in async state updates racing with the decode batch
3. As a temporary workaround, limiting concurrency to 3 via --max-num-seqs 3 avoids the trigger threshold entirely

The fix from #13673 for ngram speculative decoding would be worth examining to see if the same patch can be extended to cover the MTP path in gpu_model_runner.py.

[bwinken]

Adding a data point that strongly supports this being a
mixed-batch indexing issue rather than a state corruption issue:

Setup: Qwen3-Next-27B-FP8, TP=2 on 2x H100 PCIe, vLLM 0.19.x

• speculative_config: mtp, num_speculative_tokens=2
• enable_chunked_prefill: true
• avg input 64K tokens, avg output 1.5K tokens

Observation: break point scales inversely with max_num_batched_tokens:

max_num_batched_tokens	Break point
32768	c=3-4
16384	c=6-8

Acceptance at break point drops to ~0% on P0 while P1 conditional stays ~90%, suggesting the drafter's hidden state pickup is wrong specifically when prefill chunks share a step with decode sequences. A separate container running the same model on shorter inputs (~2-5K) stays stable at c=6+, consistent with mixed batches being the trigger.

[vektorprime]

No matter what quant or model I use for qwen3.6 27B I'm finding that it eventually outputs garbage or loops when MTP is on. This even occurs with the BF16 (native) image. MTP off works great. I can't seem to trace it down.

[kettleofketchup]

any update on this?

[PavelPaha]

any update on this?

I observed a potentially related correctness issue.

With MTP speculative decoding and prefix caching enabled, a downstream consistency check rejected substantially more generated sequences because of log-probability mismatches. The rejection rate returned close to the non-speculative baseline after updating to a revision containing #46384 and #47782.

I cannot confirm that this is the same root cause as this issue, but the behavior appears consistent with an MTP + hybrid/Mamba prefix-cache correctness problem. It may be worth testing these changes against this reproducer.

[marzukia]

Confirmed. I hit this exact issue on a different hardware/software setup.

Environment:

• Hardware: Single NVIDIA RTX 5000 Pro 48GB
• vLLM Version: v0.26.0 (Docker image vllm/vllm-openai:v0.26.0)
• Model: Qwen/Qwen3.6-27B-FP8
• Spec Decode Config: {"method": "mtp", "num_speculative_tokens": 3}

Symptoms:
Consistent output duplication/corruption (responses repeating paragraphs or entire messages) while running with --max-num-seqs 4.

Workaround:
Reduced --max-num-seqs to 2. The duplication stopped immediately and output quality returned to normal.

This confirms the concurrency threshold behavior described in the issue.