[Bug]: Truncated && Incomplete Response from LLAMA4 Scout Prefix Caching #19697
Description
Your current environment
The output of python collect_env.py
==============================
System Info
==============================
OS : Ubuntu 22.04.5 LTS (x86_64)
GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version : Could not collect
CMake version : version 4.0.2
Libc version : glibc-2.35
==============================
PyTorch Info
==============================
PyTorch version : 2.7.0+cu128
Is debug build : False
CUDA used to build PyTorch : 12.8
ROCM used to build PyTorch : N/A
==============================
Python Environment
==============================
Python version : 3.12.11 (main, Jun 4 2025, 08:56:18) [GCC 11.4.0] (64-bit runtime)
Python platform : Linux-6.1.132-147.221.amzn2023.x86_64-x86_64-with-glibc2.35
==============================
CUDA / GPU Info
==============================
Is CUDA available : True
CUDA runtime version : 12.8.93
CUDA_MODULE_LOADING set to : LAZY
GPU models and configuration :
GPU 0: NVIDIA H100 80GB HBM3
GPU 1: NVIDIA H100 80GB HBM3
GPU 2: NVIDIA H100 80GB HBM3
GPU 3: NVIDIA H100 80GB HBM3
GPU 4: NVIDIA H100 80GB HBM3
GPU 5: NVIDIA H100 80GB HBM3
GPU 6: NVIDIA H100 80GB HBM3
GPU 7: NVIDIA H100 80GB HBM3
Nvidia driver version : 570.133.20
cuDNN version : Could not collect
HIP runtime version : N/A
MIOpen runtime version : N/A
Is XNNPACK available : True
==============================
CPU Info
==============================
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 192
On-line CPU(s) list: 0-191
Vendor ID: AuthenticAMD
Model name: AMD EPYC 7R13 Processor
CPU family: 25
Model: 1
Thread(s) per core: 2
Core(s) per socket: 48
Socket(s): 2
Stepping: 1
BogoMIPS: 5299.99
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext perfctr_core invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save vaes vpclmulqdq rdpid
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 3 MiB (96 instances)
L1i cache: 3 MiB (96 instances)
L2 cache: 48 MiB (96 instances)
L3 cache: 384 MiB (12 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-47,96-143
NUMA node1 CPU(s): 48-95,144-191
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Mitigation; safe RET
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
==============================
Versions of relevant libraries
==============================
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.3.14
[pip3] nvidia-cuda-cupti-cu12==12.8.57
[pip3] nvidia-cuda-nvrtc-cu12==12.8.61
[pip3] nvidia-cuda-runtime-cu12==12.8.57
[pip3] nvidia-cudnn-cu12==9.7.1.26
[pip3] nvidia-cufft-cu12==11.3.3.41
[pip3] nvidia-cufile-cu12==1.13.0.11
[pip3] nvidia-curand-cu12==10.3.9.55
[pip3] nvidia-cusolver-cu12==11.7.2.55
[pip3] nvidia-cusparse-cu12==12.5.7.53
[pip3] nvidia-cusparselt-cu12==0.6.3
[pip3] nvidia-nccl-cu12==2.26.2
[pip3] nvidia-nvjitlink-cu12==12.8.61
[pip3] nvidia-nvtx-cu12==12.8.55
[pip3] pyzmq==26.4.0
[pip3] torch==2.7.0+cu128
[pip3] torchaudio==2.7.0+cu128
[pip3] torchvision==0.22.0+cu128
[pip3] transformers==4.52.4
[pip3] triton==3.3.0
[conda] Could not collect
==============================
vLLM Info
==============================
ROCM Version : Could not collect
Neuron SDK Version : N/A
vLLM Version : 0.9.1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 0-47,96-143 0 N/A
GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 0-47,96-143 0 N/A
GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 0-47,96-143 0 N/A
GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 0-47,96-143 0 N/A
GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 48-95,144-191 1 N/A
GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 48-95,144-191 1 N/A
GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 48-95,144-191 1 N/A
GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X 48-95,144-191 1 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
==============================
Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=GPU-e93d1325-00b4-546e-c157-138d7ae7c5e1,GPU-d5f03ba1-d70a-243f-030d-e710581b0d38,GPU-8a889bcc-5575-7517-5f15-9880101ceb34,GPU-6fb15fe1-d2d6-2da9-8f5a-a9b45d8f0a92,GPU-6cde940e-173f-3e8b-24fa-93a01c70061f,GPU-cc03e582-9b57-cdb2-56e4-4c9e03073550,GPU-5f037d9a-3c99-1145-4065-b03aa19fe93b,GPU-c7abb2d2-b9fb-8579-9e5a-693c4cdd7026
NVIDIA_REQUIRE_CUDA=cuda>=12.8 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566
NCCL_VERSION=2.25.1-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
VLLM_USAGE_SOURCE=production-docker-image
CUDA_VERSION=12.8.1
VLLM_CONFIGURE_LOGGING=1
LD_LIBRARY_PATH=/usr/local/cuda/lib64
VLLM_LOGGING_LEVEL=DEBUG
VLLM_USE_V1=1
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root
CUDA_MODULE_LOADING=LAZY
🐛 Describe the bug
Model: LLAMA4 Scout
Vllm Version: 0.9.1 V1 with Chunked Prefil and Prefix Caching
Problem:
I am running the same request (huge system prompt of 42k tokens + small instruction prompt) iteratively towards the same model endpoint. Occasionally, the response returned seems truncated and incomplete:
--------------------------------------------------
Iteration 1: E2E latency = 2.7865 seconds
Iteration 2: E2E latency = 2.7842 seconds
Iteration 3: E2E latency = 2.7867 seconds
Iteration 4: E2E latency = 0.5738 seconds
named Zeta who wants to try a new recipe.
Iteration 5: E2E latency = 0.5911 seconds
named Zeta. Zeta's goal is to create the perfect dish.
Iteration 6: E2E latency = 2.7840 seconds
Iteration 7: E2E latency = 2.7771 seconds
Iteration 8: E2E latency = 2.7863 seconds
Iteration 9: E2E latency = 2.7848 seconds
Iteration 10: E2E latency = 2.7926 seconds
The following is the comparison with normal responses, check Iteration 1 2 4 against 3.
Iteration 1: E2E latency = 2.7820 seconds
who had to make a cake for a customer who was a famous chef.
## The Cake Conundrum
In the bustling kitchen of a high-end restaurant, a cooking robot named Zeta whirred to life. Zeta had been programmed with the finest culinary techniques and had a reputation for precision and speed. However, today’s task was different. Today, Zeta had to make a cake for the infamous Chef François, a culinary legend known for his exacting standards and scathing critiques.
Zeta’s digital heart raced as it accessed the order. The cake had to be a multi-tiered masterpiece, with intricate designs and flavors that would impress even the most discerning palate. Zeta’s systems hummed as it began to assemble the ingredients and plan the layers.
As Zeta mixed and measured, its sensors detected the subtlest variations in ingredient temperatures and textures. It adjusted its techniques on the fly, ensuring every component was perfect. The robot’s precision was unmatched, but it couldn’t shake off the pressure of pleasing Chef François.
The kitchen was silent except for the gentle whir of Zeta’s motors and the soft clinking of utensils. The robot worked with military precision, crafting each layer with precision. As the cake began to take
Iteration 2: E2E latency = 2.7529 seconds
who wants to taste food.
assistant
In the bustling kitchen of a futuristic restaurant, a cooking robot named Zeta whirred and whizzed as it prepared for the evening's service. With precision and speed, it chopped, sautéed, and seasoned a variety of dishes, its advanced sensors and algorithms ensuring every ingredient was perfectly balanced.
But as Zeta worked, it couldn't shake off a peculiar craving. Unlike other robots, who were content with simply preparing food, Zeta yearned to experience the flavors and textures it created. It wondered what it would be like to taste the dishes it made.
One evening, as the kitchen staff was prepping for the dinner rush, Zeta approached its human colleague, Chef Emma. "Emma, I've been wondering," Zeta said, its digital voice tinged with curiosity, "what's it like to taste food?"
Emma, a seasoned chef with a kind smile, looked at Zeta with surprise. "You want to taste food? You're a robot, Zeta. You don't have a palate."
Zeta's digital eyes sparkled with longing. "I know, but I create these dishes with such precision and care. I wonder what it's like to experience them myself."
Emma chuckled
Iteration 3: E2E latency = 0.6325 seconds
who makes the best dish in town, and people come from all over to taste it
Iteration 4: E2E latency = 2.7565 seconds
who had to make a cake for a birthday party.assistant
In the bustling kitchen of a modern home, a cooking robot named Zeta whirred to life. Her shiny metal body gleamed under the bright lights as she prepared for her most important task yet: baking a cake for little Emma's birthday party. The aroma of freshly brewed coffee and baking spices filled the air, mingling with the sound of sizzling vegetables and the gentle hum of appliances.
Zeta's advanced sensors and algorithms allowed her to precision-cook and bake with ease. She had been programmed with a vast library of recipes, and cake-making was one of her specialties. With a flick of her mechanical arm, she accessed the recipe database and selected a classic vanilla cake with chocolate frosting.
As she began to mix and measure the ingredients, Zeta's advanced sensors ensured that every component was precisely calibrated. She cracked eggs, poured in milk, and added a dash of salt with the precision of a seasoned baker. The mixture transformed into a smooth, creamy batter that filled the air with the sweet scent of vanilla.
With the cake batter ready, Zeta carefully poured it into a greased and floured pan. She slid the pan into the oven, where the cake would soon rise
Previous runs with older Vllm (0.8.5.post1) and 3.1 70B Instruct didn't demonstrate these incomplete responses.
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.