[Bug]: Low GPU Memory Utilization for Qwen-2.5-VL family of models

[thameem-abbas]

Your current environment

The output of python collect_env.py

(app-root) /opt/app-root$ vllm collect-env
/opt/app-root/lib64/python3.12/site-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
DEBUG 11-06 14:38:18 [plugins/__init__.py:28] No plugins for group vllm.platform_plugins found.
DEBUG 11-06 14:38:18 [platforms/__init__.py:34] Checking if TPU platform is available.
DEBUG 11-06 14:38:18 [platforms/__init__.py:52] TPU platform is not available because: No module named 'libtpu'
DEBUG 11-06 14:38:18 [platforms/__init__.py:58] Checking if CUDA platform is available.
DEBUG 11-06 14:38:18 [platforms/__init__.py:78] Confirmed CUDA platform is available.
DEBUG 11-06 14:38:18 [platforms/__init__.py:106] Checking if ROCm platform is available.
DEBUG 11-06 14:38:18 [platforms/__init__.py:120] ROCm platform is not available because: No module named 'amdsmi'
DEBUG 11-06 14:38:18 [platforms/__init__.py:127] Checking if XPU platform is available.
DEBUG 11-06 14:38:18 [platforms/__init__.py:146] XPU platform is not available because: No module named 'intel_extension_for_pytorch'
DEBUG 11-06 14:38:18 [platforms/__init__.py:153] Checking if CPU platform is available.
DEBUG 11-06 14:38:18 [platforms/__init__.py:58] Checking if CUDA platform is available.
DEBUG 11-06 14:38:18 [platforms/__init__.py:78] Confirmed CUDA platform is available.
INFO 11-06 14:38:18 [platforms/__init__.py:216] Automatically detected platform cuda.
DEBUG 11-06 14:38:22 [plugins/__init__.py:36] Available plugins for group vllm.general_plugins:
DEBUG 11-06 14:38:22 [plugins/__init__.py:38] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
DEBUG 11-06 14:38:22 [plugins/__init__.py:41] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
Collecting environment information...
==============================
        System Info
==============================
OS                           : Red Hat Enterprise Linux 9.6 (Plow) (x86_64)
GCC version                  : (GCC) 11.5.0 20240719 (Red Hat 11.5.0-5)
Clang version                : Could not collect
CMake version                : Could not collect
Libc version                 : glibc-2.34

==============================
       PyTorch Info
==============================
PyTorch version              : 2.8.0
Is debug build               : False
CUDA used to build PyTorch   : 12.8
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.9 (main, Aug 14 2025, 00:00:00) [GCC 11.5.0 20240719 (Red Hat 11.5.0-5)] (64-bit runtime)
Python platform              : Linux-6.1.155-176.282.amzn2023.x86_64-x86_64-with-glibc2.34

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.8.93
CUDA_MODULE_LOADING set to   : LAZY
GPU models and configuration : 
GPU 0: NVIDIA L40S
GPU 1: NVIDIA L40S
GPU 2: NVIDIA L40S
GPU 3: NVIDIA L40S

Nvidia driver version        : 580.95.05
cuDNN version                : Probably one of the following:
/usr/lib64/libcudnn.so.9.13.1
/usr/lib64/libcudnn_adv.so.9.13.1
/usr/lib64/libcudnn_cnn.so.9.13.1
/usr/lib64/libcudnn_engines_precompiled.so.9.13.1
/usr/lib64/libcudnn_engines_runtime_compiled.so.9.13.1
/usr/lib64/libcudnn_graph.so.9.13.1
/usr/lib64/libcudnn_heuristic.so.9.13.1
/usr/lib64/libcudnn_ops.so.9.13.1
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           48 bits physical, 48 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  48
On-line CPU(s) list:                     0-47
Vendor ID:                               AuthenticAMD
Model name:                              AMD EPYC 7R13 Processor
CPU family:                              25
Model:                                   1
Thread(s) per core:                      2
Core(s) per socket:                      24
Socket(s):                               1
Stepping:                                1
BogoMIPS:                                5299.99
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext perfctr_core invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save vaes vpclmulqdq rdpid
Hypervisor vendor:                       KVM
Virtualization type:                     full
L1d cache:                               768 KiB (24 instances)
L1i cache:                               768 KiB (24 instances)
L2 cache:                                12 MiB (24 instances)
L3 cache:                                96 MiB (3 instances)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-47
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Mitigation; safe RET
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Mitigation; Clear CPU buffers
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.3.1
[pip3] numpy==2.2.6
[pip3] nvidia-cudnn-frontend==1.15.0
[pip3] nvidia-ml-py==13.580.82
[pip3] pynvml==13.0.1
[pip3] pyzmq==27.1.0
[pip3] torch==2.8.0
[pip3] torchaudio==2.8.0
[pip3] torchvision==0.23.0
[pip3] transformers==4.57.1
[pip3] triton==3.4.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.11.0+rhai1
vLLM Build Flags:
  CUDA Archs: ; ROCm: Disabled
GPU Topology:
  	GPU0	GPU1	GPU2	GPU3	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	NODE	NODE	NODE	0-47	0		N/A
GPU1	NODE	 X 	NODE	NODE	0-47	0		N/A
GPU2	NODE	NODE	 X 	NODE	0-47	0		N/A
GPU3	NODE	NODE	NODE	 X 	0-47	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.0 driver>=525.60.13
CUDA_CACHE_DISABLE=1
TORCH_CUDA_ARCH_LIST=
NVIDIA_DRIVER_CAPABILITIES=compute,utility,video
VLLM_WORKER_MULTIPROC_METHOD=spawn
VLLM_USAGE_SOURCE=rhaiis-official-build
CUDA_VERSION=12.8.1
TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=15
TORCH_NCCL_DUMP_ON_TIMEOUT=0
VLLM_USAGE_STATS_SERVER=https://console.redhat.com/api/rhaiis-stats
VLLM_HAS_FLASHINFER_CUBIN=true
VLLM_NO_USAGE_STATS=1
CUDA_HOME=/usr/local/cuda
CUDA_HOME=/usr/local/cuda
VLLM_LOGGING_LEVEL=DEBUG
CUDA_MODULE_LOADING=LAZY
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_default

🐛 Describe the bug

There appears to be a bug in the torch peak memory calculation with Qwen-2.5-VL family of models (base model/quantized). This results in a lower overall GPU memory utilization resulting in very low supported concurrency (as a side effect of low KV Cache availability).

On a 4xL20 (or L40) loading , we would see memory utilization along the lines of:

• vLLM 0.10.2 - ~45 GB of 46 GB (per GPU)
• vLLM 0.11.0 - ~35 GB of 45 GB (per GPU)

Using --skip-mm-profile brings gpu memory utilization of vLLM 0.11.0 in line with 0.10.2.

This appears to be specific to just the Qwen-2.5-VL (maybe Qwen-3-VL) family of models. I did not see the same issue with adept/fuyu-8b model.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[NickLucche]

cc @ywang96 were there any changes to Qwen VL dummy items that may affect estimation here?

[ywang96]

@thameem-abbas This was not really a bug - previously we drastically underestimated the maximum number of video frames that the model can take, which resulted in bugs that long video items getting stuck at actual inference time.

To verify, can you simply specify --limit-mm-per-prompt.video 0? Image profiling should have not been affected at all so I'm curious if the issue still persists.

[thameem-abbas]

@ywang96 Yes. You were right. That greatly helped. It was purely the video profiling that caused this.

Just for other's reference following this issue later, the correct CLI arg is : --limit-mm-per-prompt.video 0

With mm-per-prompt.video set to 0, we see KV Cache and supported concurrency along the lines of previous vLLM versions.