[Bug]: vLLM startup memory check mis-detects available VRAM (reclaimable OS memory) on UMA Systems

[sunil1511]

🐛 Describe the bug

Issue Observed on UMA GPUs (DGX Spark and GH200)

I have observed this bug on both DGX Spark and GH200 systems.

Problem Description

vLLM defaults to requesting 90% GPU memory utilization (--gpu-memory-utilization=0.9). While it's typically easier to achieve 90% free memory on non-UMA systems, on UMA systems the memory can be occupied by paged cache and buffers which are reclaimable by the OS.

Instead of accounting for this reclaimable memory, vLLM fails with:

ValueError: Free memory on device cuda:0 (74.9/142.5 GiB) on startup is less than desired GPU memory utilization (0.9, 128.25 GiB). Decrease GPU memory utilization or reduce GPU memory used by other processes.

Root Cause

On UMA systems, cudaMemGetInfo (which vLLM uses via pynvml) doesn't account for memory that could be reclaimed from the OS (page cache, SWAP, buffers). This causes vLLM to underestimate available GPU memory during deployment, leading to false failures even when sufficient memory would be available after OS-level reclamation.

Affected Systems

• DGX Spark (GB10)
• GH200
• Other UMA Systems

Workarounds

• Reduce --gpu-memory-utilization below 0.9 (e.g., 0.8 or 0.85)
• Flush buffer caches before deployment (use with caution): sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'

Expected Behavior

vLLM should account for reclaimable memory on UMA systems, similar to how NVIDIA recommends checking /proc/meminfo for MemAvailable, SwapFree, and HugePages in addition to cudaMemGetInfo.

ARB: @akshaver

[haosdent]

May check if #35356 work for you, I verified in my DGX Spark @sunil1511

[shahizat]

Hi, I always try to use --gpu-memory-utilization less than default 0.9 on the Nvidia Jetson Orin, Thor, and DGX Spark. and it would also be helpful to have a special condition in vllm to check whether the system is using a dedicated GPU or an iGPU. #35356

cc @johnnynunez

[johnnynunez]

Hi, I always try to use --gpu-memory-utilization less than default 0.9 on the Nvidia Jetson Orin, Thor, and DGX Spark. and it would also be helpful to have a special condition in vllm to check whether the system is using a dedicated GPU or an iGPU. #35356

cc @johnnynunez

in UMA always, I launch a model, i run this command to free the ram

sudo sysctl -w vm.drop_caches=3

as i know, pynvml is not compatible with unified memory

[ehfd]

I am wondering if v0.17.1 fixed this issue or not.

[xinnywinne]

@haosdent, are you still actively working on the fix?

[haosdent]

@xinnywinne yep, anything need i change?