[Usage]: vLLM v0.9.1 Free Memory on device on startup is less than desired GPU memory utilization

[batuhanunverdi]

Your current environment

How would you like to use vllm

I created a container with these arguments by using vLLM v0.9.1 by usnig Dockerode.

  --gpus all \
  --ipc=host \
  -p 8000:8000 \
  -v /home/openzeka/.cache/huggingface:/root/.cache/huggingface:rw \
  -e HF_TOKEN=<TOKEN> \
  vllm/vllm-openai:v0.9.1 \
  --max-model-len 1024 \
  --gpu-memory-utilization 0.95 \
  --max-num-seqs 1 \
  --tensor-parallel-size 2 \
  --model deepseek-ai/DeepSeek-R1-Distill-Llama-8B

But after that I got an error like. My GPUs are free to use.

[multiproc_executor.py:492] ValueError: Free memory on device (31.44/47.38 GiB) on startup is less than desired GPU memory utilization (0.95, 45.01 GiB). Decrease GPU memory utilization or reduce GPU memory used by other processes.
ERROR 06-30 13:53:19 [core.py:515] EngineCore failed to start.
ERROR 06-30 13:53:19 [core.py:515] Traceback (most recent call last):
ERROR 06-30 13:53:19 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 506, in run_engine_core
ERROR 06-30 13:53:19 [core.py:515]     engine_core = EngineCoreProc(*args, **kwargs)
ERROR 06-30 13:53:19 [core.py:515]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-30 13:53:19 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 390, in __init__
ERROR 06-30 13:53:19 [core.py:515]     super().__init__(vllm_config, executor_class, log_stats,
ERROR 06-30 13:53:19 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 76, in __init__
ERROR 06-30 13:53:19 [core.py:515]     self.model_executor = executor_class(vllm_config)
ERROR 06-30 13:53:19 [core.py:515]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-30 13:53:19 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 53, in __init__
ERROR 06-30 13:53:19 [core.py:515]     self._init_executor()
ERROR 06-30 13:53:19 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 98, in _init_executor
ERROR 06-30 13:53:19 [core.py:515]     self.workers = WorkerProc.wait_for_ready(unready_workers)
ERROR 06-30 13:53:19 [core.py:515]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-30 13:53:19 [core.py:515]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 427, in wait_for_ready
ERROR 06-30 13:53:19 [core.py:515]     raise e from None
ERROR 06-30 13:53:19 [core.py:515] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
Process EngineCore_0:
Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 519, in run_engine_core
    raise e
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 506, in run_engine_core
    engine_core = EngineCoreProc(*args, **kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 390, in __init__
    super().__init__(vllm_config, executor_class, log_stats,
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 76, in __init__
    self.model_executor = executor_class(vllm_config)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 53, in __init__
    self._init_executor()
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 98, in _init_executor
    self.workers = WorkerProc.wait_for_ready(unready_workers)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 427, in wait_for_ready
    raise e from None
Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1387, in <module>
    uvloop.run(run_server(args))
  File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 109, in run
    return __asyncio.run(
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
           ^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1323, in run_server
    await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1343, in run_server_worker
    async with build_async_engine_client(args, client_config) as engine_client:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 155, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 191, in build_async_engine_client_from_engine_args
    async_llm = AsyncLLM.from_vllm_config(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 162, in from_vllm_config
    return cls(
           ^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 124, in __init__
    self.engine_core = EngineCoreClient.make_async_mp_client(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 93, in make_async_mp_client
    return AsyncMPClient(vllm_config, executor_class, log_stats,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 716, in __init__
    super().__init__(
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 422, in __init__
    self._init_engines_direct(vllm_config, local_only,
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 491, in _init_engines_direct
    self._wait_for_engine_startup(handshake_socket, input_address,
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 511, in _wait_for_engine_startup
    wait_for_engine_startup(
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/utils.py", line 494, in wait_for_engine_startup
    raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[iAlexMlops]

How was the problem solved?

[wukaixingxp]

I am seeing similar error running 8b models

[aravinthsci]

I am still seeing similar error while running models

[gelembungsahabat]

it's because your VRAM is less than gpu_memory_utilization value.

set your gpu_memory_utilization value to 0.8 or lower instead.

hope it helps :)

[vigneshselvaraj03]

U can solve this problem by changing the gpu utilization
normally vllm looks 90% of gpu, if not available, that raise that above error,
u can easilly solve this issue by setting gpu_memory_utilization to less that '0.8'

[jankogasic]

I'm trying to deploy a model which can easily fit with ollama on 1 out of 4 GPUs that I have. vLLM is throwing the mentioned error complaining that one GPU has <90% free VRAM. Why doesn't it ignore it and load model other 3 empty GPUs?

Here is my docker compose

  vllm-gptoss20b:
    image: docker.io/vllm/vllm-openai:v0.11.2
    container_name: vllm-gptoss20b
    command: --model openai/gpt-oss-20b --port 8000
    volumes:
      - ./.cache/huggingface:/root/.cache/huggingface
    env_file:
      - ./.env
    ipc: host
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    runtime: nvidia

[polaroi8d]

I'm trying to deploy a model which can easily fit with ollama on 1 out of 4 GPUs that I have. vLLM is throwing the mentioned error complaining that one GPU has <90% free VRAM. Why doesn't it ignore it and load model other 3 empty GPUs?

Here is my docker compose

  vllm-gptoss20b:
    image: docker.io/vllm/vllm-openai:v0.11.2
    container_name: vllm-gptoss20b
    command: --model openai/gpt-oss-20b --port 8000
    volumes:
      - ./.cache/huggingface:/root/.cache/huggingface
    env_file:
      - ./.env
    ipc: host
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    runtime: nvidia

For docker you should try something like this:

services:
  vllm:
    image: vllm/vllm-openai:nightly
    container_name: vllm
    runtime: nvidia
    ports:
      - "9999:8000"
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
    environment:
      - HF_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=all
    command: >
      --model Qwen/Qwen3.6-35B-A3B
      --host 0.0.0.0
      --port 8000
      --enable-auto-tool-choice
      --tool-call-parser qwen3_coder
      --gpu-memory-utilization "0.60"