Skip to content

vllm Symmetric Memory issue #2018

Open
Open
vllm Symmetric Memory issue#2018

Description

@YCHYZW
YCHYZW
opened on Feb 2, 2026

We’re using the official Docker image. We run vLLM for acceleration on a 4× H200 setup. After running inference on a few videos, the process hangs. When we restart inference, it hangs again. The following error is reported:
INFO 01-31 02:37:56 [init.py:216] Automatically detected platform cuda.

📦 Loading vLLM model from /workspace/models/Qwen3-VL-235B-A22B-Thinking...

INFO 01-31 02:37:58 [utils.py:233] non-default args: {'trust_remote_code': True, 'max_model_len': 16394, 'tensor_parallel_size': 4, 'disable_log_stats': True, 'limit_mm_per_prompt': {'video': 1}, 'mm_encoder_tp_mode': 'data', 'model': '/workspace/models/Qwen3-VL-235B-A22B-Thinking'}

The argument trust_remote_code is to be used with Auto classes. It has no effect here and is ignored.

INFO 01-31 02:37:58 [model.py:547] Resolved architecture: Qwen3VLMoeForConditionalGeneration

torch_dtype is deprecated! Use dtype instead!

INFO 01-31 02:37:58 [model.py:1510] Using max model len 16394

INFO 01-31 02:37:58 [scheduler.py:205] Chunked prefill is enabled with max_num_batched_tokens=16384.

INFO 01-31 02:38:03 [init.py:216] Automatically detected platform cuda.

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m INFO 01-31 02:38:04 [core.py:644] Waiting for init message from front-end.

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m INFO 01-31 02:38:04 [core.py:77] Initializing a V1 LLM engine (v0.11.0) with config: model='/workspace/models/Qwen3-VL-235B-A22B-Thinking', speculative_config=None, tokenizer='/workspace/models/Qwen3-VL-235B-A22B-Thinking', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=16394, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/workspace/models/Qwen3-VL-235B-A22B-Thinking, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention","vllm.sparse_attn_indexer"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":[2,1],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"use_inductor_graph_partition":false,"pass_config":{},"max_capture_size":512,"local_cache_dir":null}

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m WARNING 01-31 02:38:04 [multiproc_executor.py:720] Reducing Torch parallelism from 96 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m INFO 01-31 02:38:04 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0, 1, 2, 3], buffer_handle=(4, 16777216, 10, 'psm_7fa79d62'), local_subscribe_addr='ipc:///tmp/b165c0a0-b806-46db-8a68-06e51119d16b', remote_subscribe_addr=None, remote_addr_ipv6=False)

INFO 01-31 02:38:07 [init.py:216] Automatically detected platform cuda.

INFO 01-31 02:38:07 [init.py:216] Automatically detected platform cuda.

INFO 01-31 02:38:07 [init.py:216] Automatically detected platform cuda.

INFO 01-31 02:38:07 [init.py:216] Automatically detected platform cuda.

W0131 02:38:12.356000 870469 torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.

W0131 02:38:12.356000 870469 torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.

W0131 02:38:12.445000 870468 torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.

W0131 02:38:12.445000 870468 torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.

W0131 02:38:12.446000 870470 torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.

W0131 02:38:12.446000 870470 torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.

W0131 02:38:12.459000 870471 torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.

W0131 02:38:12.459000 870471 torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.

INFO 01-31 02:38:12 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_50d9b1bd'), local_subscribe_addr='ipc:///tmp/ec1a1897-c5d0-41f3-ac26-b1ac351f3a22', remote_subscribe_addr=None, remote_addr_ipv6=False)

INFO 01-31 02:38:12 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_c37fe9d3'), local_subscribe_addr='ipc:///tmp/1d98de85-3613-4a38-8ee2-468ac61c4526', remote_subscribe_addr=None, remote_addr_ipv6=False)

INFO 01-31 02:38:12 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_8389a002'), local_subscribe_addr='ipc:///tmp/2d3d81b3-0a31-4492-8f6a-7958db86ccb3', remote_subscribe_addr=None, remote_addr_ipv6=False)

INFO 01-31 02:38:12 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_93d42bd1'), local_subscribe_addr='ipc:///tmp/87bba9f3-c3c3-45e0-bd00-faac926beb16', remote_subscribe_addr=None, remote_addr_ipv6=False)

[W131 02:38:13.613561191 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3

[W131 02:38:13.907319058 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3

[W131 02:38:13.053997115 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3

[W131 02:38:13.062356812 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3

[W131 02:38:13.114388735 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3

[W131 02:38:13.115606332 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3

[W131 02:38:14.709257958 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3

[W131 02:38:14.711217352 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3

[W131 02:38:14.734204549 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3

[W131 02:38:14.735565167 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3

[W131 02:38:14.759450437 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3

[W131 02:38:14.760819470 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3

[Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3

[Gloo] Rank 2 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3

[Gloo] Rank 1 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3

[Gloo] Rank 3 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3

[Gloo] Rank 0 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3

[Gloo] Rank 1 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3

[Gloo] Rank 2 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3

[Gloo] Rank 3 is connected to 3 peer ranks. Expected number of connected peer ranks is : 3

INFO 01-31 02:38:14 [init.py:1384] Found nccl from library libnccl.so.2

INFO 01-31 02:38:14 [pynccl.py:103] vLLM is using nccl==2.27.3

INFO 01-31 02:38:14 [init.py:1384] Found nccl from library libnccl.so.2

INFO 01-31 02:38:14 [pynccl.py:103] vLLM is using nccl==2.27.3

INFO 01-31 02:38:14 [init.py:1384] Found nccl from library libnccl.so.2

INFO 01-31 02:38:14 [init.py:1384] Found nccl from library libnccl.so.2

INFO 01-31 02:38:14 [pynccl.py:103] vLLM is using nccl==2.27.3

INFO 01-31 02:38:14 [pynccl.py:103] vLLM is using nccl==2.27.3

ERROR 01-31 02:38:15 [multiproc_executor.py:597] WorkerProc failed to start.

ERROR 01-31 02:38:15 [multiproc_executor.py:597] Traceback (most recent call last):

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 571, in worker_main

ERROR 01-31 02:38:15 [multiproc_executor.py:597] worker = WorkerProc(*args, **kwargs)

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 430, in init

ERROR 01-31 02:38:15 [multiproc_executor.py:597] self.worker.init_device()

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 259, in init_device

ERROR 01-31 02:38:15 [multiproc_executor.py:597] self.worker.init_device() # type: ignore

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 169, in init_device

ERROR 01-31 02:38:15 [multiproc_executor.py:597] init_worker_distributed_environment(self.vllm_config, self.rank,

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 705, in init_worker_distributed_environment

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ensure_model_parallel_initialized(

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1228, in ensure_model_parallel_initialized

ERROR 01-31 02:38:15 [multiproc_executor.py:597] initialize_model_parallel(tensor_model_parallel_size,

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1152, in initialize_model_parallel

ERROR 01-31 02:38:15 [multiproc_executor.py:597] _TP = init_model_parallel_group(group_ranks,

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 924, in init_model_parallel_group

ERROR 01-31 02:38:15 [multiproc_executor.py:597] return GroupCoordinator(

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 255, in init

ERROR 01-31 02:38:15 [multiproc_executor.py:597] self.device_communicator = device_comm_cls(

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 68, in init

ERROR 01-31 02:38:15 [multiproc_executor.py:597] self.symm_mem_comm = SymmMemCommunicator(

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/symm_mem.py", line 88, in init

ERROR 01-31 02:38:15 [multiproc_executor.py:597] handle = torch_symm_mem.rendezvous(self.buffer, self.group.group_name)

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/torch/distributed/_symmetric_memory/init.py", line 1711, in rendezvous

ERROR 01-31 02:38:15 [multiproc_executor.py:597] return _SymmetricMemory.rendezvous(tensor, group_name)

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] RuntimeError: CUDA driver error: the operation cannot be performed in the present state

ERROR 01-31 02:38:15 [multiproc_executor.py:597] WorkerProc failed to start.

ERROR 01-31 02:38:15 [multiproc_executor.py:597] Traceback (most recent call last):

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 571, in worker_main

ERROR 01-31 02:38:15 [multiproc_executor.py:597] worker = WorkerProc(*args, **kwargs)

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 430, in init

ERROR 01-31 02:38:15 [multiproc_executor.py:597] self.worker.init_device()

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 259, in init_device

ERROR 01-31 02:38:15 [multiproc_executor.py:597] self.worker.init_device() # type: ignore

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 169, in init_device

ERROR 01-31 02:38:15 [multiproc_executor.py:597] init_worker_distributed_environment(self.vllm_config, self.rank,

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 705, in init_worker_distributed_environment

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ensure_model_parallel_initialized(

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1228, in ensure_model_parallel_initialized

ERROR 01-31 02:38:15 [multiproc_executor.py:597] initialize_model_parallel(tensor_model_parallel_size,

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1152, in initialize_model_parallel

ERROR 01-31 02:38:15 [multiproc_executor.py:597] _TP = init_model_parallel_group(group_ranks,

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 924, in init_model_parallel_group

ERROR 01-31 02:38:15 [multiproc_executor.py:597] return GroupCoordinator(

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 255, in init

ERROR 01-31 02:38:15 [multiproc_executor.py:597] self.device_communicator = device_comm_cls(

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 68, in init

ERROR 01-31 02:38:15 [multiproc_executor.py:597] self.symm_mem_comm = SymmMemCommunicator(

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/symm_mem.py", line 88, in init

ERROR 01-31 02:38:15 [multiproc_executor.py:597] handle = torch_symm_mem.rendezvous(self.buffer, self.group.group_name)

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/torch/distributed/_symmetric_memory/init.py", line 1711, in rendezvous

ERROR 01-31 02:38:15 [multiproc_executor.py:597] return _SymmetricMemory.rendezvous(tensor, group_name)

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] RuntimeError: CUDA driver error: the operation cannot be performed in the present state

ERROR 01-31 02:38:15 [multiproc_executor.py:597] WorkerProc failed to start.

ERROR 01-31 02:38:15 [multiproc_executor.py:597] Traceback (most recent call last):

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 571, in worker_main

ERROR 01-31 02:38:15 [multiproc_executor.py:597] worker = WorkerProc(*args, **kwargs)

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 430, in init

ERROR 01-31 02:38:15 [multiproc_executor.py:597] self.worker.init_device()

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 259, in init_device

ERROR 01-31 02:38:15 [multiproc_executor.py:597] self.worker.init_device() # type: ignore

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 169, in init_device

ERROR 01-31 02:38:15 [multiproc_executor.py:597] init_worker_distributed_environment(self.vllm_config, self.rank,

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 705, in init_worker_distributed_environment

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ensure_model_parallel_initialized(

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1228, in ensure_model_parallel_initialized

ERROR 01-31 02:38:15 [multiproc_executor.py:597] initialize_model_parallel(tensor_model_parallel_size,

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 1152, in initialize_model_parallel

ERROR 01-31 02:38:15 [multiproc_executor.py:597] _TP = init_model_parallel_group(group_ranks,

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 924, in init_model_parallel_group

ERROR 01-31 02:38:15 [multiproc_executor.py:597] return GroupCoordinator(

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/parallel_state.py", line 255, in init

ERROR 01-31 02:38:15 [multiproc_executor.py:597] self.device_communicator = device_comm_cls(

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/cuda_communicator.py", line 68, in init

ERROR 01-31 02:38:15 [multiproc_executor.py:597] self.symm_mem_comm = SymmMemCommunicator(

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/symm_mem.py", line 88, in init

ERROR 01-31 02:38:15 [multiproc_executor.py:597] handle = torch_symm_mem.rendezvous(self.buffer, self.group.group_name)

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] File "/usr/local/lib/python3.12/dist-packages/torch/distributed/_symmetric_memory/init.py", line 1711, in rendezvous

ERROR 01-31 02:38:15 [multiproc_executor.py:597] return _SymmetricMemory.rendezvous(tensor, group_name)

ERROR 01-31 02:38:15 [multiproc_executor.py:597] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

ERROR 01-31 02:38:15 [multiproc_executor.py:597] RuntimeError: CUDA driver error: the operation cannot be performed in the present state

INFO 01-31 02:38:15 [multiproc_executor.py:558] Parent process exited, terminating worker

INFO 01-31 02:38:15 [multiproc_executor.py:558] Parent process exited, terminating worker

INFO 01-31 02:38:15 [multiproc_executor.py:558] Parent process exited, terminating worker

INFO 01-31 02:38:15 [multiproc_executor.py:558] Parent process exited, terminating worker

[rank0]:[W131 02:38:16.668150131 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m ERROR 01-31 02:38:18 [core.py:708] EngineCore failed to start.

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m ERROR 01-31 02:38:18 [core.py:708] Traceback (most recent call last):

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m ERROR 01-31 02:38:18 [core.py:708] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 699, in run_engine_core

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m ERROR 01-31 02:38:18 [core.py:708] engine_core = EngineCoreProc(*args, **kwargs)

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m ERROR 01-31 02:38:18 [core.py:708] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m ERROR 01-31 02:38:18 [core.py:708] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 498, in init

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m ERROR 01-31 02:38:18 [core.py:708] super().init(vllm_config, executor_class, log_stats,

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m ERROR 01-31 02:38:18 [core.py:708] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 83, in init

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m ERROR 01-31 02:38:18 [core.py:708] self.model_executor = executor_class(vllm_config)

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m ERROR 01-31 02:38:18 [core.py:708] ^^^^^^^^^^^^^^^^^^^^^^^^^^^

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m ERROR 01-31 02:38:18 [core.py:708] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 54, in init

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m ERROR 01-31 02:38:18 [core.py:708] self._init_executor()

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m ERROR 01-31 02:38:18 [core.py:708] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 106, in _init_executor

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m ERROR 01-31 02:38:18 [core.py:708] self.workers = WorkerProc.wait_for_ready(unready_workers)

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m ERROR 01-31 02:38:18 [core.py:708] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m ERROR 01-31 02:38:18 [core.py:708] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 509, in wait_for_ready

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m ERROR 01-31 02:38:18 [core.py:708] raise e from None

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m ERROR 01-31 02:38:18 [core.py:708] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m Process EngineCore_DP0:

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m Traceback (most recent call last):

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m self.run()

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m self._target(*self._args, **self._kwargs)

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 712, in run_engine_core

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m raise e

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 699, in run_engine_core

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m engine_core = EngineCoreProc(*args, **kwargs)

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 498, in init

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m super().init(vllm_config, executor_class, log_stats,

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 83, in init

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m self.model_executor = executor_class(vllm_config)

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 54, in init

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m self._init_executor()

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 106, in _init_executor

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m self.workers = WorkerProc.wait_for_ready(unready_workers)

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 509, in wait_for_ready

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m raise e from None

�[1;36m(EngineCore_DP0 pid=870272)�[0;0m Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.

Traceback (most recent call last):

File "/workspace/code/Qwen3-VL/inference_video_vllm.py", line 194, in 

main()

File "/workspace/code/Qwen3-VL/inference_video_vllm.py", line 72, in main

llm = LLM(

      ^^^^

File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/llm.py", line 297, in init

self.llm_engine = LLMEngine.from_engine_args(

                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/llm_engine.py", line 177, in from_engine_args

return cls(vllm_config=vllm_config,

       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/llm_engine.py", line 114, in init

self.engine_core = EngineCoreClient.make_client(

                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 80, in make_client

return SyncMPClient(vllm_config, executor_class, log_stats)

       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 602, in init

super().__init__(

File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 448, in init

with launch_core_engines(vllm_config, executor_class,

     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/lib/python3.12/contextlib.py", line 144, in exit

next(self.gen)

File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 732, in launch_core_engines

wait_for_engine_startup(

File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 785, in wait_for_engine_startup

raise RuntimeError("Engine core initialization failed. "

RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

怎么解决一下

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions