[Bug]: Ray+vllm run, then crash #13535
Description
Your current environment
Details
Collecting environment information... PyTorch version: 2.5.1+cu124 Is debug build: False CUDA used to build PyTorch: 12.4 ROCM used to build PyTorch: N/AOS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35
Python version: 3.11.0rc1 (main, Aug 12 2022, 10:02:14) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-94-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.4.131
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: Tesla T4
GPU 1: Tesla T4
Nvidia driver version: 560.35.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 40 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: GenuineIntel
Model name: Intel Xeon Processor (Skylake, IBRS)
CPU family: 6
Model: 85
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 16
Stepping: 4
BogoMIPS: 4000.02
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ibrs ibpb fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 arat avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 512 KiB (16 instances)
L1i cache: 512 KiB (16 instances)
L2 cache: 64 MiB (16 instances)
L3 cache: 256 MiB (16 instances)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Unknown: Dependent on hypervisor status
Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported
Vulnerability L1tf: Mitigation; PTE Inversion
Vulnerability Mds: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS, IBPB conditional, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Versions of relevant libraries:
[pip3] flashinfer-python==0.2.0.post2+cu124torch2.5
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.2.0
[pip3] torch==2.5.1
[pip3] torchao==0.8.0
[pip3] torchaudio==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.46.1
[pip3] triton==3.1.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.4.post1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 GPU1 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X PHB 0-15 0 N/A
GPU1 PHB X 0-15 0 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
NCCL_SOCKET_IFNAME=ens3
NCCL_NVLS_ENABLE=0
NCCL_DEBUG=error
NCCL_NET=Socket
LD_LIBRARY_PATH=/root/project/sglang/venv/lib/python3.11/site-packages/cv2/../../lib64:/usr/local/cuda-12.4/lib64:/usr/local/cuda-12.4/lib64:
NCCL_IB_DISABLE=0
CUDA_HOME=/usr/local/cuda-12.4
CUDA_HOME=/usr/local/cuda-12.4
CUDA_MODULE_LOADING=LAZY
The output of `python collect_env.py`
Your output of `python collect_env.py` here
🐛 Describe the bug
run python -m vllm.entrypoints.openai.api_server --disable-custom-all-reduce --gpu-memory-utilization 0.8 --dtype float16 --trust-remote-code --host 0.0.0.0 --served-model-name qwen_coder --tensor-parallel-size 4 --distributed-executor-backend ray --model /root/model/Qwen/Qwen2.5-7B-Instruct/
crash:
(RayWorkerWrapper pid=270291, ip=10.175.94.190) [rank3]:[E219 08:58:32.606233339 ProcessGroupNCCL.cpp:616] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=31, OpType=ALLREDUCE, NumelIn=117440512, NumelOut=117440512, Timeout(ms)=600000) ran for 600031 milliseconds before timing out.
(RayWorkerWrapper pid=270291, ip=10.175.94.190) [rank3]:[E219 08:58:32.606667496 ProcessGroupNCCL.cpp:1785] [PG ID 2 PG GUID 3 Rank 3] Exception (either an error or timeout) detected by watchdog at work: 31, last enqueued NCCL work: 58, last completed NCCL work: 30.
(RayWorkerWrapper pid=270291, ip=10.175.94.190) [rank3]:[E219 08:58:32.606693321 ProcessGroupNCCL.cpp:1834] [PG ID 2 PG GUID 3 Rank 3] Timeout at NCCL work: 31, last enqueued NCCL work: 58, last completed NCCL work: 30.
(RayWorkerWrapper pid=270291, ip=10.175.94.190) [rank3]:[E219 08:58:32.606716422 ProcessGroupNCCL.cpp:630] [Rank 3] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
(RayWorkerWrapper pid=270291, ip=10.175.94.190) [rank3]:[E219 08:58:32.606733115 ProcessGroupNCCL.cpp:636] [Rank 3] To avoid data inconsistency, we are taking the entire process down.
(RayWorkerWrapper pid=270291, ip=10.175.94.190) [rank3]:[E219 08:58:32.611865696 ProcessGroupNCCL.cpp:1595] [PG ID 2 PG GUID 3 Rank 3] Process group watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=31, OpType=ALLREDUCE, NumelIn=117440512, NumelOut=117440512, Timeout(ms)=600000) ran for 600031 milliseconds before timing out.
(RayWorkerWrapper pid=270291, ip=10.175.94.190) Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:618 (most recent call first):
(RayWorkerWrapper pid=270291, ip=10.175.94.190) frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f247c16c446 in /root/project/sglang/venv/lib/python3.11/site-packages/torch/lib/libc10.so)
(RayWorkerWrapper pid=270291, ip=10.175.94.190) frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x282 (0x7f1a3ac39772 in /root/project/sglang/venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
(RayWorkerWrapper pid=270291, ip=10.175.94.190) frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7f1a3ac40bb3 in /root/project/sglang/venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
(RayWorkerWrapper pid=270291, ip=10.175.94.190) frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f1a3ac4261d in /root/project/sglang/venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
(RayWorkerWrapper pid=270291, ip=10.175.94.190) frame #4: <unknown function> + 0xdc253 (0x7f24993a3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
(RayWorkerWrapper pid=270291, ip=10.175.94.190) frame #5: <unknown function> + 0x94ac3 (0x7f249b71cac3 in /lib/x86_64-linux-gnu/libc.so.6)
(RayWorkerWrapper pid=270291, ip=10.175.94.190) frame #6: <unknown function> + 0x126850 (0x7f249b7ae850 in /lib/x86_64-linux-gnu/libc.so.6)
(RayWorkerWrapper pid=270291, ip=10.175.94.190)
(RayWorkerWrapper pid=270291, ip=10.175.94.190) [2025-02-19 08:58:32,022 E 270291 270373] logging.cc:108: Unhandled exception: N3c1016DistBackendErrorE. what(): [PG ID 2 PG GUID 3 Rank 3] Process group watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=31, OpType=ALLREDUCE, NumelIn=117440512, NumelOut=117440512, Timeout(ms)=600000) ran for 600031 milliseconds before timing out.
(RayWorkerWrapper pid=270291, ip=10.175.94.190) Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:618 (most recent call first):
(RayWorkerWrapper pid=270291, ip=10.175.94.190) frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f247c16c446 in /root/project/sglang/venv/lib/python3.11/site-packages/torch/lib/libc10.so)
(RayWorkerWrapper pid=270291, ip=10.175.94.190) frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x282 (0x7f1a3ac39772 in /root/project/sglang/venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
(RayWorkerWrapper pid=270291, ip=10.175.94.190) frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7f1a3ac40bb3 in /root/project/sglang/venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
(RayWorkerWrapper pid=270291, ip=10.175.94.190) frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f1a3ac4261d in /root/project/sglang/venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
(RayWorkerWrapper pid=270291, ip=10.175.94.190) frame #4: <unknown function> + 0xdc253 (0x7f24993a3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
(RayWorkerWrapper pid=270291, ip=10.175.94.190) frame #5: <unknown function> + 0x94ac3 (0x7f249b71cac3 in /lib/x86_64-linux-gnu/libc.so.6)
(RayWorkerWrapper pid=270291, ip=10.175.94.190) frame #6: <unknown function> + 0x126850 (0x7f249b7ae850 in /lib/x86_64-linux-gnu/libc.so.6)
(RayWorkerWrapper pid=270291, ip=10.175.94.190)
(RayWorkerWrapper pid=270291, ip=10.175.94.190) Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1601 (most recent call first):
(RayWorkerWrapper pid=270291, ip=10.175.94.190) frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f247c16c446 in /root/project/sglang/venv/lib/python3.11/site-packages/torch/lib/libc10.so)
(RayWorkerWrapper pid=270291, ip=10.175.94.190) frame #1: <unknown function> + 0xe4271b (0x7f1a3a8af71b in /root/project/sglang/venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
(RayWorkerWrapper pid=270291, ip=10.175.94.190) frame #2: <unknown function> + 0xdc253 (0x7f24993a3253 in /lib/x86_64-linux-gnu/libstdc++.so.6)
(RayWorkerWrapper pid=270291, ip=10.175.94.190) frame #3: <unknown function> + 0x94ac3 (0x7f249b71cac3 in /lib/x86_64-linux-gnu/libc.so.6)
(RayWorkerWrapper pid=270291, ip=10.175.94.190) frame #4: <unknown function> + 0x126850 (0x7f249b7ae850 in /lib/x86_64-linux-gnu/libc.so.6)
(RayWorkerWrapper pid=270291, ip=10.175.94.190)
(RayWorkerWrapper pid=2667395)
(RayWorkerWrapper pid=2667395)
(RayWorkerWrapper pid=2667395)
(RayWorkerWrapper pid=270237, ip=10.175.94.190)
(RayWorkerWrapper pid=270237, ip=10.175.94.190)
(RayWorkerWrapper pid=270237, ip=10.175.94.190)
(RayWorkerWrapper pid=270237, ip=10.175.94.190) [2025-02-19 08:58:32,122 E 270237 270370] logging.cc:115: Stack trace:
(RayWorkerWrapper pid=270237, ip=10.175.94.190) /root/project/sglang/venv/lib/python3.11/site-packages/ray/_raylet.so(+0x1141c3a) [0x7f4072e63c3a] ray::operator<<()
(RayWorkerWrapper pid=270237, ip=10.175.94.190) /root/project/sglang/venv/lib/python3.11/site-packages/ray/_raylet.so(+0x1144ec2) [0x7f4072e66ec2] ray::TerminateHandler()
(RayWorkerWrapper pid=270237, ip=10.175.94.190) /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae20c) [0x7f4071ba420c]
(RayWorkerWrapper pid=270237, ip=10.175.94.190) /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae277) [0x7f4071ba4277]
(RayWorkerWrapper pid=270237, ip=10.175.94.190) /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae1fe) [0x7f4071ba41fe]
(RayWorkerWrapper pid=270237, ip=10.175.94.190) /root/project/sglang/venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so(+0xe427c9) [0x7f360fd207c9] c10d::ProcessGroupNCCL::ncclCommWatchdog()
(RayWorkerWrapper pid=270237, ip=10.175.94.190) /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f4071bd2253]
(RayWorkerWrapper pid=270237, ip=10.175.94.190) /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f4073f4bac3]
(RayWorkerWrapper pid=270237, ip=10.175.94.190) /lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x7f4073fdd850]
(RayWorkerWrapper pid=270237, ip=10.175.94.190)
(RayWorkerWrapper pid=270237, ip=10.175.94.190) *** SIGABRT received at time=1739955512 on cpu 15 ***
(RayWorkerWrapper pid=270237, ip=10.175.94.190) PC: @ 0x7f4073f4d9fc (unknown) pthread_kill
(RayWorkerWrapper pid=270237, ip=10.175.94.190) @ 0x7f4073ef9520 (unknown) (unknown)
(RayWorkerWrapper pid=270237, ip=10.175.94.190) [2025-02-19 08:58:32,123 E 270237 270370] logging.cc:440: *** SIGABRT received at time=1739955512 on cpu 15 ***
(RayWorkerWrapper pid=270237, ip=10.175.94.190) [2025-02-19 08:58:32,123 E 270237 270370] logging.cc:440: PC: @ 0x7f4073f4d9fc (unknown) pthread_kill
(RayWorkerWrapper pid=270237, ip=10.175.94.190) [2025-02-19 08:58:32,123 E 270237 270370] logging.cc:440: @ 0x7f4073ef9520 (unknown) (unknown)
(RayWorkerWrapper pid=270237, ip=10.175.94.190) Fatal Python error: Aborted
(RayWorkerWrapper pid=270237, ip=10.175.94.190)
(RayWorkerWrapper pid=270237, ip=10.175.94.190)
(RayWorkerWrapper pid=270237, ip=10.175.94.190) Extension modules: msgpack._cmsgpack, google._upb._message, psutil._psutil_linux, psutil._psutil_posix, setproctitle, yaml._yaml, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, uvloop.loop, ray._raylet, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, markupsafe._speedups, PIL._imaging, msgspec._core, sentencepiece._sentencepiece, regex._regex, PIL._imagingft, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, frozenlist._frozenlist, pyarrow.lib, pyarrow._json, zmq.backend.cython._zmq (total: 52)
(RayWorkerWrapper pid=270291, ip=10.175.94.190) /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae20c) [0x7f249937520c]
(RayWorkerWrapper pid=270291, ip=10.175.94.190) /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae277) [0x7f2499375277]
(RayWorkerWrapper pid=270291, ip=10.175.94.190) /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae1fe) [0x7f24993751fe]
(RayWorkerWrapper pid=270291, ip=10.175.94.190) /root/project/sglang/venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so(+0xe427c9) [0x7f1a3a8af7c9] c10d::ProcessGroupNCCL::ncclCommWatchdog()
(RayWorkerWrapper pid=270291, ip=10.175.94.190) /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f24993a3253]
(RayWorkerWrapper pid=270291, ip=10.175.94.190) /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f249b71cac3]
(RayWorkerWrapper pid=270291, ip=10.175.94.190) /lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x7f249b7ae850]
(RayWorkerWrapper pid=270291, ip=10.175.94.190)
(RayWorkerWrapper pid=270291, ip=10.175.94.190)
(RayWorkerWrapper pid=270291, ip=10.175.94.190)
(RayWorkerWrapper pid=2667395) /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae20c) [0x7fdb8762020c]
(RayWorkerWrapper pid=2667395) /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae277) [0x7fdb87620277]
(RayWorkerWrapper pid=2667395) /lib/x86_64-linux-gnu/libstdc++.so.6(+0xae1fe) [0x7fdb876201fe]
(RayWorkerWrapper pid=2667395) /root/project/sglang/venv/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so(+0xe427c9) [0x7fd15b3007c9] c10d::ProcessGroupNCCL::ncclCommWatchdog()
(RayWorkerWrapper pid=2667395) /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7fdb8764e253]
(RayWorkerWrapper pid=2667395) /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fdb899c4ac3]
(RayWorkerWrapper pid=2667395) /lib/x86_64-linux-gnu/libc.so.6(+0x126850) [0x7fdb89a56850]
(RayWorkerWrapper pid=2667395)
(RayWorkerWrapper pid=2667395)
(RayWorkerWrapper pid=2667395)
(raylet) A worker died or was killed while executing a task by an unexpected system error. To troubleshoot the problem, check the logs for the dead worker. RayTask ID: ffffffffffffffffa55573c318fcb45089df416702000000 Worker ID: 40060cc4698e6b41b7073182fde862a6a5ea32e8f6ac18bc2f4e1c15 Node ID: 1a6050c7a4f77ea40a9aa1d8f08ba6cfbec96c0138c62b2190c918cb Worker IP address: 10.175.94.190 Worker port: 10005 Worker PID: 270291 Worker exit type: SYSTEM_ERROR Worker exit detail: Worker unexpectedly exits with a connection error code 2. End of file. There are some potential root causes. (1) The process is killed by SIGKILL by OOM killer due to high memory usage. (2) ray stop --force is called. (3) The worker is crashed unexpectedly due to SIGSEGV or other unexpected errors.
(RayWorkerWrapper pid=2667395) INFO 02-19 16:47:57 model_runner.py:1077] Loading model weights took 3.5546 GB [repeated 2x across cluster]
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.