[Bug]: RAY_CGRAPH_get_timeout is not set successfully. Ray still detects default timeout value. #19703
Description
Your current environment
The output of python collect_env.py
INFO 06-16 10:48:23 [__init__.py:244] Automatically detected platform cuda.
Collecting environment information...
==============================
System Info
==============================
OS : Ubuntu 22.04.5 LTS (aarch64)
GCC version : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version : Could not collect
CMake version : version 4.0.2
Libc version : glibc-2.35
==============================
PyTorch Info
==============================
PyTorch version : 2.7.0+cu128
Is debug build : False
CUDA used to build PyTorch : 12.8
ROCM used to build PyTorch : N/A
==============================
Python Environment
==============================
Python version : 3.12.11 (main, Jun 4 2025, 08:56:18) [GCC 11.4.0] (64-bit runtime)
Python platform : Linux-6.8.0-1028-nvidia-64k-aarch64-with-glibc2.35
==============================
CUDA / GPU Info
==============================
Is CUDA available : True
CUDA runtime version : 12.8.93
CUDA_MODULE_LOADING set to : LAZY
GPU models and configuration :
GPU 0: NVIDIA GB200
GPU 1: NVIDIA GB200
GPU 2: NVIDIA GB200
GPU 3: NVIDIA GB200
Nvidia driver version : 570.158.01
cuDNN version : Could not collect
HIP runtime version : N/A
MIOpen runtime version : N/A
Is XNNPACK available : True
==============================
CPU Info
==============================
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 144
On-line CPU(s) list: 0-143
Vendor ID: ARM
Model name: Neoverse-V2
Model: 0
Thread(s) per core: 1
Core(s) per socket: 72
Socket(s): 2
Stepping: r0p0
Frequency boost: disabled
CPU max MHz: 3393.0000
CPU min MHz: 81.0000
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm ssbs sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh bti
L1d cache: 9 MiB (144 instances)
L1i cache: 9 MiB (144 instances)
L2 cache: 144 MiB (144 instances)
L3 cache: 228 MiB (2 instances)
NUMA node(s): 34
NUMA node0 CPU(s): 0-71
NUMA node1 CPU(s): 72-143
NUMA node2 CPU(s):
NUMA node3 CPU(s):
NUMA node4 CPU(s):
NUMA node5 CPU(s):
NUMA node6 CPU(s):
NUMA node7 CPU(s):
NUMA node8 CPU(s):
NUMA node9 CPU(s):
NUMA node10 CPU(s):
NUMA node11 CPU(s):
NUMA node12 CPU(s):
NUMA node13 CPU(s):
NUMA node14 CPU(s):
NUMA node15 CPU(s):
NUMA node16 CPU(s):
NUMA node17 CPU(s):
NUMA node18 CPU(s):
NUMA node19 CPU(s):
NUMA node20 CPU(s):
NUMA node21 CPU(s):
NUMA node22 CPU(s):
NUMA node23 CPU(s):
NUMA node24 CPU(s):
NUMA node25 CPU(s):
NUMA node26 CPU(s):
NUMA node27 CPU(s):
NUMA node28 CPU(s):
NUMA node29 CPU(s):
NUMA node30 CPU(s):
NUMA node31 CPU(s):
NUMA node32 CPU(s):
NUMA node33 CPU(s):
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Not affected
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
==============================
Versions of relevant libraries
==============================
[pip3] numpy==2.2.6
[pip3] pytorch-triton==3.3.0+gitab727c40
[pip3] pyzmq==26.4.0
[pip3] torch==2.7.0+cu128
[pip3] torchaudio==2.7.0
[pip3] torchvision==0.22.0
[pip3] transformers==4.52.4
[pip3] triton==3.3.0
[conda] Could not collect
==============================
vLLM Info
==============================
ROCM Version : Could not collect
Neuron SDK Version : N/A
vLLM Version : 0.9.2.dev29+gc7ea0b56c (git sha: c7ea0b56c)
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 GPU1 GPU2 GPU3 NIC0 NIC1 NIC2 NIC3 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV18 NV18 NV18 NODE NODE SYS SYS 0-71 0 2
GPU1 NV18 X NV18 NV18 NODE NODE SYS SYS 0-71 0 10
GPU2 NV18 NV18 X NV18 SYS SYS NODE NODE 72-143 1 18
GPU3 NV18 NV18 NV18 X SYS SYS NODE NODE 72-143 1 26
NIC0 NODE NODE SYS SYS X NODE SYS SYS
NIC1 NODE NODE SYS SYS NODE X SYS SYS
NIC2 SYS SYS NODE NODE SYS SYS X NODE
NIC3 SYS SYS NODE NODE SYS SYS NODE X
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks
NIC Legend:
NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_4
NIC3: mlx5_5
==============================
Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.8 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566
CUDA_CACHE_DISABLE=1
NCCL_VERSION=2.25.1-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
VLLM_USAGE_SOURCE=production-docker-image
CUDA_DEVICE_ORDER=PCI_BUS_ID
CUDA_VERSION=12.8.1
NCCL_SHARP_GROUP_SIZE_THRESH=2
NCCL_COLLNET_ENABLE=1
NCCL_IB_SL=1
LD_LIBRARY_PATH=/usr/local/cuda/lib64
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY
🐛 Describe the bug
Background
I am trying to run multi-node on GB200. I first encountered an error when setting up Ray for the server and that was because CuPy did not support aarch64 so I installed CuPy on the cupy main branch to get around the issue. Reference: ray-project/ray#53128
Issue
Now I am hitting another Ray error when running the server:
INFO 06-12 10:46:22 [ray_distributed_executor.py:579] RAY_CGRAPH_get_timeout is set to 300
ERROR 06-12 10:46:34 [core.py:517] EngineCore encountered a fatal error.
ERROR 06-12 10:46:34 [core.py:517] Traceback (most recent call last):
ERROR 06-12 10:46:34 [core.py:517] File "/usr/local/lib/python3.12/dist-packages/ray/dag/compiled_dag_node.py", line 2515, in _execute_until
ERROR 06-12 10:46:34 [core.py:517] result = self._dag_output_fetcher.read(timeout)
ERROR 06-12 10:46:34 [core.py:517] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-12 10:46:34 [core.py:517] File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/common.py", line 309, in read
ERROR 06-12 10:46:34 [core.py:517] outputs = self._read_list(timeout)
ERROR 06-12 10:46:34 [core.py:517] ^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-12 10:46:34 [core.py:517] File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/common.py", line 400, in _read_list
ERROR 06-12 10:46:34 [core.py:517] raise e
ERROR 06-12 10:46:34 [core.py:517] File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/common.py", line 382, in _read_list
ERROR 06-12 10:46:34 [core.py:517] result = c.read(min(remaining_timeout, iteration_timeout))
ERROR 06-12 10:46:34 [core.py:517] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-12 10:46:34 [core.py:517] File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/shared_memory_channel.py", line 776, in read
ERROR 06-12 10:46:34 [core.py:517] return self._channel_dict[self._resolve_actor_id()].read(timeout)
ERROR 06-12 10:46:34 [core.py:517] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-12 10:46:34 [core.py:517] File "/usr/local/lib/python3.12/dist-packages/ray/experimental/channel/shared_memory_channel.py", line 480, in read
ERROR 06-12 10:46:34 [core.py:517] ret = self._worker.get_objects(
ERROR 06-12 10:46:34 [core.py:517] ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-12 10:46:34 [core.py:517] File "/usr/local/lib/python3.12/dist-packages/ray/_private/worker.py", line 911, in get_objects
ERROR 06-12 10:46:34 [core.py:517] ] = self.core_worker.get_objects(
ERROR 06-12 10:46:34 [core.py:517] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 06-12 10:46:34 [core.py:517] File "python/ray/_raylet.pyx", line 3162, in ray._raylet.CoreWorker.get_objects
ERROR 06-12 10:46:34 [core.py:517] File "python/ray/includes/common.pxi", line 106, in ray._raylet.check_status
ERROR 06-12 10:46:34 [core.py:517] ray.exceptions.RayChannelTimeoutError: System error: Timed out waiting for object available to read. ObjectID: 00fc911a7833d5d08cfd39cf6b6b31a5fbea02990100000002e1f505
ERROR 06-12 10:46:34 [core.py:517]
ERROR 06-12 10:46:34 [core.py:517] The above exception was the direct cause of the following exception:
ERROR 06-12 10:46:34 [core.py:517]
ERROR 06-12 10:46:34 [core.py:517] Traceback (most recent call last):
ERROR 06-12 10:46:34 [core.py:517] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 508, in run_engine_core
ERROR 06-12 10:46:34 [core.py:517] engine_core.run_busy_loop()
ERROR 06-12 10:46:34 [core.py:517] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 535, in run_busy_loop
ERROR 06-12 10:46:34 [core.py:517] self._process_engine_step()
ERROR 06-12 10:46:34 [core.py:517] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 560, in _process_engine_step
ERROR 06-12 10:46:34 [core.py:517] outputs, model_executed = self.step_fn()
ERROR 06-12 10:46:34 [core.py:517] ^^^^^^^^^^^^^^
ERROR 06-12 10:46:34 [core.py:517] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 279, in step_with_batch_queue
ERROR 06-12 10:46:34 [core.py:517] model_output = future.result()
ERROR 06-12 10:46:34 [core.py:517] ^^^^^^^^^^^^^^^
ERROR 06-12 10:46:34 [core.py:517] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/ray_distributed_executor.py", line 25, in result
ERROR 06-12 10:46:34 [core.py:517] return self.ref.get()
ERROR 06-12 10:46:34 [core.py:517] ^^^^^^^^^^^^^^
ERROR 06-12 10:46:34 [core.py:517] File "/usr/local/lib/python3.12/dist-packages/ray/experimental/compiled_dag_ref.py", line 115, in get
ERROR 06-12 10:46:34 [core.py:517] self._dag._execute_until(
ERROR 06-12 10:46:34 [core.py:517] File "/usr/local/lib/python3.12/dist-packages/ray/dag/compiled_dag_node.py", line 2525, in _execute_until
ERROR 06-12 10:46:34 [core.py:517] raise RayChannelTimeoutError(
ERROR 06-12 10:46:34 [core.py:517] ray.exceptions.RayChannelTimeoutError: System error: If the execution is expected to take a long time, increase RAY_CGRAPH_get_timeout which is currently 10 seconds. Otherwise, this may indicate that the execution is hanging.
From the error log RAY_CGRAPH_get_timeout is supposed to be set to 300, but Ray failed saying that it hit the timeout because RAY_CGRAPH_get_timeout is 10.
I manually set RAY_CGRAPH_get_timeout to 300 and it passed. I think vLLM did not successfully set RAY_CGRAPH_get_timeout to 300.
Reproduce commands:
#!/bin/bash
#SBATCH --output=output.log
#SBATCH --nodes=2
### Give all resources to a single Ray task, ray can manage the resources internally
#SBATCH --ntasks-per-node=1
################# DON NOT CHANGE THINGS HERE UNLESS YOU KNOW WHAT YOU ARE DOING ###############
# This script is a modification to the implementation suggest by gregSchwartz18 here:
# https://github.com/ray-project/ray/issues/826#issuecomment-522116599
set -x
# Trying to set RAY_CGRAPH_get_timeout as the potential fix. If this is not set vLLM will fail.
export RAY_CGRAPH_get_timeout=300
num_gpus_per_node=$(nvidia-smi -L | wc -l)
nodes=$(scontrol show hostnames $SLURM_JOB_NODELIST) # Getting the node names
nodes_array=($nodes)
node_1=${nodes_array[0]}
ip=$(srun --nodes=1 --ntasks=1 -w $node_1 hostname --ip-address) # making redis-address
if [[ $ip == *" "* ]]; then
IFS=' ' read -ra ADDR <<<"$ip"
if [[ ${#ADDR[0]} > 16 ]]; then
ip=${ADDR[1]}
else
ip=${ADDR[0]}
fi
echo "We detect space in ip! You are using IPV6 address. We split the IPV4 address as $ip"
fi
port_head=6379
ip_head=$ip:$port_head
export ip_head
echo "IP Head: $ip_head"
export container=<container>
export hf_model="meta-llama/Llama-3.1-8B-Instruct"
export server_port=8000
export head_node_srun="srun --nodes=1 --ntasks=1 -w $node_1 --no-container-mount-home --overlap --container-name head --container-image $container"
export head_node_srun_exec="${head_node_srun}:exec"
echo "Prestarting the head container"
$head_node_srun true
echo "STARTING HEAD at $node_1"
$head_node_srun \
ray start --head --node-ip-address=$ip --port=$port_head --block --disable-usage-stats --temp-dir=/workspace/raytmp/ &
sleep 30
worker_num=$(($SLURM_JOB_NUM_NODES - 1)) #number of nodes other than the head node
for ((i = 1; i <= $worker_num; i++)); do
node_i=${nodes_array[$i]}
echo "STARTING WORKER $i at $node_i"
srun --nodes=1 --ntasks=1 -w $node_i --no-container-mount-home --container-image $container \
ray start --address $ip_head --block --disable-usage-stats &
sleep 5
done
extract_num_gpus() {
status_output=$($head_node_srun ray status)
if echo "$status_output" | grep -q "GPU"; then
num_gpus=$(echo "$status_output" | grep "GPU" | awk -F'[/. ]' '{print $4}')
echo $num_gpus
else
echo 0
fi
}
while true; do
num_gpus=$(extract_num_gpus)
if [ "$num_gpus" -eq "$(($num_gpus_per_node * $SLURM_JOB_NUM_NODES))" ]; then
break
fi
sleep 5
done
echo "Ray cluster is configured correctly!"
##############################################################################################
rm -rf $PWD/vllm_stdout.log
# server launch
#### call your code below, adjust parallelism to your desired value
$head_node_srun \
env RAY_ADDRESS=$ip_head vllm serve \
$hf_model \
--host $ip \
--port $server_port \
--tensor-parallel-size 4 \
--pipeline-parallel-size 2 \
2>&1 > $PWD/vllm_stdout.log &
# server health check
$head_node_srun server_health --host $ip --port $server_port --endpoint "/health" --process-names "vllm serve" --timeout 15 --pid-timeout 60
# client
# Send requests to the ip and port here
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.