nccl does not work on multiple devices

[junet123123]

os:ubuntu 20.04
NCCL version 2.14.3+cuda11.7
pytorch '1.13.1+cu117'
Python 3.8.10

device0 command:
export NCCL_DEBUG=INFO
export NCCL_SOCKET_IFNAME=enp129s0f0
export NCCL_DEBUG_SUBSYS=ENV
torchrun --nproc_per_node=1 --nnodes=2 --node_rank=0 --master_addr="10.0.62.153" --master_port "9678" test.py

device0 output:
Model loaded succeed
No protocol specified
0
28433
ubuntu-SYS-4028GR-TR:25203:25203 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to enp129s0f0
NCCL version 2.14.3+cuda11.7
ubuntu-SYS-4028GR-TR:25203:25332 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to enp129s0f0
ubuntu-SYS-4028GR-TR:25203:25332 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to enp129s0f0

device1 command:
export NCCL_DEBUG=INFO
export NCCL_SOCKET_IFNAME=enp129s0f0
export NCCL_DEBUG_SUBSYS=ENV
torchrun --nproc_per_node=1 --nnodes=2 --node_rank=1 --master_addr="10.0.62.153" --master_port "9678" test.py

device1 output
Model loaded succeed
No protocol specified
0
28433
ubuntu1-NF5588M4S:11567:11567 [0] NCCL INFO Bootstrap : Using enp129s0f0:10.0.119.255<0>
ubuntu1-NF5588M4S:11567:11567 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
ubuntu1-NF5588M4S:11567:11567 [0] NCCL INFO cudaDriverVersion 12000
NCCL version 2.14.3+cuda11.7
ubuntu1-NF5588M4S:11567:11697 [0] NCCL INFO NET/IB : No device found.
ubuntu1-NF5588M4S:11567:11697 [0] NCCL INFO NET/Socket : Using [0]enp129s0f0:10.0.119.255<0>
ubuntu1-NF5588M4S:11567:11697 [0] NCCL INFO Using network Socket

NCCL not work while device output: NCCL INFO Using network Socket
I waited a long time and the program still not new output
Firewalls are not activated on either device
Tow devices can ping for each other

How can I solve this problem? thx.

[junet123123]

nvcc info:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

[sjeaugey]

Can you provide the full NCCL log? There should be more NCCL INFO lines after that. Are we reaching the end of NCCL init or are we stopping before that?

[junet123123]

Thank you for your reply.I don't know how to provide more detailed info. Can you give me some tips?
when i interrupt the program,the program output:
Model loaded succeed
0
28433
ubuntu1-NF5588M4S:13063:13063 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to enp129s0f0
NCCL version 2.14.3+cuda11.7
ubuntu1-NF5588M4S:13063:13192 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
ubuntu1-NF5588M4S:13063:13192 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to enp129s0f0
^C
WARNING:torch.distributed.elastic.agent.server.api:Received 2 death signal, shutting down workers
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 13063 closing signal SIGINT
WARNING:torch.distributed.elastic.multiprocessing.api:Unable to shutdown process 13063 via 2, forcefully exitting via 9
Traceback (most recent call last):
File "/home/ubuntu1/work/fsdp/fsdp/.env/bin/torchrun", line 8, in
sys.exit(main())
File "/home/ubuntu1/work/fsdp/fsdp/.env/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper
return f(*args, **kwargs)
File "/home/ubuntu1/work/fsdp/fsdp/.env/lib/python3.8/site-packages/torch/distributed/run.py", line 762, in main
run(args)
File "/home/ubuntu1/work/fsdp/fsdp/.env/lib/python3.8/site-packages/torch/distributed/run.py", line 753, in run
elastic_launch(
File "/home/ubuntu1/work/fsdp/fsdp/.env/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 132, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/ubuntu1/work/fsdp/fsdp/.env/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 237, in launch_agent
result = agent.run()
File "/home/ubuntu1/work/fsdp/fsdp/.env/lib/python3.8/site-packages/torch/distributed/elastic/metrics/api.py", line 129, in wrapper
result = f(*args, **kwargs)
File "/home/ubuntu1/work/fsdp/fsdp/.env/lib/python3.8/site-packages/torch/distributed/elastic/agent/server/api.py", line 709, in run
result = self._invoke_run(role)
File "/home/ubuntu1/work/fsdp/fsdp/.env/lib/python3.8/site-packages/torch/distributed/elastic/agent/server/api.py", line 850, in _invoke_run
time.sleep(monitor_interval)
File "/home/ubuntu1/work/fsdp/fsdp/.env/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 62, in _terminate_process_handler
raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
torch.distributed.elastic.multiprocessing.api.SignalException: Process 12995 got signal: 2

[sjeaugey]

I see. It looks like we're not stuck in NCCL but somewhere else in PyTorch, although I'm not very good at recognizing PyTorch issues so I could be wrong. Perhaps you should start by asking on the PyTorch project and move back here if you see a backtrace pointing more clearly at NCCL?

[junet123123]

I see. It looks like we're not stuck in NCCL but somewhere else in PyTorch, although I'm not very good at recognizing PyTorch issues so I could be wrong. Perhaps you should start by asking on the PyTorch project and move back here if you see a backtrace pointing more clearly at NCCL?

Thank you! I'll ask this question in the pytorch project.