Server socket error when training while another task already run

[robotnc]

Describe the bug

I have a server with 4GPU gtx1080 ubuntu 16.4

When I run train process using run.sh, if already another train task was already running, it will occur error:

Start training ...
[W socket.cpp:401] [c10d] The server socket has failed to bind to [::]:29400 (errno: 98 - Address already in use).
[W socket.cpp:401] [c10d] The server socket has failed to bind to 0.0.0.0:29400 (errno: 98 - Address already in use).
[E socket.cpp:435] [c10d] The server socket has failed to listen on any local network address.

How to solve this case ?

[mlxu995]

this (https://zhuanlan.zhihu.com/p/439077183) may help you

[robotnc]

not work.

[robotnc]

Found the solution. If run in single machine multiple different task, you can set in the run.sh:

export LOCAL_RANK=0
export WORLD_SIZE=1

#and replace torchrun to python

#torchrun --standalone --nnodes=1 --nproc_per_node=$num_gpus
python
wekws/bin/train.py --gpus $gpus
--config $config
--train_data data/train/enc_data.list
--cv_data data/dev/enc_data.list \