[fix] fix rank setup, solve the problem that multiple processes cannot be started, test multi-node training

[czy97]

Compare the log between multi-node and single-node training (above single node):

To train the code on multi-node, we can directly change the line
torchrun --standalone --nnodes=1 --nproc_per_node=$num_gpus
in run.sh to
torchrun --nnodes $node_num --nproc_per_node $num_process_per_node --rdzv_backend=c10d --rdzv_endpoint=ip_address_of_one_machine:port

• node_num: the number of the node (machine)
• num_process_per_node: the process number running on the machine
• ip_address_of_one_machine: randomly select one machine for this setup
• port: the port for communication

The above command should be run individually on each machine (node).
For more detailed information, refer to MULTINODE TRAINING.

[JiJiJiang]

Well done!