nccl/v2.14/nccl-tests/build_mpi$ mpirun -x LD_LIBRARY_PATH=nccl/v2.14/nccl-build/lib -x NCCL_DEBUG=info -x NCCL_DEBUG_FILE= -H localhost:8 -bind-to none -map-by slot -mca pml ob1 -mca btl ^openib -mca btl_tcp_if_include 10.20.0.0/16 -np 8 ./reduce_scatter_perf -b 1417M -e 1417M -f 1 -g 1 -c 0 # nThread 1 nGpus 1 minBytes 1485832192 maxBytes 1485832192 step: 1048576(bytes) warmup iters: 5 iters: 20 agg iters: 1 validation: 0 graph: 0 # # Using devices # Rank 0 Group 0 Pid 3254803 on h710 device 0 [0x01] NVIDIA A40 # Rank 1 Group 0 Pid 3254804 on h710 device 1 [0x25] NVIDIA A40 # Rank 2 Group 0 Pid 3254805 on h710 device 2 [0x41] NVIDIA A40 # Rank 3 Group 0 Pid 3254806 on h710 device 3 [0x61] NVIDIA A40 # Rank 4 Group 0 Pid 3254808 on h710 device 4 [0x81] NVIDIA A40 # Rank 5 Group 0 Pid 3254810 on h710 device 5 [0xa1] NVIDIA A40 # Rank 6 Group 0 Pid 3254811 on h710 device 6 [0xc1] NVIDIA A40 # Rank 7 Group 0 Pid 3254812 on h710 device 7 [0xe1] NVIDIA A40 h710:3254803:3254803 [0] NCCL INFO Bootstrap : Using ibs8f0:10.21.128.131<0> h710:3254803:3254803 [0] NCCL INFO NET/Plugin : Plugin load returned 17 : libnccl-net.so: cannot open shared object file: No such file or directory. h710:3254803:3254803 [0] NCCL INFO cudaDriverVersion 11040 NCCL version 2.14.3+cuda11.2 h710:3254803:3255072 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibs8f0:10.21.128.131<0> h710:3254803:3255072 [0] NCCL INFO Using network IB h710:3254806:3254806 [3] NCCL INFO cudaDriverVersion 11040 h710:3254812:3254812 [7] NCCL INFO cudaDriverVersion 11040 h710:3254811:3254811 [6] NCCL INFO cudaDriverVersion 11040 h710:3254804:3254804 [1] NCCL INFO cudaDriverVersion 11040 h710:3254808:3254808 [4] NCCL INFO cudaDriverVersion 11040 h710:3254805:3254805 [2] NCCL INFO cudaDriverVersion 11040 h710:3254810:3254810 [5] NCCL INFO cudaDriverVersion 11040 h710:3254812:3254812 [7] NCCL INFO Bootstrap : Using ibs8f0:10.21.128.131<0> h710:3254812:3254812 [7] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation h710:3254806:3254806 [3] NCCL INFO Bootstrap : Using ibs8f0:10.21.128.131<0> h710:3254806:3254806 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation h710:3254811:3254811 [6] NCCL INFO Bootstrap : Using ibs8f0:10.21.128.131<0> h710:3254811:3254811 [6] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation h710:3254806:3255205 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibs8f0:10.21.128.131<0> h710:3254806:3255205 [3] NCCL INFO Using network IB h710:3254812:3255204 [7] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibs8f0:10.21.128.131<0> h710:3254812:3255204 [7] NCCL INFO Using network IB h710:3254811:3255206 [6] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibs8f0:10.21.128.131<0> h710:3254811:3255206 [6] NCCL INFO Using network IB h710:3254808:3254808 [4] NCCL INFO Bootstrap : Using ibs8f0:10.21.128.131<0> h710:3254808:3254808 [4] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation h710:3254808:3255212 [4] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibs8f0:10.21.128.131<0> h710:3254808:3255212 [4] NCCL INFO Using network IB h710:3254805:3254805 [2] NCCL INFO Bootstrap : Using ibs8f0:10.21.128.131<0> h710:3254805:3254805 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation h710:3254804:3254804 [1] NCCL INFO Bootstrap : Using ibs8f0:10.21.128.131<0> h710:3254804:3254804 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation h710:3254810:3254810 [5] NCCL INFO Bootstrap : Using ibs8f0:10.21.128.131<0> h710:3254810:3254810 [5] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation h710:3254805:3255215 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibs8f0:10.21.128.131<0> h710:3254805:3255215 [2] NCCL INFO Using network IB h710:3254804:3255217 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibs8f0:10.21.128.131<0> h710:3254804:3255217 [1] NCCL INFO Using network IB h710:3254810:3255219 [5] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibs8f0:10.21.128.131<0> h710:3254810:3255219 [5] NCCL INFO Using network IB h710:3254803:3255072 [0] NCCL INFO Setting affinity for GPU 0 to f000,0000f000 h710:3254810:3255219 [5] NCCL INFO Setting affinity for GPU 5 to 0f000000,0f000000 h710:3254805:3255215 [2] NCCL INFO Setting affinity for GPU 2 to f0,000000f0 h710:3254806:3255205 [3] NCCL INFO Setting affinity for GPU 3 to 0f,0000000f h710:3254808:3255212 [4] NCCL INFO Setting affinity for GPU 4 to f0000000,f0000000 h710:3254811:3255206 [6] NCCL INFO Setting affinity for GPU 6 to f00000,00f00000 h710:3254804:3255217 [1] NCCL INFO Setting affinity for GPU 1 to 0f00,00000f00 h710:3254812:3255204 [7] NCCL INFO Setting affinity for GPU 7 to 0f0000,000f0000 h710:3254803:3255072 [0] NCCL INFO Channel 00/08 : 0 1 2 3 4 5 6 7 h710:3254803:3255072 [0] NCCL INFO Channel 01/08 : 0 1 2 3 4 5 6 7 h710:3254803:3255072 [0] NCCL INFO Channel 02/08 : 0 3 2 5 4 7 6 1 h710:3254803:3255072 [0] NCCL INFO Channel 03/08 : 0 3 2 5 4 7 6 1 h710:3254803:3255072 [0] NCCL INFO Channel 04/08 : 0 1 2 3 4 5 6 7 h710:3254803:3255072 [0] NCCL INFO Channel 05/08 : 0 1 2 3 4 5 6 7 h710:3254803:3255072 [0] NCCL INFO Channel 06/08 : 0 3 2 5 4 7 6 1 h710:3254803:3255072 [0] NCCL INFO Channel 07/08 : 0 3 2 5 4 7 6 1 h710:3254803:3255072 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 3/-1/-1->0->-1 [3] 3/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 3/-1/-1->0->-1 [7] 3/-1/-1->0->-1 h710:3254811:3255206 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 1/-1/-1->6->7 [3] 1/-1/-1->6->7 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 1/-1/-1->6->7 [7] 1/-1/-1->6->7 h710:3254806:3255205 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 2/-1/-1->3->0 [3] 2/-1/-1->3->0 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 2/-1/-1->3->0 [7] 2/-1/-1->3->0 h710:3254810:3255219 [5] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 4/-1/-1->5->2 [3] 4/-1/-1->5->2 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 4/-1/-1->5->2 [7] 4/-1/-1->5->2 h710:3254805:3255215 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 5/-1/-1->2->3 [3] 5/-1/-1->2->3 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 5/-1/-1->2->3 [7] 5/-1/-1->2->3 h710:3254812:3255204 [7] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] 6/-1/-1->7->4 [3] 6/-1/-1->7->4 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] 6/-1/-1->7->4 [7] 6/-1/-1->7->4 h710:3254804:3255217 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] -1/-1/-1->1->6 [3] -1/-1/-1->1->6 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] -1/-1/-1->1->6 [7] -1/-1/-1->1->6 h710:3254808:3255212 [4] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 7/-1/-1->4->5 [3] 7/-1/-1->4->5 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 7/-1/-1->4->5 [7] 7/-1/-1->4->5 h710:3254810:3255219 [5] NCCL INFO Channel 00/0 : 5[a1000] -> 6[c1000] via P2P/IPC h710:3254810:3255219 [5] NCCL INFO Channel 01/0 : 5[a1000] -> 6[c1000] via P2P/IPC h710:3254812:3255204 [7] NCCL INFO Channel 00/0 : 7[e1000] -> 0[1000] via P2P/IPC h710:3254810:3255219 [5] NCCL INFO Channel 04/0 : 5[a1000] -> 6[c1000] via P2P/IPC h710:3254803:3255072 [0] NCCL INFO Channel 00/0 : 0[1000] -> 1[25000] via P2P/IPC h710:3254804:3255217 [1] NCCL INFO Channel 00/0 : 1[25000] -> 2[41000] via P2P/IPC h710:3254811:3255206 [6] NCCL INFO Channel 00/0 : 6[c1000] -> 7[e1000] via P2P/IPC h710:3254808:3255212 [4] NCCL INFO Channel 00/0 : 4[81000] -> 5[a1000] via P2P/IPC h710:3254806:3255205 [3] NCCL INFO Channel 00/0 : 3[61000] -> 4[81000] via P2P/IPC h710:3254812:3255204 [7] NCCL INFO Channel 01/0 : 7[e1000] -> 0[1000] via P2P/IPC h710:3254805:3255215 [2] NCCL INFO Channel 00/0 : 2[41000] -> 3[61000] via P2P/IPC h710:3254803:3255072 [0] NCCL INFO Channel 01/0 : 0[1000] -> 1[25000] via P2P/IPC h710:3254810:3255219 [5] NCCL INFO Channel 05/0 : 5[a1000] -> 6[c1000] via P2P/IPC h710:3254804:3255217 [1] NCCL INFO Channel 01/0 : 1[25000] -> 2[41000] via P2P/IPC h710:3254811:3255206 [6] NCCL INFO Channel 01/0 : 6[c1000] -> 7[e1000] via P2P/IPC h710:3254808:3255212 [4] NCCL INFO Channel 01/0 : 4[81000] -> 5[a1000] via P2P/IPC h710:3254806:3255205 [3] NCCL INFO Channel 01/0 : 3[61000] -> 4[81000] via P2P/IPC h710:3254812:3255204 [7] NCCL INFO Channel 04/0 : 7[e1000] -> 0[1000] via P2P/IPC h710:3254805:3255215 [2] NCCL INFO Channel 01/0 : 2[41000] -> 3[61000] via P2P/IPC h710:3254803:3255072 [0] NCCL INFO Channel 04/0 : 0[1000] -> 1[25000] via P2P/IPC h710:3254804:3255217 [1] NCCL INFO Channel 04/0 : 1[25000] -> 2[41000] via P2P/IPC h710:3254811:3255206 [6] NCCL INFO Channel 04/0 : 6[c1000] -> 7[e1000] via P2P/IPC h710:3254808:3255212 [4] NCCL INFO Channel 04/0 : 4[81000] -> 5[a1000] via P2P/IPC h710:3254806:3255205 [3] NCCL INFO Channel 04/0 : 3[61000] -> 4[81000] via P2P/IPC h710:3254812:3255204 [7] NCCL INFO Channel 05/0 : 7[e1000] -> 0[1000] via P2P/IPC h710:3254805:3255215 [2] NCCL INFO Channel 04/0 : 2[41000] -> 3[61000] via P2P/IPC h710:3254803:3255072 [0] NCCL INFO Channel 05/0 : 0[1000] -> 1[25000] via P2P/IPC h710:3254804:3255217 [1] NCCL INFO Channel 05/0 : 1[25000] -> 2[41000] via P2P/IPC h710:3254811:3255206 [6] NCCL INFO Channel 05/0 : 6[c1000] -> 7[e1000] via P2P/IPC h710:3254808:3255212 [4] NCCL INFO Channel 05/0 : 4[81000] -> 5[a1000] via P2P/IPC h710:3254806:3255205 [3] NCCL INFO Channel 05/0 : 3[61000] -> 4[81000] via P2P/IPC h710:3254805:3255215 [2] NCCL INFO Channel 05/0 : 2[41000] -> 3[61000] via P2P/IPC h710:3254803:3255072 [0] NCCL INFO Channel 02/0 : 0[1000] -> 3[61000] via P2P/IPC h710:3254811:3255206 [6] NCCL INFO Channel 02/0 : 6[c1000] -> 1[25000] via P2P/IPC h710:3254808:3255212 [4] NCCL INFO Channel 02/0 : 4[81000] -> 7[e1000] via P2P/IPC h710:3254805:3255215 [2] NCCL INFO Channel 02/0 : 2[41000] -> 5[a1000] via P2P/IPC h710:3254811:3255206 [6] NCCL INFO Channel 03/0 : 6[c1000] -> 1[25000] via P2P/IPC h710:3254803:3255072 [0] NCCL INFO Channel 03/0 : 0[1000] -> 3[61000] via P2P/IPC h710:3254808:3255212 [4] NCCL INFO Channel 03/0 : 4[81000] -> 7[e1000] via P2P/IPC h710:3254805:3255215 [2] NCCL INFO Channel 03/0 : 2[41000] -> 5[a1000] via P2P/IPC h710:3254808:3255212 [4] NCCL INFO Channel 06/0 : 4[81000] -> 7[e1000] via P2P/IPC h710:3254811:3255206 [6] NCCL INFO Channel 06/0 : 6[c1000] -> 1[25000] via P2P/IPC h710:3254805:3255215 [2] NCCL INFO Channel 06/0 : 2[41000] -> 5[a1000] via P2P/IPC h710:3254803:3255072 [0] NCCL INFO Channel 06/0 : 0[1000] -> 3[61000] via P2P/IPC h710:3254811:3255206 [6] NCCL INFO Channel 07/0 : 6[c1000] -> 1[25000] via P2P/IPC h710:3254808:3255212 [4] NCCL INFO Channel 07/0 : 4[81000] -> 7[e1000] via P2P/IPC h710:3254805:3255215 [2] NCCL INFO Channel 07/0 : 2[41000] -> 5[a1000] via P2P/IPC h710:3254803:3255072 [0] NCCL INFO Channel 07/0 : 0[1000] -> 3[61000] via P2P/IPC h710:3254812:3255204 [7] NCCL INFO Channel 02/0 : 7[e1000] -> 6[c1000] via P2P/IPC h710:3254812:3255204 [7] NCCL INFO Channel 03/0 : 7[e1000] -> 6[c1000] via P2P/IPC h710:3254810:3255219 [5] NCCL INFO Channel 02/0 : 5[a1000] -> 4[81000] via P2P/IPC h710:3254812:3255204 [7] NCCL INFO Channel 06/0 : 7[e1000] -> 6[c1000] via P2P/IPC h710:3254810:3255219 [5] NCCL INFO Channel 03/0 : 5[a1000] -> 4[81000] via P2P/IPC h710:3254804:3255217 [1] NCCL INFO Channel 02/0 : 1[25000] -> 0[1000] via P2P/IPC h710:3254812:3255204 [7] NCCL INFO Channel 07/0 : 7[e1000] -> 6[c1000] via P2P/IPC h710:3254810:3255219 [5] NCCL INFO Channel 06/0 : 5[a1000] -> 4[81000] via P2P/IPC h710:3254804:3255217 [1] NCCL INFO Channel 03/0 : 1[25000] -> 0[1000] via P2P/IPC h710:3254806:3255205 [3] NCCL INFO Channel 02/0 : 3[61000] -> 2[41000] via P2P/IPC h710:3254810:3255219 [5] NCCL INFO Channel 07/0 : 5[a1000] -> 4[81000] via P2P/IPC h710:3254804:3255217 [1] NCCL INFO Channel 06/0 : 1[25000] -> 0[1000] via P2P/IPC h710:3254806:3255205 [3] NCCL INFO Channel 03/0 : 3[61000] -> 2[41000] via P2P/IPC h710:3254804:3255217 [1] NCCL INFO Channel 07/0 : 1[25000] -> 0[1000] via P2P/IPC h710:3254806:3255205 [3] NCCL INFO Channel 06/0 : 3[61000] -> 2[41000] via P2P/IPC h710:3254806:3255205 [3] NCCL INFO Channel 07/0 : 3[61000] -> 2[41000] via P2P/IPC h710:3254808:3255212 [4] NCCL INFO Connected all rings h710:3254808:3255212 [4] NCCL INFO Channel 02/0 : 4[81000] -> 5[a1000] via P2P/IPC h710:3254810:3255219 [5] NCCL INFO Connected all rings h710:3254808:3255212 [4] NCCL INFO Channel 03/0 : 4[81000] -> 5[a1000] via P2P/IPC h710:3254811:3255206 [6] NCCL INFO Connected all rings h710:3254811:3255206 [6] NCCL INFO Channel 02/0 : 6[c1000] -> 7[e1000] via P2P/IPC h710:3254812:3255204 [7] NCCL INFO Connected all rings h710:3254808:3255212 [4] NCCL INFO Channel 06/0 : 4[81000] -> 5[a1000] via P2P/IPC h710:3254811:3255206 [6] NCCL INFO Channel 03/0 : 6[c1000] -> 7[e1000] via P2P/IPC h710:3254805:3255215 [2] NCCL INFO Connected all rings h710:3254805:3255215 [2] NCCL INFO Channel 02/0 : 2[41000] -> 3[61000] via P2P/IPC h710:3254806:3255205 [3] NCCL INFO Connected all rings h710:3254808:3255212 [4] NCCL INFO Channel 07/0 : 4[81000] -> 5[a1000] via P2P/IPC h710:3254804:3255217 [1] NCCL INFO Connected all rings h710:3254804:3255217 [1] NCCL INFO Channel 02/0 : 1[25000] -> 6[c1000] via P2P/IPC h710:3254803:3255072 [0] NCCL INFO Connected all rings h710:3254811:3255206 [6] NCCL INFO Channel 06/0 : 6[c1000] -> 7[e1000] via P2P/IPC h710:3254805:3255215 [2] NCCL INFO Channel 03/0 : 2[41000] -> 3[61000] via P2P/IPC h710:3254804:3255217 [1] NCCL INFO Channel 03/0 : 1[25000] -> 6[c1000] via P2P/IPC h710:3254811:3255206 [6] NCCL INFO Channel 07/0 : 6[c1000] -> 7[e1000] via P2P/IPC h710:3254805:3255215 [2] NCCL INFO Channel 06/0 : 2[41000] -> 3[61000] via P2P/IPC h710:3254804:3255217 [1] NCCL INFO Channel 06/0 : 1[25000] -> 6[c1000] via P2P/IPC h710:3254805:3255215 [2] NCCL INFO Channel 07/0 : 2[41000] -> 3[61000] via P2P/IPC h710:3254804:3255217 [1] NCCL INFO Channel 07/0 : 1[25000] -> 6[c1000] via P2P/IPC h710:3254810:3255219 [5] NCCL INFO Channel 02/0 : 5[a1000] -> 2[41000] via P2P/IPC h710:3254810:3255219 [5] NCCL INFO Channel 03/0 : 5[a1000] -> 2[41000] via P2P/IPC h710:3254812:3255204 [7] NCCL INFO Channel 02/0 : 7[e1000] -> 4[81000] via P2P/IPC h710:3254810:3255219 [5] NCCL INFO Channel 06/0 : 5[a1000] -> 2[41000] via P2P/IPC h710:3254812:3255204 [7] NCCL INFO Channel 03/0 : 7[e1000] -> 4[81000] via P2P/IPC h710:3254806:3255205 [3] NCCL INFO Channel 02/0 : 3[61000] -> 0[1000] via P2P/IPC h710:3254810:3255219 [5] NCCL INFO Channel 07/0 : 5[a1000] -> 2[41000] via P2P/IPC h710:3254812:3255204 [7] NCCL INFO Channel 06/0 : 7[e1000] -> 4[81000] via P2P/IPC h710:3254806:3255205 [3] NCCL INFO Channel 03/0 : 3[61000] -> 0[1000] via P2P/IPC h710:3254812:3255204 [7] NCCL INFO Channel 07/0 : 7[e1000] -> 4[81000] via P2P/IPC h710:3254806:3255205 [3] NCCL INFO Channel 06/0 : 3[61000] -> 0[1000] via P2P/IPC h710:3254806:3255205 [3] NCCL INFO Channel 07/0 : 3[61000] -> 0[1000] via P2P/IPC h710:3254812:3255204 [7] NCCL INFO Channel 00/0 : 7[e1000] -> 6[c1000] via P2P/IPC h710:3254812:3255204 [7] NCCL INFO Channel 01/0 : 7[e1000] -> 6[c1000] via P2P/IPC h710:3254812:3255204 [7] NCCL INFO Channel 04/0 : 7[e1000] -> 6[c1000] via P2P/IPC h710:3254812:3255204 [7] NCCL INFO Channel 05/0 : 7[e1000] -> 6[c1000] via P2P/IPC h710:3254808:3255212 [4] NCCL INFO Channel 00/0 : 4[81000] -> 3[61000] via P2P/IPC h710:3254811:3255206 [6] NCCL INFO Channel 00/0 : 6[c1000] -> 5[a1000] via P2P/IPC h710:3254804:3255217 [1] NCCL INFO Channel 00/0 : 1[25000] -> 0[1000] via P2P/IPC h710:3254808:3255212 [4] NCCL INFO Channel 01/0 : 4[81000] -> 3[61000] via P2P/IPC h710:3254811:3255206 [6] NCCL INFO Channel 01/0 : 6[c1000] -> 5[a1000] via P2P/IPC h710:3254804:3255217 [1] NCCL INFO Channel 01/0 : 1[25000] -> 0[1000] via P2P/IPC h710:3254806:3255205 [3] NCCL INFO Channel 00/0 : 3[61000] -> 2[41000] via P2P/IPC h710:3254805:3255215 [2] NCCL INFO Channel 00/0 : 2[41000] -> 1[25000] via P2P/IPC h710:3254810:3255219 [5] NCCL INFO Channel 00/0 : 5[a1000] -> 4[81000] via P2P/IPC h710:3254808:3255212 [4] NCCL INFO Channel 04/0 : 4[81000] -> 3[61000] via P2P/IPC h710:3254811:3255206 [6] NCCL INFO Channel 04/0 : 6[c1000] -> 5[a1000] via P2P/IPC h710:3254804:3255217 [1] NCCL INFO Channel 04/0 : 1[25000] -> 0[1000] via P2P/IPC h710:3254806:3255205 [3] NCCL INFO Channel 01/0 : 3[61000] -> 2[41000] via P2P/IPC h710:3254805:3255215 [2] NCCL INFO Channel 01/0 : 2[41000] -> 1[25000] via P2P/IPC h710:3254810:3255219 [5] NCCL INFO Channel 01/0 : 5[a1000] -> 4[81000] via P2P/IPC h710:3254811:3255206 [6] NCCL INFO Channel 05/0 : 6[c1000] -> 5[a1000] via P2P/IPC h710:3254808:3255212 [4] NCCL INFO Channel 05/0 : 4[81000] -> 3[61000] via P2P/IPC h710:3254804:3255217 [1] NCCL INFO Channel 05/0 : 1[25000] -> 0[1000] via P2P/IPC h710:3254806:3255205 [3] NCCL INFO Channel 04/0 : 3[61000] -> 2[41000] via P2P/IPC h710:3254805:3255215 [2] NCCL INFO Channel 04/0 : 2[41000] -> 1[25000] via P2P/IPC h710:3254810:3255219 [5] NCCL INFO Channel 04/0 : 5[a1000] -> 4[81000] via P2P/IPC h710:3254805:3255215 [2] NCCL INFO Channel 05/0 : 2[41000] -> 1[25000] via P2P/IPC h710:3254806:3255205 [3] NCCL INFO Channel 05/0 : 3[61000] -> 2[41000] via P2P/IPC h710:3254810:3255219 [5] NCCL INFO Channel 05/0 : 5[a1000] -> 4[81000] via P2P/IPC h710:3254812:3255204 [7] NCCL INFO Connected all trees h710:3254812:3255204 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 h710:3254812:3255204 [7] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer h710:3254803:3255072 [0] NCCL INFO Connected all trees h710:3254803:3255072 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 h710:3254803:3255072 [0] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer h710:3254811:3255206 [6] NCCL INFO Connected all trees h710:3254811:3255206 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 h710:3254811:3255206 [6] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer h710:3254804:3255217 [1] NCCL INFO Connected all trees h710:3254804:3255217 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 h710:3254804:3255217 [1] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer h710:3254808:3255212 [4] NCCL INFO Connected all trees h710:3254808:3255212 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 h710:3254808:3255212 [4] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer h710:3254805:3255215 [2] NCCL INFO Connected all trees h710:3254805:3255215 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 h710:3254805:3255215 [2] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer h710:3254810:3255219 [5] NCCL INFO Connected all trees h710:3254810:3255219 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 h710:3254810:3255219 [5] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer h710:3254806:3255205 [3] NCCL INFO Connected all trees h710:3254806:3255205 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 h710:3254806:3255205 [3] NCCL INFO 8 coll channels, 8 p2p channels, 2 p2p channels per peer h710:3254812:3255204 [7] NCCL INFO comm 0x557e7ad054c0 rank 7 nranks 8 cudaDev 7 busId e1000 - Init COMPLETE h710:3254810:3255219 [5] NCCL INFO comm 0x55a00d1ba640 rank 5 nranks 8 cudaDev 5 busId a1000 - Init COMPLETE h710:3254803:3255072 [0] NCCL INFO comm 0x559e566e7c90 rank 0 nranks 8 cudaDev 0 busId 1000 - Init COMPLETE # # out-of-place in-place # size count type redop root time algbw busbw #wrong time algbw busbw #wrong # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) h710:3254811:3255206 [6] NCCL INFO comm 0x5599b6d535c0 rank 6 nranks 8 cudaDev 6 busId c1000 - Init COMPLETE h710:3254808:3255212 [4] NCCL INFO comm 0x55bd1dbbaaf0 rank 4 nranks 8 cudaDev 4 busId 81000 - Init COMPLETE h710:3254805:3255215 [2] NCCL INFO comm 0x55d4f2099710 rank 2 nranks 8 cudaDev 2 busId 41000 - Init COMPLETE h710:3254806:3255205 [3] NCCL INFO comm 0x55cb14a304f0 rank 3 nranks 8 cudaDev 3 busId 61000 - Init COMPLETE h710:3254804:3255217 [1] NCCL INFO comm 0x55ac39a96890 rank 1 nranks 8 cudaDev 1 busId 25000 - Init COMPLETE 1485832192 46432256 float sum -1 49997 29.72 26.00 N/A 49958 29.74 26.02 N/A h710:3254803:3254803 [0] NCCL INFO comm 0x559e566e7c90 rank 0 nranks 8 cudaDev 0 busId 1000 - Destroy COMPLETE # Out of bounds values : 0 OK # Avg bus bandwidth : 26.0137 # h710:3254812:3254812 [7] NCCL INFO comm 0x557e7ad054c0 rank 7 nranks 8 cudaDev 7 busId e1000 - Destroy COMPLETE h710:3254804:3254804 [1] NCCL INFO comm 0x55ac39a96890 rank 1 nranks 8 cudaDev 1 busId 25000 - Destroy COMPLETE h710:3254811:3254811 [6] NCCL INFO comm 0x5599b6d535c0 rank 6 nranks 8 cudaDev 6 busId c1000 - Destroy COMPLETE h710:3254806:3254806 [3] NCCL INFO comm 0x55cb14a304f0 rank 3 nranks 8 cudaDev 3 busId 61000 - Destroy COMPLETE h710:3254805:3254805 [2] NCCL INFO comm 0x55d4f2099710 rank 2 nranks 8 cudaDev 2 busId 41000 - Destroy COMPLETE h710:3254808:3254808 [4] NCCL INFO comm 0x55bd1dbbaaf0 rank 4 nranks 8 cudaDev 4 busId 81000 - Destroy COMPLETE h710:3254810:3254810 [5] NCCL INFO comm 0x55a00d1ba640 rank 5 nranks 8 cudaDev 5 busId a1000 - Destroy COMPLETE