Why setting NCCL_NET_GDR_READ to 0 perform better than seeting NCCL_NET_GDR_READ to 1 on PCI-E platform with multiple nodes?

[shanleo2024]

Hi @sjeaugey, I have one question as like the title.

I have tested the case of setting NCCL_NET_GDR_READ to 0 and setting NCCL_NET_GDR_READ to 1 on my PCI-E platform with multiple nodes, the rccl_test performs better when setting NCCL_NET_GDR_READ to 0.
The official documents also shows that: https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-net-gdr-read
I wonder why? As I know, the rccl_test should perform better with GDR than without GDR, this is the advantage of GDR.
On the PCI-E platform, the send side will not open GDR with setting NCCL_NET_GDR_READ to 0, while the recv side will open GDR wiht setting NCCL_NET_GDR_READ to 1, is this how it is implemented?

Wish your response to help me to solve my trouble.
Thank you very much.