Disable NVLINK for P2P transport.

[shubhamkakde111]

I am running nccl-tests (sendrecv_perf) with two MPI processes on a single node, using two NVIDIA H100 GPUs.

My goal is to use the NCCL P2P transport, but specifically over PCIe, without using NVLink. I also want to disable the shared-memory (SHM) transport.

In other words, for intra-node communication between the two GPUs, I would like NCCL to:

Enable P2P communication.
Prevent the use of NVLink.
Disable SHM transport.
Use PCIe-based GPU-to-GPU communication if possible.

Could someone please confirm the correct NCCL environment variables for this configuration and the recommended way to verify that communication is actually occurring over PCIe rather than NVLink?

For reference, I am launching the test using mpirun with two processes and one process per GPU on a single node.

[xiaofanl-nvidia]

@shubhamkakde111 To NCCL, nvlink p2p and pcie p2p are the same - CUDA would return true from cudaDeviceCanAccessPeer and then NCCL just accesses peer memory with a VA mapping.
So if you want to disable NVLINK, but keep pcie enabled, you would need some driver side toggle, not NCCL env var.
Maybe you can look around in this file if you know how to: https://github.com/NVIDIA/open-gpu-kernel-modules/blob/main/src/nvidia/interface/nvrm_registry.h
But note there is not much production level guarantee on this interface in general so you might get into untested, closed code.

[shubhamkakde111]

thanks for the reply @xiaofanl-nvidia