Rework the P2P/IPC/SHM container fix

[AddyLaddy]

Fixes: #155

Now we use /proc/sys/kernel/random/boot_id instead of the
hostname and NS uts,mnt info
A new env var; NCCL_HOSTID can be used to overload this string.

[AddyLaddy]

@alsrgv Here is the latest P2P/IPC/SHM fix that we are intending to add to the next NCCL release

[alsrgv]

@AddyLaddy, @sjeaugey, I'm a bit confused by this change. Before, /uts and /mnt were used to differentiate between containers, which helped to properly disable P2P/SHM/IPC as necessary:

asergeev@server:~$ readlink /proc/$$/ns/uts
uts:[4026531838]
asergeev@server:~$ nvidia-docker run -it --shm-size 1g -v `pwd`:/mnt --rm horovod/horovod:0.17.0.post1-tf1.14.0-torch1.2.0-mxnet1.5.0-py3.6
root@e8864a4a48b1:/examples# readlink /proc/$$/ns/uts
uts:[4026533300]
root@e8864a4a48b1:/examples#

How will P2P/IPC/SHM be auto-disabled between unconnected containers (e.g. just two nvidia-docker run xyz w/o any sharing) after this change?

[AddyLaddy]

Sorry I missed a component of that fix which should also check the MAJOR:MINOR of the /dev/shm device before attempting to use it in the SHM transport. I'll try and updtae the PR. It will be good to know if this is sufficient for your use case.
If not, then you could recreate the old behaviour by setting NCCL_HOSTID using a wrapper script.

[alsrgv]

What about CUDA IPC? It does not work across containers, too.

[AddyLaddy]

The motivation for this patch is to enable P2P/IPC/SHM between containers as I am told that it is typical for HPC apps to launch a container per rank. @3XX0 Is our container expert and perhaps he can comment?

[sjeaugey]

I think the rationale for IPCs is that if we are sure we are on the same node, then we can work on NVML devices and see if they can use P2P.

[3XX0]

Yes, /proc/sys/kernel/random/boot_id will check if the containers are on the same node.

Next check is the major:minor of /dev/shm to know whether or not they share the same POSIX SHM (which is required for CUDA IPCs).

Last check is cuDeviceCanAccessPeer() which will check if the GPUs are visible and support P2P.

This should handle things like:

# No P2P, no SHM
nvidia-docker run -e NVIDIA_VISIBLE_DEVICES=none horovod
nvidia-docker run -e NVIDIA_VISIBLE_DEVICES=none horovod

# No P2P, SHM possible
nvidia-docker run --name=A -e NVIDIA_VISIBLE_DEVICES=0 horovod
nvidia-docker run --name=B -e NVIDIA_VISIBLE_DEVICES=1 --ipc=container:A horovod

# P2P
nvidia-docker run -e NVIDIA_VISIBLE_DEVICES=0,1 --ipc=host horovod
nvidia-docker run -e NVIDIA_VISIBLE_DEVICES=0,1 --ipc=host horovod

[alsrgv]

Do we check cuDeviceCanAccessPeer() now? I remember CUDA IPC crashing before across containers.

[sjeaugey]

We should check it here : https://github.com/NVIDIA/nccl/blob/master/src/transport/p2p.cc#L98

[3XX0]

I remember CUDA IPC crashing before across containers.

Probably a driver issue, IIRC at one point there was an issue where you also needed to share the PID namespace for IPCs to work, but that's definitely not the case anymore.

[alsrgv]

I've tested this PR and it looks good. Thanks, this makes it much more convenient to use NCCL with various container environments!

[AddyLaddy]

Great thanks for the feedback Alex!

[sjeaugey]

Merged in 2.5.6. Closing.