Question about nvidia-peermem.ko

[sakaia]

The template below is mostly useful for bug reports and support questions. Feel free to remove anything which doesn't apply to you and add more information where it makes sense.

1. Quick Debug Checklist

• Are you running on an Ubuntu 18.04 node?
• Are you running Kubernetes v1.13+?
• Are you running Docker (>= 18.06) or CRIO (>= 1.13+)?
• Do you have i2c_core and ipmi_msghandler loaded on the nodes?
• Did you apply the CRD (kubectl describe clusterpolicies --all-namespaces)

1. Issue or feature description

Question about nvidia-peermem.ko

In my understanging, nvidia-peermem.ko is not installed by CUDA Driver Container.
If I want to install nvidia-peermem.ko, Which way I should do

1)Install nvidia-peermem.ko and use CUDA Container Driver.
2) Install CUDA-Driver/NVIDIA-Driver on host.

If method 1 is available, would you provide me a instruction of nvidia-peermem.ko

References
　https://docs.nvidia.com/cuda/gpudirect-rdma/index.html#nvidia-peermem
　https://gitlab.com/nvidia/container-images/driver/-/blob/master/ubuntu20.04/nvidia-driver#L201-L204

2. Steps to reproduce the issue

3. Information to attach (optional if deemed irrelevant)

• kubernetes pods status: kubectl get pods --all-namespaces

• kubernetes daemonset status: kubectl get ds --all-namespaces

• If a pod/ds is in an error state or pending state kubectl describe pod -n NAMESPACE POD_NAME

• If a pod/ds is in an error state or pending state kubectl logs -n NAMESPACE POD_NAME

• Output of running a container on the GPU machine: docker run -it alpine echo foo

• Docker configuration file: cat /etc/docker/daemon.json

• Docker runtime configuration: docker info | grep runtime

• NVIDIA shared directory: ls -la /run/nvidia

• NVIDIA packages directory: ls -la /usr/local/nvidia/toolkit

• NVIDIA driver directory: ls -la /run/nvidia/driver

• kubelet logs journalctl -u kubelet > kubelet.logs

[elezar]

@sakaia release 1.8.0 of the GPU operator includes nvpeermem support. This does, however from the release notes:

This feature is available only when used with R470 drivers on Ubuntu 20.04 LTS.

I think @cdesiniotis or @shivamerla may be able to point you to additional documentation on how to get started if you meet these requirements.

[shivamerla]

@sakaia Yes, we do support installing nvidia-peermem.ko through driver container starting from 470.57.02 driver for Ubuntu20.04. In-order to use this feature you need to install GPU Operator with --set driver.rdma.enabled=true. This will be only loaded on systems with GPU + Mellanox cards. Also, note that you need to install MOFED drivers through NVIDIA Network Operator and not directly on the host. We currently don't support building/loading nvidia-peermem with MOFED drivers directly installed on host.

[sakaia]

Thank you for your reply. I’ts helpful.