enable Infiniband suppport with openmpi (ethernet setup)

[puneet336]

Hi ,
We have an HPC deployment scenario as follows:

Initially, we will bring the cluster online using only the ethernet interface (management network). Subsequently, after approximately 7 days, we will introduce InfiniBand switches into the environment and proceed to install IB-related drivers, adapters, etc., on the compute and management servers.

The question is: if we install OpenMPI after step 1 (when only the ethernet interface is present), will OpenMPI automatically detect and utilize the InfiniBand interface once it becomes available in future? Is there any specific flag or configuration we need to set during installation to enable IB support in advance?

prte_info.txt

or will it be necessary to reinstall OpenMPI after the InfiniBand switches and drivers are deployed onto the server?

[puneet336]

from the following 2 links,
https://docs.open-mpi.org/en/v5.0.7/tuning-apps/networking/ib-and-roce.html
https://docs.open-mpi.org/en/main/release-notes/networks.html

here is what i see -

1. for openmpi 5:
   In the Open MPI v5.0.x series, InfiniBand and RoCE devices are supported via the UCX (ucx) PML.
   Once Open MPI is configured to use UCX, the runtime will automatically select the ucx PML if one of the supported networks is detected (e.g., InfiniBand). In this case, the network port with the highest bandwidth on the system will be used for inter-node communication, and shared memory will be used for intra-node communication.

Here my understanding is that we can install ucx then ucx can autodetect and use infiniband. But I see that ucx has to be compiled and verifications are needed -
https://ucx-py.readthedocs.io/en/latest/ucx-debug.html

user@pdgx:~$ ucx_info -d | grep -i rdma
# Memory domain: rdmacm
#     Component: rdmacm
# Connection manager: rdmacm

i am not sure if ucx has a flag where in we can enable rdmacm without infiniband drivers and hardware.

2. for openmpi v4
   Prior versions of Open MPI also included the openib BTL for InfiniBand and RoCE devices.
   Here i think we can use wither UCX or --with-openib to enable infiniband support?
   Here as well, I am not sure if without the infiniband drivers we can enable IB support.

[bosilca]

If you install all required headers and libraries for verbs, UCX will buld with them but will disable them at runtime. When you finally get your IB switches in, make sure the necessary modules are loaded and everything else should be automatically taken care of.

[puneet336]

Thank you for the response @bosilca
We have RockyLinux 9 and RHEL 9 OS in our environment,
installation of following libraries would be sufficient across all login and compute nodes to enable IB support for UCX?
UCX is planned to be installed at a common NFS share path (and available as tcl module)

yum install rdma-core libibverbs libibverbs-utils librdmacm libibumad opensm infiniband-diags librdmacm-utils

[bosilca]

I don't think you need opensm on the compute nodes, only on the master SM. I'm not doing system management so take my advice with a grain a salt, but these are indeed the packges I use in my containers.

[puneet336]

Thank you for the suggestion!