[5.0.0rc12] does not allow to build with ROCm, but run without ROCm

[lahwaacz]

The documentation for CUDA says that "Open MPI supports building with CUDA libraries and running on systems without CUDA libraries or hardware." I was expecting the same for ROCm support, but it does not seem to be the case at least for 5.0.0rc12 where even running ompi_info does not work if OpenMPI was built with ROCm, but installed on a system without ROCm:

$ ompi_info | grep "MPI extensions"
ompi_info: error while loading shared libraries: libamdhip64.so.5: cannot open shared object file: No such file or directory

It is because ompi_info links to libopen-pal.so.80 which links to libamdhip64.so.5.

It is also weird that the --with-rocm configure flag actually requires HIP (it checks for hip/hip_runtime.h and libamdhip64) which is a high-level abstraction layer for ROCm and CUDA.

[lahwaacz]

Actually, libopen-pal.so.80 also links to libcuda.so.1 when --with-cuda is used as of 5.0.0rc12. This was not the case with /usr/lib/libopen-pal.so.40 from OpenMPI 4.1.x so there is likely some kind of regression.

[edgargabriel]

I suspect that the issue why this doesn't work anymore is that components are not compiled as dso's anymore by default in Open MPI 5.0. I can test this scenario tomorrow.

Regarding the rocm vs. hip: ROCm is the name if the overall software stack of AMD GPUs. We are using the HIP API to implement the required functionality in Open MPI, hence the necessity to check for the hip header files. Despite of using HIP, the component should be able to detect AMD GPU buffers allocated using other methods than HIP (e.g. OpenMP buffers or similarly) correctly, hence the decision to name the component ROCm, not HIP.

While you are correct that a HIP application can also be compiled to run on Nvidia GPUs, the rocm component in Open MPI is expected to be used only for AMD GPUs. A buffer allocated with HIP but configured to create code for Nvidia GPUs should be recognized by the cuda accelerator component and there is no benefit or necessity to use the rocm component for that scenario.

[lahwaacz]

@edgargabriel I tried building with --enable-mca-dso=yes based on your suggestion, but the test suite does not pass:

=====================================================
   Open MPI 5.0.0rc12: test/threads/test-suite.log
=====================================================

# TOTAL: 3
# PASS:  2
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: opal_condition
====================

--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    No Accelerators Found
from the file:
    /usr/share/openmpi/help-accelerator-base.txt: No such file or directory
But I couldn't find that topic in the file.  Sorry!
--------------------------------------------------------------------------
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    opal_init:startup:internal-failure
from the file:
    /usr/share/openmpi/help-opal-runtime.txt: No such file or directory
But I couldn't find that topic in the file.  Sorry!
--------------------------------------------------------------------------
 Failure : Comparison failure
 Expected result: 0
 Test result: -6
SUPPORT: OMPI Test failed: opal_condition_t (1 of 1 failed)
FAIL opal_condition (exit status: 1)

Nevertheless, when I run an MPI program on a system with ROCm and without CUDA, I get the following messages from MPI_Init:

[gp8:1721709] mca_base_component_repository_open: unable to open mca_accelerator_cuda: libcuda.so.1: cannot open shared object file: No such file or directory (ignored)
[gp8:1721708] mca_base_component_repository_open: unable to open mca_accelerator_cuda: libcuda.so.1: cannot open shared object file: No such file or directory (ignored)
[gp8:1721710] mca_base_component_repository_open: unable to open mca_accelerator_cuda: libcuda.so.1: cannot open shared object file: No such file or directory (ignored)
[gp8:1721711] mca_base_component_repository_open: unable to open mca_accelerator_cuda: libcuda.so.1: cannot open shared object file: No such file or directory (ignored)
[gp8:1721708] mca_base_component_repository_open: unable to open mca_rcache_gpusm: libcuda.so.1: cannot open shared object file: No such file or directory (ignored)
[gp8:1721709] mca_base_component_repository_open: unable to open mca_rcache_gpusm: libcuda.so.1: cannot open shared object file: No such file or directory (ignored)
[gp8:1721709] mca_base_component_repository_open: unable to open mca_rcache_rgpusm: libcuda.so.1: cannot open shared object file: No such file or directory (ignored)
[gp8:1721708] mca_base_component_repository_open: unable to open mca_rcache_rgpusm: libcuda.so.1: cannot open shared object file: No such file or directory (ignored)
[gp8:1721711] mca_base_component_repository_open: unable to open mca_rcache_gpusm: libcuda.so.1: cannot open shared object file: No such file or directory (ignored)
[gp8:1721710] mca_base_component_repository_open: unable to open mca_rcache_gpusm: libcuda.so.1: cannot open shared object file: No such file or directory (ignored)
[gp8:1721711] mca_base_component_repository_open: unable to open mca_rcache_rgpusm: libcuda.so.1: cannot open shared object file: No such file or directory (ignored)
[gp8:1721710] mca_base_component_repository_open: unable to open mca_rcache_rgpusm: libcuda.so.1: cannot open shared object file: No such file or directory (ignored)

And on a system with CUDA but without ROCm:

[gp1:2287402] mca_base_component_repository_open: unable to open mca_accelerator_rocm: libamdhip64.so.5: cannot open shared object file: No such file or directory (ignored)
[gp1:2287403] mca_base_component_repository_open: unable to open mca_accelerator_rocm: libamdhip64.so.5: cannot open shared object file: No such file or directory (ignored)
[gp1:2287404] mca_base_component_repository_open: unable to open mca_accelerator_rocm: libamdhip64.so.5: cannot open shared object file: No such file or directory (ignored)
[gp1:2287405] mca_base_component_repository_open: unable to open mca_accelerator_rocm: libamdhip64.so.5: cannot open shared object file: No such file or directory (ignored)

And in both cases MPI_Finalize leads to a segfault, which did not happen without --enable-mca-dso=yes.

[lahwaacz]

When I build only components that link to CUDA or ROCm libraries as run-time loadable (--enable-mca-dso=accelerator_cuda,rcache_gpusm,rcache_rgpusm,accelerator_rocm), the segfaults from MPI_Finalize are gone and the OpenMPI test suite also passes. But the messages from MPI_Init are still not nice...

[edgargabriel]

@lahwaacz thank you, I think this is a good as it might get at the moment. I can check whether there is a way that we could get rid of the warnings at component open, but it might take me 2-3 weeks before I can get to it.

[lahwaacz]

@edgargabriel Any progress with getting rid of the warnings?

[lahwaacz]

I found out that running mpirun with --mca mca_base_component_show_load_errors '^accelerator,rcache' disables these warnings. On the other hand, building OpenMPI 5.0.2 with the --disable-show-load-errors-by-default configure option, which should toggle the default behavior according to https://docs.open-mpi.org/en/main/installing-open-mpi/configure-cli-options/installation.html, does not work - configure still reports

checking for default value of mca_base_component_show_load_errors... enabled for all

The behavior was changed in #10830 whenever and the correct configure option is --with-show-load-errors='^accelerator,rcache'.

[jsquyres]

@lahwaacz Many thanks for following up and updating the corresponding docs.