nvcc link failure with Kokkos_ENABLE_LIBDL and glibc >= 2.34

[jedbrown]

Glibc 2.34 integrated libdl into libc. The residual /usr/bin/libdl.a is a text stub

$ cat /usr/lib/libpthread.a
!<arch>

gcc knows how to handle this just fine, but nvcc does not.

$ echo 'int main() {return 0;}' > a.c
$ gcc a.c /usr/lib/libdl.a
$ nvcc a.c /usr/lib/libdl.a
nvlink fatal   : Could not open input file '/usr/lib/libdl.a'
$ nvcc a.c -ldl
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Fri_Dec_17_18:16:03_PST_2021
Cuda compilation tools, release 11.6, V11.6.55
Build cuda_11.6.r11.6/compiler.30794723_0

This results in link failures unless configuring -DKokkos_ENABLE_LIBDL=OFF. Maybe this can be turned off by default or use -ldl instead of CMake's explicit /usr/lib/libdl.a.

[crtrott]

Does this problem go away with the latest cmake? This looks like something cmake should fix, but we can try and add a workaround.

[crtrott]

Note turning LIBDL off the way you did will disable the ability to load profiling tools during runtime.

[jedbrown]

This has been with cmake-3.22.2

[jedbrown]

We disabled LIBDL in PETSc as a workaround. I don't know if it's relevant to our users. https://gitlab.com/petsc/petsc/-/merge_requests/4890

[crtrott]

Generally the profiling tools are pretty relevant for downstream users, because it allows proper kernel names like KokkosKernels::gemv etc. to show up inside of the major third party profiling tools like Tau, VTune and Nsight Compute. But its probably an acceptable short term solution for you. We are looking into how to put a workaround in on our side.

[jedbrown]

Perhaps just check if the symbols are available despite not explicitly linking libdl?

[crtrott]

What about this?

diff --git a/cmake/kokkos_tpls.cmake b/cmake/kokkos_tpls.cmake
index 54c6b520b..4dfdb6457 100644
--- a/cmake/kokkos_tpls.cmake
+++ b/cmake/kokkos_tpls.cmake
@@ -77,7 +77,7 @@ ENDIF()
 KOKKOS_IMPORT_TPL(HWLOC)
 KOKKOS_IMPORT_TPL(LIBNUMA)
 KOKKOS_IMPORT_TPL(LIBRT)
-KOKKOS_IMPORT_TPL(LIBDL)
+#KOKKOS_IMPORT_TPL(LIBDL)
 KOKKOS_IMPORT_TPL(MEMKIND)
 IF (NOT WIN32)
   KOKKOS_IMPORT_TPL(THREADS INTERFACE)
diff --git a/core/src/CMakeLists.txt b/core/src/CMakeLists.txt
index 88cca93f3..9c5db5994 100644
--- a/core/src/CMakeLists.txt
+++ b/core/src/CMakeLists.txt
@@ -95,7 +95,7 @@ IF (NOT KOKKOS_ENABLE_COMPILE_AS_CMAKE_LANGUAGE)
   KOKKOS_LINK_TPL(kokkoscore PUBLIC CUDA)
 ENDIF()
 KOKKOS_LINK_TPL(kokkoscore PUBLIC HPX)
-KOKKOS_LINK_TPL(kokkoscore PUBLIC LIBDL)
+#KOKKOS_LINK_TPL(kokkoscore PUBLIC LIBDL)
 KOKKOS_LINK_TPL(kokkoscore PUBLIC LIBRT)
 # On *nix-like systems (Linux, macOS) we need pthread for C++ std::thread
 IF (NOT WIN32)
@@ -111,6 +111,8 @@ IF (KOKKOS_ENABLE_IMPL_DESUL_ATOMICS AND
     (KOKKOS_ENABLE_OPENMPTARGET OR (CMAKE_CXX_COMPILER_ID STREQUAL XLClang)))
   target_link_libraries(kokkoscore PUBLIC atomic)
 ENDIF()
-
+IF (KOKKOS_ENABLE_LIBDL)
+  target_link_libraries(kokkoscore PUBLIC ${CMAKE_DL_LIBS})
+ENDIF()

 KOKKOS_LINK_TPL(kokkoscore PUBLIC LIBQUADMATH)

[dalg24]

The potential issue with your suggestion is we lose the find_path call to locate the dlfcn.h header and it is not added to the list of include directories.

[ajpowelsnl]

Discuss and assign on Wednesday

[ajpowelsnl]

Thx @masterleinad