Fix use of OpenMP with Cuda or HIP as compile language

[dalg24]

We recently discovered that configuring with

-DKokkos_ENABLE_{CUDA,HIP}=ON \
-DKokkos_ENABLE_COMPILE_AS_CMAKE_LANGUAGE=ON \
-DKokkos_ENABLE_OPENMP=ON`

was broken and that the compiler flags to enable OpenMP support were not properly set, triggering an internal sanity check at compile time

In file included from <kokkos>/core/src/impl/Kokkos_Core.cpp:21:
In file included from <kokkos>/core/src/Kokkos_Core.hpp:45:
In file included from <kokkos>/build/compile_language/KokkosCore_Config_DeclareBackend.hpp:22:
In file included from <kokkos>/core/src/decl/Kokkos_Declare_OPENMP.hpp:21:
In file included from <kokkos>/core/src/OpenMP/Kokkos_OpenMP.hpp:203:
<kokkos>/core/src/OpenMP/Kokkos_OpenMP_Instance.hpp:23:2: error: "You enabled Kokkos OpenMP support without enabling OpenMP in the compiler!"
#error \

This PR adds these flag for compiling the CUDA/HIP source files. In contrast to the solution suggested by Daniel in #6965 (review), I am adding these flags in the cmake file that handles TPLs. The reason for doing it there instead of in the kokkos_arch.cmake file is that TPLs are only processed after which means that OpenMP_CXX_FLAGS is not defined the first time Kokkos is configured. I expect it only worked for Daniel because it was a reconfigure (that is 2nd time parsing the file with OpenMP now having been found).

I have tested (might be a strong word, I only checked the flag is properly added and the library builds) on Frontier with ROCm 5.7.1 but I did not check that it works for CUDA. @masterleinad do you mind testing it?

[masterleinad]

This works for me on polaris with the Cuda backend. It seems we don't need to add any linker flags. I'm fine with looking into that if it ever comes up.

[Rombur]

I tried this PR on Frontier and it works fine but the library does not contain the path to libomp. I needed to set LD_LIBRARY_PATH. I don't know if this is expected.

[ndellingwood]

Does this merit a changelog entry for 4.4 #6914 , or was this a fixup only applicable to recent changes on develop?

[masterleinad]

Does this merit a changelog entry for 4.4 #6914 , or was this a fixup only applicable to recent changes on develop?

done