Configuring with `Trilinos` somehow adds cuda flags on `expanse` #14711

marcfehling · 2023-01-23T06:18:22Z

I tried to compile the current master on expanse in a configuration I left otherwise unchanged since the last release, but now cmake complains about CUDA although I never instructed it to touch CUDA.

I could boil the problem down to the following configuration with Trilinos:

NOTE: I used the aocc compiler (based on clang) and I had to pass the MPI wrappers as compilers since cmake had trouble to find the mpi libraries otherwise. I used the same compiler to build Trilinos.

NOTE: When compiling deal.II with gcc using the same installation of Trilinos (built with aocc), the same CUDA error occurs.

cmake -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_Fortran_COMPILER=mpifort -DDEAL_II_WITH_MPI=ON -DDEAL_II_WITH_TRILINOS=ON -DTRILINOS_DIR={...}/trilinos-13.0.1 ../dealii/

Configuration succeeds until cmake compiles an example program to check for usable flags. This is the cmake output:

-- Performing Test DEAL_II_HAVE_USABLE_FLAGS_DEBUG
-- Performing Test DEAL_II_HAVE_USABLE_FLAGS_DEBUG - Failed
-- Unable to compile a simple test program. Trying to drop "-fuse-ld=lld" from the linker flags.
-- Performing Test DEAL_II_HAVE_USABLE_FLAGS_DEBUG
-- Performing Test DEAL_II_HAVE_USABLE_FLAGS_DEBUG - Failed
-- Unable to compile a simple test program. Trying to drop "-fuse-ld=gold" from the linker flags.
-- Performing Test DEAL_II_HAVE_USABLE_FLAGS_DEBUG
-- Performing Test DEAL_II_HAVE_USABLE_FLAGS_DEBUG - Failed
CMake Error at cmake/setup_finalize.cmake:137 (message):
  

    Configuration error: Cannot compile a test program with the final set of
    compiler and linker flags:
      CXX flags (DEBUG): -pedantic -fPIC -Wall -Wextra -Wmissing-braces -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wsuggest-override -Wswitch -Wsynth -Wwrite-strings -Wno-placement-new -Wno-deprecated-declarations -Wno-literal-suffix -Wno-psabi -Wno-class-memaccess -fopenmp-simd -Wno-parentheses -Wno-unused-local-typedefs -x cuda -O0 -ggdb -Wa,--compress-debug-sections
      LD flags  (DEBUG): -rdynamic -ggdb
      LIBRARIES (DEBUG): /usr/lib64/libz.so;/usr/lib64/libboost_iostreams.so;/usr/lib64/libboost_serialization.so;/usr/lib64/libboost_system.so;/usr/lib64/libboost_thread.so;/usr/lib64/libboost_regex.so;/usr/lib64/libboost_chrono.so;/usr/lib64/libboost_date_time.so;/usr/lib64/libboost_atomic.so;{...}/trilinos-13.0.1/lib/librol.so;{...}/trilinos-13.0.1/lib/libtempus.so;{...}/trilinos-13.0.1/lib/libmuelu-adapters.so;{...}/trilinos-13.0.1/lib/libmuelu-interface.so;{...}/trilinos-13.0.1/lib/libmuelu.so;{...}/trilinos-13.0.1/lib/liblocathyra.so;{...}/trilinos-13.0.1/lib/liblocaepetra.so;{...}/trilinos-13.0.1/lib/liblocalapack.so;{...}/trilinos-13.0.1/lib/libloca.so;{...}/trilinos-13.0.1/lib/libnoxepetra.so;{...}/trilinos-13.0.1/lib/libnoxlapack.so;{...}/trilinos-13.0.1/lib/libnox.so;{...}/trilinos-13.0.1/lib/libintrepid2.so;{...}/trilinos-13.0.1/lib/libintrepid.so;{...}/trilinos-13.0.1/lib/libteko.so;{...}/trilinos-13.0.1/lib/libstratimikos.so;{...}/trilinos-13.0.1/lib/libstratimikosbelos.so;{...}/trilinos-13.0.1/lib/libstratimikosamesos2.so;{...}/trilinos-13.0.1/lib/libstratimikosaztecoo.so;{...}/trilinos-13.0.1/lib/libstratimikosamesos.so;{...}/trilinos-13.0.1/lib/libstratimikosml.so;{...}/trilinos-13.0.1/lib/libstratimikosifpack.so;{...}/trilinos-13.0.1/lib/libifpack2-adapters.so;{...}/trilinos-13.0.1/lib/libifpack2.so;{...}/trilinos-13.0.1/lib/libanasazitpetra.so;{...}/trilinos-13.0.1/lib/libModeLaplace.so;{...}/trilinos-13.0.1/lib/libanasaziepetra.so;{...}/trilinos-13.0.1/lib/libanasazi.so;{...}/trilinos-13.0.1/lib/libamesos2.so;{...}/trilinos-13.0.1/lib/libtacho.so;{...}/trilinos-13.0.1/lib/libshylu_nodehts.so;{...}/trilinos-13.0.1/lib/libbelosxpetra.so;{...}/trilinos-13.0.1/lib/libbelostpetra.so;{...}/trilinos-13.0.1/lib/libbelosepetra.so;{...}/trilinos-13.0.1/lib/libbelos.so;{...}/trilinos-13.0.1/lib/libml.so;{...}/trilinos-13.0.1/lib/libifpack.so;{...}/trilinos-13.0.1/lib/libzoltan2.so;{...}/trilinos-13.0.1/lib/libpamgen_extras.so;{...}/trilinos-13.0.1/lib/libpamgen.so;{...}/trilinos-13.0.1/lib/libamesos.so;{...}/trilinos-13.0.1/lib/libgaleri-xpetra.so;{...}/trilinos-13.0.1/lib/libgaleri-epetra.so;{...}/trilinos-13.0.1/lib/libaztecoo.so;{...}/trilinos-13.0.1/lib/libisorropia.so;{...}/trilinos-13.0.1/lib/libxpetra-sup.so;{...}/trilinos-13.0.1/lib/libxpetra.so;{...}/trilinos-13.0.1/lib/libthyratpetra.so;{...}/trilinos-13.0.1/lib/libthyraepetraext.so;{...}/trilinos-13.0.1/lib/libthyraepetra.so;{...}/trilinos-13.0.1/lib/libthyracore.so;{...}/trilinos-13.0.1/lib/libtrilinosss.so;{...}/trilinos-13.0.1/lib/libtpetraext.so;{...}/trilinos-13.0.1/lib/libtpetrainout.so;{...}/trilinos-13.0.1/lib/libtpetra.so;{...}/trilinos-13.0.1/lib/libkokkostsqr.so;{...}/trilinos-13.0.1/lib/libtpetraclassiclinalg.so;{...}/trilinos-13.0.1/lib/libtpetraclassicnodeapi.so;{...}/trilinos-13.0.1/lib/libtpetraclassic.so;{...}/trilinos-13.0.1/lib/libepetraext.so;{...}/trilinos-13.0.1/lib/libtriutils.so;{...}/trilinos-13.0.1/lib/libshards.so;{...}/trilinos-13.0.1/lib/libzoltan.so;{...}/trilinos-13.0.1/lib/libepetra.so;{...}/trilinos-13.0.1/lib/libminitensor.so;{...}/trilinos-13.0.1/lib/libsacado.so;{...}/trilinos-13.0.1/lib/librtop.so;{...}/trilinos-13.0.1/lib/libkokkoskernels.so;{...}/trilinos-13.0.1/lib/libteuchoskokkoscomm.so;{...}/trilinos-13.0.1/lib/libteuchoskokkoscompat.so;{...}/trilinos-13.0.1/lib/libteuchosremainder.so;{...}/trilinos-13.0.1/lib/libteuchosnumerics.so;{...}/trilinos-13.0.1/lib/libteuchoscomm.so;{...}/trilinos-13.0.1/lib/libteuchosparameterlist.so;{...}/trilinos-13.0.1/lib/libteuchosparser.so;{...}/trilinos-13.0.1/lib/libteuchoscore.so;{...}/trilinos-13.0.1/lib/libkokkosalgorithms.so;{...}/trilinos-13.0.1/lib/libkokkoscontainers.so;{...}/trilinos-13.0.1/lib/libkokkoscore.so;{...}/trilinos-13.0.1/lib/libgtest.so;{...}/libflame-5.2.0/lib/libflame.so;/cm/shared/apps/spack/cpu/opt/spack/linux-centos8-zen2/clang-10.0.0/amdblis-2.2-kbqr6zs6th3hfxrpps5vahsuudmv2zot/lib/libblis.so;dl;

This is the error message in CMakeFiles/CMakeError.log:

/cm/shared/apps/spack/cpu/opt/spack/linux-centos8-zen/clang-10.0.0/openmpi-4.0.4-nz2g566opuddnwt5ymjorz2cdgb4spfm/bin/mpicxx   -DDEAL_II_HAVE_USABLE_FLAGS_DEBUG -pedantic -fPIC -Wall -Wextra -Wmissing-braces -Woverloaded-virtual -Wpointer-arith -Wsign-compare -Wswitch -Wsynth -Wwrite-strings -Wno-deprecated-declarations -Wfloat-conversion -Qunused-arguments -openmp-simd -Wno-parentheses -march=znver2 -Wno-unused-local-typedefs -x cuda -Og -ggdb -Wa,--compress-debug-sections -g -rdynamic -fuse-ld=lld -ggdb -O3 -DNDEBUG -o CMakeFiles/cmTC_5d71f.dir/src.cxx.o -c /expanse/lustre/projects/ucd150/mfehling/build-dealii/CMakeFiles/CMakeTmp/src.cxx
clang-10: error: cannot find libdevice for sm_20. Provide path to different CUDA installation via --cuda-path, or pass -nocudalib to build without linking with libdevice.
clang-10: error: cannot find CUDA installation.  Provide its path via --cuda-path, or pass -nocudainc to build without CUDA includes.
clang-10: error: cannot find CUDA installation.  Provide its path via --cuda-path, or pass -nocudainc to build without CUDA includes.

I noticed the -x cuda flag that somehow snug into the CXX flags list, although I never intended nor specified to configure deal.II with CUDA. ccmake states that both DEAL_II_WITH_CUDA and DEAL_II_MPI_WITH_CUDA_SUPPORT are set to OFF. Further, DEAL_II_ALLOW_PLATFORM_INTROSPECTION and DEAL_II_ALLOW_AUTODETECTION are ON by default.

Any ideas? Since I could compile deal.II with the same installation before, I marked this issue as a bug, but it can always be possible that I screwed something up. This is my Trilinos configuration:

cmake -DCMAKE_BUILD_TYPE=RELEASE          \
      -DCMAKE_C_COMPILER=mpicc            \
      -DCMAKE_CXX_COMPILER=mpicxx         \
      -DCMAKE_Fortran_COMPILER=mpifort    \
      -DTrilinos_ENABLE_Amesos=ON         \
      -DTrilinos_ENABLE_Epetra=ON         \
      -DTrilinos_ENABLE_EpetraExt=ON      \
      -DTrilinos_ENABLE_Ifpack=ON         \
      -DTrilinos_ENABLE_AztecOO=ON        \
      -DTrilinos_ENABLE_Sacado=ON         \
      -DTrilinos_ENABLE_Teuchos=ON        \
      -DTrilinos_ENABLE_MueLu=ON          \
      -DTrilinos_ENABLE_ML=ON             \
      -DTrilinos_ENABLE_ROL=ON            \
      -DTrilinos_ENABLE_Tpetra=ON         \
      -DTrilinos_ENABLE_COMPLEX_DOUBLE=ON \
      -DTrilinos_ENABLE_COMPLEX_FLOAT=ON  \
      -DTrilinos_ENABLE_Zoltan=ON         \
      -DTrilinos_VERBOSE_CONFIGURE=OFF    \
      -DTPL_ENABLE_MPI=ON                 \
      -DTPL_ENABLE_BLAS=ON                \
      -DTPL_ENABLE_LAPACK=ON              \
      -DBLAS_INCLUDE_DIRS=${AMDBLISHOME}/include \
      -DBLAS_LIBRARY_NAMES="blis"                \
      -DBLAS_LIBRARY_DIRS=${AMDBLISHOME}/lib     \
      -DLAPACK_INCLUDE_DIRS=${BIN_DIR}/libflame-5.2.0/include \
      -DLAPACK_LIBRARY_NAMES="flame"                          \
      -DLAPACK_LIBRARY_DIRS=${BIN_DIR}/libflame-5.2.0/lib     \
      -DBUILD_SHARED_LIBS=ON              \
      -DCMAKE_VERBOSE_MAKEFILE=OFF        \
      -DCMAKE_INSTALL_PREFIX:PATH=${BIN_DIR}/trilinos-13.0.1 \
      ..

The text was updated successfully, but these errors were encountered:

tamiko · 2023-01-23T20:42:16Z

@marcfehling We do not set any of these flags by hand. In order to figure out where it is coming from, would you mind commenting out the check in setup_finalize.cmake and running cmake again? This way we can have a look at detailed.log to see what is going on.

masterleinad · 2023-01-23T20:58:28Z

This very clearly looks like Trilinos (and thus Kokkos bundled with Trilinos was configured with CUDA support).

masterleinad · 2023-01-23T21:01:37Z

What does the Kokkos_ENABLE_CUDA CMake variable in Trilinos say?

marcfehling · 2023-01-24T01:57:51Z

What does the Kokkos_ENABLE_CUDA CMake variable in Trilinos say?

I no longer have the build directory to check it, but I found this particular line in my Trilinos installation which is likely the cause:

$ grep -r "x.cuda"
trilinos-13.0.1/lib/cmake/Kokkos/KokkosConfig.cmake:    INTERFACE_COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:CXX>:-x;cuda>"

I will install Trilinos again setting Kokkos_ENABLE_CUDA=OFF explicitly and will report back to you.

marcfehling · 2023-01-24T02:19:43Z

Interesting. I have configured trilinos-13.0.1 in the same configuration as stated in my original message and grepping in the build directory gives me these contradicting statements:

$ grep -r "Kokkos_ENABLE_CUDA"
CMakeCache.txt:Kokkos_ENABLE_CUDA:BOOL=OFF
CMakeCache.txt:Kokkos_ENABLE_CUDA_CONSTEXPR:BOOL=OFF
CMakeCache.txt:Kokkos_ENABLE_CUDA_LAMBDA:BOOL=OFF
CMakeCache.txt:Kokkos_ENABLE_CUDA_LDG_INTRINSIC:BOOL=OFF
CMakeCache.txt:Kokkos_ENABLE_CUDA_RELOCATABLE_DEVICE_CODE:BOOL=OFF
CMakeCache.txt:Kokkos_ENABLE_CUDA_UVM:BOOL=OFF

$ grep -r "x.cuda"       
CMakeCache.txt:KOKKOS_CUDA_OPTIONS:INTERNAL=-x;cuda
packages/kokkos/CMakeFiles/KokkosConfig_install.cmake:    INTERFACE_COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:CXX>:-x;cuda>"
packages/kokkos/KokkosTrilinosConfig.cmake:    INTERFACE_COMPILE_OPTIONS "$<$<COMPILE_LANGUAGE:CXX>:-x;cuda>"

Maybe it's bug in Trilinos that has been fixed in more recent releases? I will check.

marcfehling · 2023-01-24T03:34:12Z

I tried to configure trilinos-13.4.1 and found the same entries in CMakeCache.txt. Compiling and installing Trilinos succeeds in this configuration.

marcfehling · 2023-01-24T04:23:39Z

@marcfehling We do not set any of these flags by hand. In order to figure out where it is coming from, would you mind commenting out the check in setup_finalize.cmake and running cmake again? This way we can have a look at detailed.log to see what is going on.

Attached you'll find the detailed.log from my most recent minimal configuration, which contains the entry TRILINOS_CXX_FLAGS = -x cuda. See also

dealii/cmake/modules/FindDEAL_II_TRILINOS.cmake

Lines 177 to 178 in 069d370

    
           CXX_FLAGS 
        
             OPTIONAL KOKKOS_COMPILE_FLAGS

This detailed-old.log is from the installation I used before the last release.

marcfehling · 2023-01-25T22:46:49Z

In deal.II we pick up these flags from get_property()

dealii/cmake/modules/FindDEAL_II_TRILINOS.cmake

Line 132 in 069d370

    
           get_property(KOKKOS_COMPILE_FLAGS_FULL TARGET Kokkos::kokkos PROPERTY INTERFACE_COMPILE_OPTIONS)

INTERFACE_COMPILE_OPTIONS is set to KOKKOS_ALL_COMPILE_OPTIONS here
https://github.com/kokkos/kokkos/blob/5ad609661e570ba6aa7716a26a91cb67d559f8a2/cmake/KokkosTrilinosConfig.cmake.in#L14

-x cuda will be appended to KOKKOS_CUDA_OPTIONS here
https://github.com/kokkos/kokkos/blob/5ad609661e570ba6aa7716a26a91cb67d559f8a2/cmake/kokkos_arch.cmake#L177

Adding KOKKOS_CUDA_OPTIONS to KOKKOS_ALL_COMPILE_OPTIONS is guarded by KOKKOS_ENABLE_CUDA here
https://github.com/kokkos/kokkos/blob/5ad609661e570ba6aa7716a26a91cb67d559f8a2/CMakeLists.txt#L245-L247

So why do we, or rather Trilinos, pick up -x cuda, when it's guarded by KOKKOS_ENABLE_CUDA?

masterleinad · 2023-01-25T23:07:37Z

I just tried with

cmake -DCMAKE_BUILD_TYPE=RELEASE -DTrilinos_ENABLE_Amesos=ON -DTrilinos_ENABLE_Epetra=ON -DTrilinos_ENABLE_EpetraExt=ON -DTrilinos_ENABLE_Ifpack=ON -DTrilinos_ENABLE_AztecOO=ON -DTrilinos_ENABLE_Sacado=ON -DTrilinos_ENABLE_Teuchos=ON -DTrilinos_ENABLE_MueLu=ON -DTrilinos_ENABLE_ML=ON -DTrilinos_ENABLE_ROL=ON -DTrilinos_ENABLE_Tpetra=ON -DTrilinos_ENABLE_COMPLEX_DOUBLE=ON -DTrilinos_ENABLE_COMPLEX_FLOAT=ON -DTrilinos_ENABLE_Zoltan=ON -DTrilinos_VERBOSE_CONFIGURE=OFF -DTPL_ENABLE_MPI=ON -DTPL_ENABLE_BLAS=ON -DTPL_ENABLE_LAPACK=ON -DBUILD_SHARED_LIBS=ON -DCMAKE_VERBOSE_MAKEFILE=OFF ..

and

$ grep -r "x.cuda"

doesn't print anything for me.

marcfehling · 2023-01-25T23:13:34Z

This seems to only occur when you are using clang.
https://github.com/kokkos/kokkos/blob/5ad609661e570ba6aa7716a26a91cb67d559f8a2/cmake/kokkos_arch.cmake#L175-L177

Did you configure with clang?

masterleinad · 2023-01-25T23:31:59Z

OK. With clang++, I'm seeing

$ grep -r "x.cuda"
./CMakeCache.txt:KOKKOS_CUDA_OPTIONS:INTERNAL=-x;cuda

which looks OK.

marcfehling · 2023-01-25T23:47:34Z

Interesting. So there is no -x cuda in INTERFACE_COMPILE_OPTIONS in your Trilinos configuration.

Let me double check my configuration for trilinos-13.4.1. Maybe this problem only occurred in earlier versions (like my trilinos-13.0.1).

marcfehling · 2023-01-25T23:55:27Z

So why do we, or rather Trilinos, pick up -x cuda, when it's guarded by KOKKOS_ENABLE_CUDA?

I checked Kokkos commit history. This used to be a problem and has been fixed with Kokkos 3.3 in kokkos/kokkos#3387.

Trilinos 13.0 uses Kokkos 3.1.1. Trilinos 13.2 uses Kokkos 3.4.01. Trilinos 13.4 uses Kokkos 3.6.0.

Should we warn users about this? Maybe add an error for KOKKOS_ENABLE_CUDA=OFF, CMAKE_CXX_COMPILER=clang++, KOKKOS_VERSION < 3.3 (or TRILINOS_VERSION < 13.2)? Or apply the patch in that configuration?

masterleinad · 2023-01-26T01:07:09Z

Ah, I checked with a develop version from last December.

masterleinad · 2023-01-26T01:15:27Z

For standalone Kokkos we require 3.7.00 anyway so that won't be a problem (and IIRC it was only an issue for Trilinos anyway). We could of course filter the flags in that case not to include -x cuda.

marcfehling · 2023-01-26T19:48:53Z

I tried to compile deal.II with the legacy Trilinos 13.0.1 after removing -x cuda from INTERFACE_COMPILE_OPTIONS.

Configuration succeeds, but compiling deal.II now complains about a missing Kokkos_Macros.hpp header. The header has been introduced in deal.II with #14453. But the file exists in trilinos-13.0.1/include/Kokkos_Macros.hpp, and the directory is listed in TRILINOS_INCLUDE_DIRS in detailed.log.

In file included from {...}/dealii/source/arborx/access_traits.cc:16:
In file included from {...}/dealii/include/deal.II/arborx/access_traits.h:19:
In file included from {...}/build-dealii/include/deal.II/base/config.h:562:
{...}/dealii/include/deal.II/base/numbers.h:28:10: fatal error: 'Kokkos_Macros.hpp' file not found
#include <Kokkos_Macros.hpp>
         ^~~~~~~~~~~~~~~~~~~
1 error generated.
make[2]: *** [source/arborx/CMakeFiles/obj_arborx_debug.dir/build.make:82: source/arborx/CMakeFiles/obj_arborx_debug.dir/access_traits.cc.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:5475: source/arborx/CMakeFiles/obj_arborx_debug.dir/all] Error 2
make: *** [Makefile:149: all] Error

UPDATE: Problem also exists with Trilinos 13.4.1. Is it again a clang problem?

masterleinad · 2023-01-26T21:40:22Z

UPDATE: Problem also exists with Trilinos 13.4.1. Is it again a clang problem?

I just tried myself with 13.4.1 and clang++ and get /tmp/trilinos-install/include included for deal.II (and that has Kokkos_Macros.hpp. What does the

-- Defining target: interface_trilinos
--     VERSION:             13.4.1
...

when running CMake print for you?

marcfehling · 2023-01-26T22:21:57Z

Kokkos_Macros.hpp is in the include folder of both 13.0.1 and 13.4.1 installations of Trilinos. Somehow I still get this error.

What does the [...] when running CMake print for you?

Where do I find it? I do not see these lines in the console output of neither dealii nor Trilinos, even after starting in a fresh build directory. expanse provides cmake version 3.18.2.

masterleinad · 2023-01-26T22:27:05Z

@tamiko added interface targets for all dependencies in a recent pull request and you should see such an output when configuring deal.II.

marcfehling · 2023-01-26T22:56:39Z

I worked on the master branch from Jan 22 which didn't contain this feature yet. I had to check out the branch of #14738 to run the dealii configuration and got:

-- Defining target: interface_trilinos
--     VERSION:             13.4.1
--     LINK_LIBRARIES:      {...}/trilinos-13.4.1/lib/librol.so;{...}/trilinos-13.4.1/lib/libtempus.so;{...}/trilinos-13.4.1/lib/libmuelu-adapters.so;{...}/trilinos-13.4.1/lib/libmuelu-interface.so;{...}/trilinos-13.4.1/lib/libmuelu.so;{...}/trilinos-13.4.1/lib/liblocatpetra.so;{...}/trilinos-13.4.1/lib/liblocathyra.so;{...}/trilinos-13.4.1/lib/liblocaepetra.so;{...}/trilinos-13.4.1/lib/liblocalapack.so;{...}/trilinos-13.4.1/lib/libloca.so;{...}/trilinos-13.4.1/lib/libnoxepetra.so;{...}/trilinos-13.4.1/lib/libnoxlapack.so;{...}/trilinos-13.4.1/lib/libnox.so;{...}/trilinos-13.4.1/lib/libintrepid2.so;{...}/trilinos-13.4.1/lib/libintrepid.so;{...}/trilinos-13.4.1/lib/libteko.so;{...}/trilinos-13.4.1/lib/libstratimikos.so;{...}/trilinos-13.4.1/lib/libstratimikosbelos.so;{...}/trilinos-13.4.1/lib/libstratimikosamesos2.so;{...}/trilinos-13.4.1/lib/libstratimikosaztecoo.so;{...}/trilinos-13.4.1/lib/libstratimikosamesos.so;{...}/trilinos-13.4.1/lib/libstratimikosml.so;{...}/trilinos-13.4.1/lib/libstratimikosifpack.so;{...}/trilinos-13.4.1/lib/libifpack2-adapters.so;{...}/trilinos-13.4.1/lib/libifpack2.so;{...}/trilinos-13.4.1/lib/libanasazitpetra.so;{...}/trilinos-13.4.1/lib/libModeLaplace.so;{...}/trilinos-13.4.1/lib/libanasaziepetra.so;{...}/trilinos-13.4.1/lib/libanasazi.so;{...}/trilinos-13.4.1/lib/libamesos2.so;{...}/trilinos-13.4.1/lib/libtacho.so;{...}/trilinos-13.4.1/lib/libshylu_nodehts.so;{...}/trilinos-13.4.1/lib/libbelosxpetra.so;{...}/trilinos-13.4.1/lib/libbelostpetra.so;{...}/trilinos-13.4.1/lib/libbelosepetra.so;{...}/trilinos-13.4.1/lib/libbelos.so;{...}/trilinos-13.4.1/lib/libml.so;{...}/trilinos-13.4.1/lib/libifpack.so;{...}/trilinos-13.4.1/lib/libzoltan2.so;{...}/trilinos-13.4.1/lib/libpamgen_extras.so;{...}/trilinos-13.4.1/lib/libpamgen.so;{...}/trilinos-13.4.1/lib/libamesos.so;{...}/trilinos-13.4.1/lib/libgaleri-xpetra.so;{...}/trilinos-13.4.1/lib/libgaleri-epetra.so;{...}/trilinos-13.4.1/lib/libaztecoo.so;{...}/trilinos-13.4.1/lib/libisorropia.so;{...}/trilinos-13.4.1/lib/libxpetra-sup.so;{...}/trilinos-13.4.1/lib/libxpetra.so;{...}/trilinos-13.4.1/lib/libthyratpetra.so;{...}/trilinos-13.4.1/lib/libthyraepetraext.so;{...}/trilinos-13.4.1/lib/libthyraepetra.so;{...}/trilinos-13.4.1/lib/libthyracore.so;{...}/trilinos-13.4.1/lib/libtrilinosss.so;{...}/trilinos-13.4.1/lib/libtpetraext.so;{...}/trilinos-13.4.1/lib/libtpetrainout.so;{...}/trilinos-13.4.1/lib/libtpetra.so;{...}/trilinos-13.4.1/lib/libkokkostsqr.so;{...}/trilinos-13.4.1/lib/libtpetraclassiclinalg.so;{...}/trilinos-13.4.1/lib/libtpetraclassicnodeapi.so;{...}/trilinos-13.4.1/lib/libtpetraclassic.so;{...}/trilinos-13.4.1/lib/libepetraext.so;{...}/trilinos-13.4.1/lib/libtriutils.so;{...}/trilinos-13.4.1/lib/libshards.so;{...}/trilinos-13.4.1/lib/libzoltan.so;{...}/trilinos-13.4.1/lib/libepetra.so;{...}/trilinos-13.4.1/lib/libminitensor.so;{...}/trilinos-13.4.1/lib/libsacado.so;{...}/trilinos-13.4.1/lib/librtop.so;{...}/trilinos-13.4.1/lib/libkokkoskernels.so;{...}/trilinos-13.4.1/lib/libteuchoskokkoscomm.so;{...}/trilinos-13.4.1/lib/libteuchoskokkoscompat.so;{...}/trilinos-13.4.1/lib/libteuchosremainder.so;{...}/trilinos-13.4.1/lib/libteuchosnumerics.so;{...}/trilinos-13.4.1/lib/libteuchoscomm.so;{...}/trilinos-13.4.1/lib/libteuchosparameterlist.so;{...}/trilinos-13.4.1/lib/libteuchosparser.so;{...}/trilinos-13.4.1/lib/libteuchoscore.so;{...}/trilinos-13.4.1/lib/libkokkosalgorithms.so;{...}/trilinos-13.4.1/lib/libkokkoscontainers.so;{...}/trilinos-13.4.1/lib/libkokkoscore.so;{...}/trilinos-13.4.1/lib/libgtest.so;{...}/libflame-5.2.0/lib/libflame.so;/cm/shared/apps/spack/cpu/opt/spack/linux-centos8-zen2/clang-10.0.0/amdblis-2.2-kbqr6zs6th3hfxrpps5vahsuudmv2zot/lib/libblis.so;/usr/lib64/libdl.so
--     INCLUDE_DIRECTORIES: {...}/trilinos-13.4.1/include;/usr/include;{...}/libflame-5.2.0/include;/cm/shared/apps/spack/cpu/opt/spack/linux-centos8-zen2/clang-10.0.0/amdblis-2.2-kbqr6zs6th3hfxrpps5vahsuudmv2zot/include
-- DEAL_II_WITH_TRILINOS successfully set up with external dependencies.

Luckily, the 'Kokkos_Macros.hpp' file not found error does no longer occur, on both trilinos 13.0.1 and 13.4.1. Some patch on dealii since Sunday must've fixed this. Thank you!

marcfehling added Bug Build system labels Jan 23, 2023

marcfehling added the but definitely not our fault 😡 label Jan 26, 2023

masterleinad self-assigned this Jan 26, 2023

masterleinad mentioned this issue Jan 26, 2023

Strip "-x cuda" from Trilinos flags without configured Cuda support #14739

Merged

marcfehling closed this as completed in #14739 Jan 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Configuring with `Trilinos` somehow adds cuda flags on `expanse` #14711

Configuring with `Trilinos` somehow adds cuda flags on `expanse` #14711

marcfehling commented Jan 23, 2023 •

edited

tamiko commented Jan 23, 2023

masterleinad commented Jan 23, 2023

masterleinad commented Jan 23, 2023

marcfehling commented Jan 24, 2023 •

edited

marcfehling commented Jan 24, 2023

marcfehling commented Jan 24, 2023 •

edited

marcfehling commented Jan 24, 2023 •

edited

marcfehling commented Jan 25, 2023 •

edited

masterleinad commented Jan 25, 2023

marcfehling commented Jan 25, 2023

masterleinad commented Jan 25, 2023

marcfehling commented Jan 25, 2023

marcfehling commented Jan 25, 2023 •

edited

masterleinad commented Jan 26, 2023

masterleinad commented Jan 26, 2023

marcfehling commented Jan 26, 2023 •

edited

masterleinad commented Jan 26, 2023

marcfehling commented Jan 26, 2023

masterleinad commented Jan 26, 2023

marcfehling commented Jan 26, 2023

Configuring with Trilinos somehow adds cuda flags on expanse #14711

Configuring with Trilinos somehow adds cuda flags on expanse #14711

Comments

marcfehling commented Jan 23, 2023 • edited

tamiko commented Jan 23, 2023

masterleinad commented Jan 23, 2023

masterleinad commented Jan 23, 2023

marcfehling commented Jan 24, 2023 • edited

marcfehling commented Jan 24, 2023

marcfehling commented Jan 24, 2023 • edited

marcfehling commented Jan 24, 2023 • edited

marcfehling commented Jan 25, 2023 • edited

masterleinad commented Jan 25, 2023

marcfehling commented Jan 25, 2023

masterleinad commented Jan 25, 2023

marcfehling commented Jan 25, 2023

marcfehling commented Jan 25, 2023 • edited

masterleinad commented Jan 26, 2023

masterleinad commented Jan 26, 2023

marcfehling commented Jan 26, 2023 • edited

masterleinad commented Jan 26, 2023

marcfehling commented Jan 26, 2023

masterleinad commented Jan 26, 2023

marcfehling commented Jan 26, 2023

Configuring with `Trilinos` somehow adds cuda flags on `expanse` #14711

Configuring with `Trilinos` somehow adds cuda flags on `expanse` #14711

marcfehling commented Jan 23, 2023 •

edited

marcfehling commented Jan 24, 2023 •

edited

marcfehling commented Jan 24, 2023 •

edited

marcfehling commented Jan 24, 2023 •

edited

marcfehling commented Jan 25, 2023 •

edited

marcfehling commented Jan 25, 2023 •

edited

marcfehling commented Jan 26, 2023 •

edited