Issue compiling Trilinos

[ishovkun]

I am unable to compile Trilinos.

My host-config: https://pastebin.com/mBuvcBVy
Trilinos build log (up to the first errors): https://pastebin.com/Apckm1ju
Log of the overall cmake run on TPL repo: https://pastebin.com/mF0ZnViy

Additional details:
Platform: Manjaro Linux
Compiler suite: gcc 10.2

Let me know should I post additional details.

[ishovkun]

Pinging @klevzoff and @corbett5.

[corbett5]

Weird I've never seen that error before. Can you paste the contents of packages/kokkos/cmake/KokkosCore_config.h.in, packages/kokkos/core/cmake/KokkosCore_config.h.in, and build-linux-gcc-release/trilinos/src/trilinos-build/KokkosCore_config.h? The first two files should be in build-linux-gcc-release/trilinos/src/trilinos-src/ or something like that.

[klevzoff]

Never seen this either, but other people have: spack/spack#16308
Apparently cmake's configure_file adds those backslashes for no reason sometimes?

[corbett5]

Wow, nice find! Since they're using gcc9 that might be the issue, although it seems that this should be completely orthogonal to the compiler.

[ishovkun]

The contents of packages/kokkos/core/cmake/KokkosCore_config.h.in:

/* The trivial 'src/build_common.sh' creates a config
 * that must stay in sync with this file.
 */
#cmakedefine KOKKOS_FOR_SIERRA

#if !defined(KOKKOS_FOR_SIERRA)

#if !defined(KOKKOS_MACROS_HPP) || defined(KOKKOS_CORE_CONFIG_H)
#error "Don't include KokkosCore_config.h directly; include Kokkos_Macros.hpp instead."
#else
#define KOKKOS_CORE_CONFIG_H
#endif

#cmakedefine KOKKOS_ENABLE_CUDA
#cmakedefine KOKKOS_ENABLE_OPENMP
#cmakedefine KOKKOS_ENABLE_PTHREAD
#cmakedefine KOKKOS_ENABLE_QTHREADS
#cmakedefine KOKKOS_ENABLE_SERIAL
#cmakedefine KOKKOS_ENABLE_Winthread

#cmakedefine KOKKOS_ENABLE_HWLOC
#cmakedefine KOKKOS_ENABLE_HBWSPACE
#cmakedefine KOKKOS_ENABLE_LIBRT

#cmakedefine KOKKOS_ENABLE_DEBUG
#cmakedefine KOKKOS_ENABLE_DEBUG_BOUNDS_CHECK
#cmakedefine KOKKOS_ENABLE_DEBUG_DUALVIEW_MODIFY_CHECK
#cmakedefine KOKKOS_ENABLE_PROFILING
#cmakedefine KOKKOS_ENABLE_PROFILING_LOAD_PRINT

#cmakedefine KOKKOS_ENABLE_AGGRESSIVE_VECTORIZATION

#ifdef KOKKOS_ENABLE_CUDA

#cmakedefine KOKKOS_ENABLE_CUDA_LDG_INTRINSIC

// mfh 16 Sep 2014: If passed in on the command line, that overrides
// any value of KOKKOS_USE_CUDA_UVM here.  Doing this should prevent build
// warnings like this one:
//
// packages/kokkos/core/src/KokkosCore_config.h:13:1: warning: "KOKKOS_USE_CUDA_UVM" redefined
//
// At some point, we should edit the test-build scripts in
// Trilinos/cmake/ctest/drivers/perseus/, and take
// -DKOKKOS_USE_CUDA_UVM from the command-line arguments there.  I
// hesitate to do that now, because I'm not sure if all the files are
// including KokkosCore_config.h (or a header file that includes it) like
// they should.
#ifndef KOKKOS_USE_CUDA_UVM
#cmakedefine KOKKOS_USE_CUDA_UVM
#endif

#cmakedefine KOKKOS_ENABLE_CUDA_RELOCATABLE_DEVICE_CODE

#cmakedefine KOKKOS_ENABLE_CUDA_LAMBDA

#endif

#cmakedefine KOKKOS_IMPL_CUDA_CLANG_WORKAROUND

#ifndef __CUDA_ARCH__
#cmakedefine KOKKOS_ENABLE_ISA_X86_64
#cmakedefine KOKKOS_ENABLE_ISA_KNC
#cmakedefine KOKKOS_ENABLE_ISA_POWERPCLE
#endif

#cmakedefine KOKKOS_ARCH_ARMV80 1
#cmakedefine KOKKOS_ARCH_ARMV81 1
#cmakedefine KOKKOS_ARCH_ARMV8_THUNDERX 1
#cmakedefine KOKKOS_ARCH_AVX 1
#cmakedefine KOKKOS_ARCH_AVX2 1
#cmakedefine KOKKOS_ARCH_AVX512MIC 1
#cmakedefine KOKKOS_ARCH_AVX512XEON 1
#cmakedefine KOKKOS_ARCH_KNC 1
#cmakedefine KOKKOS_ARCH_POWER8 1
#cmakedefine KOKKOS_ARCH_POWER9 1
#cmakedefine KOKKOS_ARCH_KEPLER 1
#cmakedefine KOKKOS_ARCH_KEPLER30 1
#cmakedefine KOKKOS_ARCH_KEPLER32 1
#cmakedefine KOKKOS_ARCH_KEPLER35 1
#cmakedefine KOKKOS_ARCH_KEPLER37 1
#cmakedefine KOKKOS_ARCH_MAXWELL 1
#cmakedefine KOKKOS_ARCH_MAXWELL50 1
#cmakedefine KOKKOS_ARCH_MAXWELL52 1
#cmakedefine KOKKOS_ARCH_MAXWELL53 1
#cmakedefine KOKKOS_ARCH_PASCAL 1
#cmakedefine KOKKOS_ARCH_PASCAL60 1
#cmakedefine KOKKOS_ARCH_PASCAL61 1
#cmakedefine KOKKOS_ARCH_VOLTA70 1

// TODO: These are currently not used in Kokkos.  Should they be removed?
#cmakedefine KOKKOS_ENABLE_MPI
#cmakedefine KOKKOS_ENABLE_CUSPARSE

// TODO: No longer options in Kokkos.  Need to be removed.
#cmakedefine KOKKOS_USING_DEPRECATED_VIEW
#cmakedefine KOKKOS_ENABLE_CXX11

#endif // !defined(KOKKOS_FOR_SIERRA)

[ishovkun]

I do not see KokkosCore_config.h.in in packages/kokkos/cmake, but I do see a KokkosConfig.cmake.in. Do you want to see that?

[ishovkun]

Here is the contents of build-linux-gcc-release/trilinos/src/trilinos-build/KokkosCore_config.h:

/* ---------------------------------------------
Makefile constructed configuration:
Mon 19 Oct 2020 11:38:02 AM PDT
----------------------------------------------*/
\#if !defined(KOKKOS_MACROS_HPP) || defined(KOKKOS_CORE_CONFIG_H)
\#error "Do not include KokkosCore_config.h directly; include Kokkos_Macros.hpp instead."
\#else
\#define KOKKOS_CORE_CONFIG_H
\#endif
/* Execution Spaces */
\#define KOKKOS_ENABLE_SERIAL
/* General Settings */
\#define KOKKOS_ENABLE_CXX11
\#define KOKKOS_ENABLE_PROFILING
\#define KOKKOS_ENABLE_DEPRECATED_CODE
/* Optimization Settings */
/* Cuda Settings */

[corbett5]

Actually they're using cmake@3.17 which is more likely to be the issue.

[ishovkun]

I use cmake-3.18.1

[corbett5]

Yeah sorry I was looking at a newer version of Trilinos so packages/kokkos/cmake/KokkosCore_config.h.in doesn't exist. Can you try and install cmake 3.14.5 and see if that fixes it?

[ishovkun]

I'll try.

[corbett5]

Once you've configured just do make trilinos again so you don't have to wait around for the other libraries to build.

[klevzoff]

There's some indication that it might have something to with version of make 4.3: Debian added a custom patch earlier this year to work around it (but it's for an older trilinos release and obviously not backported into mainstream) https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=961708

On my system make is 4.2.1 and I'm not seeing the problem. Just to confirm, what is make --version for you @ishovkun ?

[ishovkun]

my Make is 4.3

[klevzoff]

In fact looking at build-linux-gcc-release/trilinos/src/trilinos/packages/kokkos/Makefile.kokkos you can see where those backslashes come from. It might be the case that older make would interpret them as escape symbols and newer does not. I'm not sure why is it using that Makefile at all rather than fully relying on CMake-generated build system.

[ishovkun]

@corbett5 Using CMake-3.14.5 did not help: the build still fails with the same type of errors:

/home/ishovkun/dev/thirdPartyLibs/build-linux-gcc-release/trilinos/src/trilinos-build/KokkosCore_config.h:5:1: error: stray ‘\’ in program
    5 | \#if !defined(KOKKOS_MACROS_HPP) || defined(KOKKOS_CORE_CONFIG_H)                                                                                                                                          
      | ^ 

[ishovkun]

@klevzoff So do you suggest to use an older make?

[corbett5]

Yeah I'd give that a try.

It's funny because it seems they're aware of some issue with make 4.3
https://github.com/trilinos/Trilinos/blob/4285cf9c76dcad84dcea1afe565006a6ae136bbe/packages/kokkos/Makefile.kokkos#L446

[klevzoff]

So it looks like they also fixed it in upstream. If it did make it into 13.0 release, we're going to switch to that hopefully soon enough and the problem will be fixed. Otherwise, we might need to make our own patch.

In the meantime, trying a pre-4.3 make would be a workaround, if feasible without breaking your whole system.

[ishovkun]

@klevzoff Which one do you suggest to go with? https://ftp.gnu.org/gnu/make/

[klevzoff]

@klevzoff Which one do you suggest to go with? https://ftp.gnu.org/gnu/make/

4.2.1 definitely works and is the closest to the one you have, so I'd suggest that perhaps

[ishovkun]

So I compiled my own make (4.2.1) and used it during /path/to/make-4.2.1/make trilinos. I get literally the same errors.
Maybe I should cmake to this make instance prior to pyhon2 configure script invocation?

[corbett5]

I think you'd need to put it in your path so that which make returns your make 4.2.1 and not the system make 4.3. This might be tricky.

[ishovkun]

OK, so I did

export PATH=/path/to/make-4.2.1:$PATH

I confirm that Trilinos compiles. Thank you @corbett5 and @klevzoff so much for your help.
I will confirm in the chat if I am able to build the whole TPL repo.