Build and install downstream Trilinos packages against pre-installed Kokkos using native CMake build system

[bartlettroscoe]

Description

This issue will track Trilinos efforts to pre-build/install Kokkos (using TriBITS) and the build downstream packages against pre-built/installed Kokkos by setting (TPL_ENABLE_Kokkos=ON). This uses the new support in TriBITS for this in:

• Complete initial implementation for treating internal packages as external found with find_package() (#63) TriBITSPub/TriBITS#560

which is part of:

• EPIC: Merge TriBITS concepts and implementation of Packages and TPLs TriBITSPub/TriBITS#63

Initially we will use the TriBITS build system for Kokkos because it provides the subpackages expected by downstream Trilinos packages and provides the rest that is needed by TriBITS (since it will automatically be a TriBITS-compliant external package). Also, initially we can comment out the special logic about pulling in compiler options from Kokkos and will not try a CUDA build to make this easier.

Then we can get this working for the native Kokkos CMake build system.

Tasks

• Pre-build/install Kokkos, then KokkosKernels, then the test of Trilinos using TriBITS build system (see PR Tweaks to Trilinos to get working with pre-installed Kokkos and KokkosKernels (with TriBITS updates) (#11545) #11546) [@bartlettroscoe ]
• Remove TriBITS subpackages from TriBITS build of Kokkos [@bartlettroscoe]
  • Merge PR Remove usage of subpackages from TriBITS Kokkos build and downstream Trilinos packages (#11545) #11808 to 'develop'
  • Get feedback from Trilinos customers testing against 'develop' to see if there are major problems with removing Kokkos subpackages ... Addressed some of the issues in PR #11863
• Getting Trilinos to build and pass all tests and against pre-installed Kokkos using Kokkos native non-TriBITS CMake build system (see Trilinos PR Get pre-installed Kokkos and Trilinos working with native Kokkos non-TriBITS CMake build system (#11545)  #11863) [@bartlettroscoe]
• Add PR to remove duplication created by refactoring in PR #11808 ... See PR Base CMakeLists.txt cleanup, reduced tarball testing, CI enable tweak, TriBITS update (#11545, #11976) #12044

[ibaned]

Thanks for working on this! I will greatly help our Sandia production code when this is done. I'd be happy to help with making the native Kokkos build system export everything TriBITS expects.

[bartlettroscoe]

FYI: I am working to remove subpackages from Kokkos and usage in all downstream TriBITS packages. I am developing this refactoring as a script that you run on downstream TriBITS packages and it will make all of the needed changes automatically (so TriBITS packages outside of Trilinos can run this single script and absorb the changes).

NOTE: Removing Kokkos subpackages definingly breaks backward compatibility for both downstream TriBITS and non-TriBITS CMake packages and for users that are configuring Trilinos (as it changes the names of some of the enable vars). (But it should be easy to absorb the changes in both cases given the scripts I am producing.) But this refactoring is not like falling off a log.

Update: 4/18/2023 While this change can break backward compatibility for some customers, it may not for many/most customers. If they are just depending on Kokkos then they may not need to change anything.

[bartlettroscoe]

With the merge of PR:

• Base CMakeLists.txt cleanup, reduced tarball testing, CI enable tweak, TriBITS update (#11545, #11976) #12044

This is story is not fully complete

[bartlettroscoe]

FYI: Here is the Spack PR that has the Trilinos Spack package using the Kokkos Spack package:

• trilinos@14.4.0 +kokkos: use external kokkos spack/spack#39397

[bartlettroscoe]

FYI: Note that there have been some issues in the Spack packages downstream from Trilinos for this change in:

• trilinos@14.4.0 +kokkos: use external kokkos spack/spack#39397

You can see these in:

• datatransferkit build error. xsdk-project/xsdk-issues#214
• phist build issue xsdk-project/xsdk-issues#216

These are different issues but there was some hard-coded logic that expected Kokkos to be installed along with Trilinos. There was no reason to do that (and that has been the case for at least 10 years). The possible solutions for the problems are described in:

• datatransferkit build error. xsdk-project/xsdk-issues#214 (comment)

In summary, my recommendation was/is:

Option-3: Have DTK and other downstream Spack packages stop calling find_package(Trilinos COMPONENTS Kokkos Tpetra <pkgi> ...) and instead call find_packag(Kokkos), find_package(Tpetra), find_package(<pkgi>). (This would require changing any downstream CMake project that calls find_package(Trilinos COMPONENTS ... Kokkos ...).)

My advice would be to choose option-3. That is backward compatible to older versions of Trilinos and it starts to move the package ecosystem away for a Trilinos-centric world. (It should just be that different Spack packages provide different subsets of Trilinos packages as installed CMake packages with installed <Package>Config.cmake files. Trilinos should be a coordination project and provide for very efficient co-development workflows but one should stop assuming that "Trilinos" will be one big installed CMake meta-package called "Trilinos".)

and @jwillenbring agreed in xsdk-project/xsdk-issues#214 (comment).

In my opinion, that is the direction that Trilinos and the package ecosystem should be going.

[sebrowne]

I agree with this approach. I had no idea that worked (find_package(Tpetra), for instance), but that's excellent!

[bartlettroscoe]

I agree with this approach. I had no idea that worked (find_package(Tpetra), for instance), but that's excellent!

@sebrowne, this has been supported for a least 10 years (support for which was not written by me). I have added more explicit documentation for this as part of this TriBITS/Trilinos CMake modernization work. For example, see:

• https://docs.trilinos.org/files/TrilinosBuildReference.html#using-the-installed-software-in-downstream-cmake-projects

and see a (tested) example specific to Trilinos at:

• https://github.com/trilinos/Trilinos/tree/develop/demos/simpleBuildAgainstTrilinos#3-configure-the-example-project
• https://github.com/trilinos/Trilinos/blob/develop/demos/simpleBuildAgainstTrilinos/CMakeLists.by_package.cmake