[INSTALL]: kokkos & kokkos-kernels

[benkozi]

Package name

kokkos kokkos-kernels

Package version/tag

latest

Build options

+openmp for both; GPU variants will be targeted in the future so we could try on Ursa (+cuda?)

Installation timeframe

Next release is okay. I'm happy to help with testing on Ursa (best if GPU variant attempted) or Gaea-C6.

Other information

Two UFS-WM atmospheric composition modeling components (CATChem & CECE) will use kokkos kokkos-kernels as part of their core execution engine. Other modeling libraries and utilities are also anticipating using the packages (e.g., WW4). We would like them installed as part of the unified environment as well.

Thank you!

cc @bbakernoaa @HendrikTolman-NOAA @JacobCarley-NOAA

[climbfuji]

These should be implemented as variants that are enabled on the UFS platforms only for now. I'll assign this to the EPIC team.

[rickgrubin-noaa]

@benkozi some questions:

kokkos

• set variant +openmp
• any other variants besides +cuda for now?

kokkos-kernels

• set variant +openmp
• any other variants? (see: kokkos_kernels/package.py)

It would be helpful to know how these packages are currently being configured / installed for use with CATChem & CECE.

[benkozi]

@rickgrubin-noaa - +cuda is a nice-to-have, as usage and testing of the GPU accelerator backend has been minimal. If kokkos+cuda isn't a blocker, I know we’d be interested in testing with it.

+openmp is currently the only requirement for CATChem & CECE that I’m aware of. However, looking at the spack variants, it could make sense to add +serial to both packages. It appears the serial backend may not be built by default.

I compiled and tested CATChem in a local containerized spack-stack environment (current develop branch) with kokkos+openmp kokkos-kernels+openmp. Pretty simple at this point. I expect there will be interest in optimization variants in the future, but nothing more to request at this time.

I cross-checked against the raw CMake commands used in the CECE dev container below. One thing to note: C++17 could be used with kokkos 4.x if C++20 is not an option.

RUN git clone -b 5.1.1 https://github.com/kokkos/kokkos.git /tmp/kokkos \
    && cd /tmp/kokkos \
    && cmake -B build \
      -DCMAKE_INSTALL_PREFIX=/usr/local \
      -DCMAKE_CXX_STANDARD=20 \
      -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
      -DKokkos_ENABLE_OPENMP=ON \
      -DKokkos_ENABLE_SERIAL=ON \
    && cmake --build build --parallel $(nproc) \
    && cmake --install build \
    && rm -rf /tmp/kokkos

RUN git clone --depth 1 https://github.com/kokkos/kokkos-kernels.git /tmp/kokkos-kernels \
    && cd /tmp/kokkos-kernels \
    && cmake -B build \
      -DCMAKE_INSTALL_PREFIX=/usr/local \
      -DCMAKE_CXX_STANDARD=20 \
      -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
      -DKokkosKernels_ENABLE_ALL_COMPONENTS=OFF \
      -DKokkosKernels_ENABLE_COMPONENT_SPARSE=ON \
      -DKokkosKernels_ENABLE_COMPONENT_BLAS=OFF \
      -DKokkosKernels_ENABLE_COMPONENT_GRAPH=OFF \
      -DKokkosKernels_ENABLE_COMPONENT_BATCHED=OFF \
      -DKokkosKernels_ENABLE_COMPONENT_LAPACK=OFF \
      -DKokkosKernels_ENABLE_COMPONENT_ODE=OFF \
      -DKokkosKernels_ADD_DEFAULT_ETI=OFF \
    && cmake --build build --parallel $(nproc) \
    && cmake --install build \
    && rm -rf /tmp/kokkos-kernels

[bbakernoaa]

I agree with @benkozi +cuda +serial and +openmp should cover our needs for quite a while.

[rickgrubin-noaa]

Thanks @benkozi @bbakernoaa -- first cut at building a full unified-dev environment will be without +cuda, and with +openmp and +serial for both packages.

Once it's clear that works with at least respect to building it, I'll point you at it for testing, and proceed to modify the package recipe to add +cuda.