Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initialize Kokkos with MultithreadInfo::n_threads() #14869

Merged
merged 4 commits into from
Mar 12, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
10 changes: 9 additions & 1 deletion source/base/kokkos.cc
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
// ---------------------------------------------------------------------

#include <deal.II/base/kokkos.h>
#include <deal.II/base/multithread_info.h>

#include <deal.II/lac/la_parallel_vector.h>
#include <deal.II/lac/vector_memory.h>
Expand All @@ -34,7 +35,14 @@ namespace internal
// only execute once
static bool dummy = [] {
dealii_initialized_kokkos = true;
Kokkos::initialize();
#if KOKKOS_VERSION >= 30700
const auto settings =
Kokkos::InitializationSettings().set_num_threads(
MultithreadInfo::n_threads());
#else
const Kokkos::InitArguments settings(MultithreadInfo::n_threads());
#endif
Kokkos::initialize(settings);
std::atexit(Kokkos::finalize);
return true;
}();
Expand Down
23 changes: 22 additions & 1 deletion source/base/mpi.cc
Original file line number Diff line number Diff line change
Expand Up @@ -776,7 +776,28 @@ namespace Utilities
#endif

// Initialize Kokkos
Kokkos::initialize(argc, argv);
{
// argv has argc+1 elements and the last one is a nullptr. For appending
// one element we thus create a new argv by copying the first argc
// elements, append the new option, and then a nullptr.
std::vector<char *> argv_new(argc + 2);
for (int i = 0; i < argc; ++i)
argv_new[i] = argv[i];
std::stringstream threads_flag;
#if KOKKOS_VERSION >= 30700
threads_flag << "--kokkos-num-threads=" << MultithreadInfo::n_threads();
#else
threads_flag << "--kokkos-threads=" << MultithreadInfo::n_threads();
#endif
std::string threads_flag_string = threads_flag.str();
argv_new[argc] = const_cast<char *>(threads_flag_string.c_str());
argv_new[argc + 1] = nullptr;
// The first argument in Kokkos::initialzie is of type int&. Hence, we
// need to define a new variable to pass to it (insted of using argc+1
// inline).
drwells marked this conversation as resolved.
Show resolved Hide resolved
int argc_new = argc + 1;
Kokkos::initialize(argc_new, argv_new.data());
}

// we are allowed to call MPI_Init ourselves and PETScInitialize will
// detect this. This allows us to use MPI_Init_thread instead.
Expand Down