Fix compiler flags for PGI/NVHPC

[masterleinad]

-tp=host and native are not accepted by pgc++ 20.1 (only -tp=pwr8 and -tp=pwr9) and nvc++ accepts -tp=host, -tp=native, -tp=host, -tp=pwr8 and -tp=pwr9. Thus, using -tp=pwr8 and -tp=pwr9 seems to be the best compromise. I don't know if these flags are available for older PGI versions (and if we care if not).
Also, there was another branch where we need to replace PGI by NVHPC in our CMake setup.

[DavidPoliakoff]

Does this need to be reflected in the Makefile system?

[dalg24]

In our README we say we support 18.7

kokkos/README.md

Line 71 in 0530271

* PGI 18.7

although we only enforce it is at least 17.4 in cmake

kokkos/cmake/kokkos_compiler_id.cmake

Line 141 in 0530271

SET(KOKKOS_MESSAGE_TEXT "${KOKKOS_MESSAGE_TEXT}\n PGI 17.4 or higher\n")

and in our code

kokkos/core/src/Kokkos_Macros.hpp

Line 185 in 0530271

#if (1740 > KOKKOS_COMPILER_PGI)

According the 18.7 documentation does not list pwr8 nor pwr9
https://docs.nvidia.com/hpc-sdk/pgi-compilers/18.7/x86/pgi-ref-guide/index.htm#tp
Neither does the 20.1 doc even though it is there on Summit. Oh fun...

[dalg24]

Does this need to be reflected in the Makefile system?

diff --git a/Makefile.kokkos b/Makefile.kokkos
index ace24d63f..fe0e055f1 100644
--- a/Makefile.kokkos
+++ b/Makefile.kokkos
@@ -910,7 +910,8 @@ ifeq ($(KOKKOS_INTERNAL_USE_ARCH_POWER8), 1)
   tmp := $(call kokkos_append_header,"$H""define KOKKOS_ARCH_POWER8")
 
   ifeq ($(KOKKOS_INTERNAL_COMPILER_PGI), 1)
-
+        KOKKOS_CXXFLAGS += -tp=pwr8
+        KOKKOS_LDFLAGS  += -tp=pwr8
   else
     ifeq ($(KOKKOS_INTERNAL_COMPILER_XL), 1) 
         KOKKOS_CXXFLAGS += -mcpu=power8 -mtune=power8
@@ -931,7 +932,8 @@ ifeq ($(KOKKOS_INTERNAL_USE_ARCH_POWER9), 1)
   tmp := $(call kokkos_append_header,"$H""define KOKKOS_ARCH_POWER9")
 
   ifeq ($(KOKKOS_INTERNAL_COMPILER_PGI), 1)
-
+        KOKKOS_CXXFLAGS += -tp=pwr9
+        KOKKOS_LDFLAGS  += -tp=pwr9
   else
     ifeq ($(KOKKOS_INTERNAL_COMPILER_XL), 1) 
         KOKKOS_CXXFLAGS += -mcpu=power9 -mtune=power9

[dalg24]

This change is non obvious to me. Can you elaborate?

[masterleinad]

The ideas was to unify PGI and NVHPC so that we don't need to distinguish between different CMake versions. The idea was that nvc++ should accept the same flags as pgc++.
We already did this translation a few lines higher if KOKKOS_CXX_COMPILER_ID was PGI. This didn't account for the case that KOKKOS_CXX_COMPILER_ID was NVIDIA and KOKKOS_CXX_HOST_COMPILER_ID was PGI. This line fixes it.

[masterleinad]

Does this need to be reflected in the Makefile system?

We should agree if we want to add that or not. In the corresponding Slack discussion, it was said that the compiler will optimize for the host compiler by default and thus this compiler option is only really relevant for cross-compilation.

[dalg24]

OK looks good to me regardless of the makefile support