Use the same flags in Makefile.kokkos for POWER7/8/9 as for CMake

[masterleinad]

Fixes #4247. Note that this is also the logic used for Power7 already.

[stanmoore1]

Thanks @masterleinad!

[masterleinad]

The corresponding CMake options are

kokkos/cmake/kokkos_arch.cmake

Lines 335 to 358 in f9f0d3d

	IF (KOKKOS_ARCH_POWER7)
	COMPILER_SPECIFIC_FLAGS(
	COMPILER_ID KOKKOS_CXX_HOST_COMPILER_ID
	NVHPC NO-VALUE-SPECIFIED
	DEFAULT -mcpu=power7 -mtune=power7
	)
	SET(KOKKOS_USE_ISA_POWERPCBE ON)
	ENDIF()
	IF (KOKKOS_ARCH_POWER8)
	COMPILER_SPECIFIC_FLAGS(
	COMPILER_ID KOKKOS_CXX_HOST_COMPILER_ID
	NVHPC -tp=pwr8
	DEFAULT -mcpu=power8 -mtune=power8
	)
	ENDIF()
	IF (KOKKOS_ARCH_POWER9)
	COMPILER_SPECIFIC_FLAGS(
	COMPILER_ID KOKKOS_CXX_HOST_COMPILER_ID
	NVHPC -tp=pwr9
	DEFAULT -mcpu=power9 -mtune=power9
	)
	ENDIF()

.

[dalg24]

Why did you keep ifeq ($(KOKKOS_INTERNAL_COMPILER_PGI), 1)?

[masterleinad]

Why did you keep ifeq ($(KOKKOS_INTERNAL_COMPILER_PGI), 1)?

We discussed before that PGI/NVC++ by default optimizes for the host architecture so that not using flags here should be fine. I didn't want to discuss/change that here but just fixing flags for other compilers that don't do that.
The problem with the PGI flags is that their existence very much depends on the installation (and the version). Of course, you could argue that compilers on Power architectures should be properly configured so that this works.
Anyway, if we want the behavior for Power architectures to be consistent as well, I am happy to change that here, too.

On a separate note, I'm not sure whether Makefile.kokkos recognizes nvc++ at all.