PGI compiler being passed the gcc -fopenmp flag

[zackgalbreath]

We are running into the following error when attempting to build kokkos (953d796) on summit:

pgc++-Error-Unknown switch: -fopenmp

i.e. it's using the gcc flag, not the PGI one.

Here are the modules we're using:

module load pgi
module load spectrum-mpi
module load cuda
module load cmake

And we are the CMake configuration variables we're using:

-DCMAKE_BUILD_TYPE=Release
-DBUILD_TESTING=OFF
-DKokkos_ENABLE_CUDA=ON
-DKokkos_ENABLE_OPENMP=ON
-DKokkos_ENABLE_SERIAL=ON
-DKokkos_ARCH_VOLTA70=ON
-DKokkos_ARCH_POWER9=ON
-DKokkos_ENABLE_CUDA_LAMBDA=ON
-DKokkos_ENABLE_AGGRESSIVE_VECTORIZATION=ON

Please let us know if we're doing something wrong.

@ascheinb

[masterleinad]

No, it's not your fault. The current logic is

kokkos/cmake/kokkos_enable_devices.cmake

Lines 46 to 54 in 9cd19f1

	COMPILER_SPECIFIC_FLAGS(
	Clang ${ClangOpenMPFlag}
	AppleClang -Xpreprocessor -fopenmp
	PGI -mp
	NVIDIA -Xcompiler -fopenmp
	Cray NO-VALUE-SPECIFIED
	XL -qsmp=omp
	DEFAULT -fopenmp
	)

Since the compiler id is NVIDIA (I guess you are using nvcc_wrapper) we use -Xcompiler -fopenmp instead of -Xcompiler KOKKOS_HOST_OPENMP_FLAG with a KOKKOS_HOST_OPENMP_FLAG to be implemented.

[masterleinad]

@zackgalbreath Can you check if #3127 works for you?

[zackgalbreath]

@zackgalbreath Can you check if #3127 works for you?

Thanks for the quick fix! I can confirm that this works for us on summit

[ascheinb]

@masterleinad @zackgalbreath I'm trying to compile with this fix using the develop branch since it was merged in. On Summit with PGI, getting this error running CMake:

-- Using --c++11;-A for C++11 standard as feature
CMake Error at cmake/kokkos_test_cxx_std.cmake:32 (IF):
  if given arguments:

    "NVIDIA" "STREQUAL" "NVIDIA" "AND" "NOT" "--c++11" "-A" "IN_LIST" "SUPPORTED_NVCC_FLAGS"

  Unknown arguments specified
Call Stack (most recent call first):
  cmake/kokkos_test_cxx_std.cmake:67 (kokkos_set_cxx_standard_feature)
  cmake/kokkos_tribits.cmake:201 (INCLUDE)
  CMakeLists.txt:161 (KOKKOS_SETUP_BUILD_ENVIRONMENT)

Any ideas?

[ascheinb]

I'm guessing --c++11;-A is causing some syntax to break

[masterleinad]

@ascheinb Can you check if you can compile successfully with CUDA support, nvcc_wrapper and PGI as host compiler when you comment the check

kokkos/cmake/kokkos_test_cxx_std.cmake

Lines 32 to 34 in 0970579

	IF (${KOKKOS_CXX_COMPILER_ID} STREQUAL NVIDIA AND NOT ${${STANDARD_NAME}} IN_LIST SUPPORTED_NVCC_FLAGS)
	MESSAGE(FATAL_ERROR "CMake wants to use ${${STANDARD_NAME}} which is not supported by NVCC. Using a more recent host compiler or a more recent CMake version might help.")
	ENDIF()

?
It probably only makes sense to have this check for gcc and clang. Still, I am not quite sure if -A causes some other problems.

[ascheinb]

When I commented out the check, I got a ton of these sorts of errors:

/usr/lib/gcc/ppc64le-redhat-linux/4.8.5/../../../../include/c++/4.8.5/ppc64le-redhat-linux/bits/c++config.h(190): error: identifier "nullptr" is undefined

as if it's not recognizing the c++11 flag...

[masterleinad]

Hmm... Do you know if you also had these problems with previous Kokkos releases?

The -A comes from CMake (https://github.com/Kitware/CMake/search?p=1&q=CMAKE_CXX11_STANDARD_COMPILE_OPTION&unscoped_q=CMAKE_CXX11_STANDARD_COMPILE_OPTION) to enforce ISO-C++-conforming code (https://www.pgroup.com/resources/docs/19.7/x86/pgi-ref-guide/index.htm#a-uc).

Can you give us the full compile line (make VERBOSE=1) for the failing compilation unit?

[masterleinad]

@ascheinb Can you have a look if #3239 works for you?

[ascheinb]

@masterleinad Yes, #3239 (checked out this morning) does work for me when I tried it on summit with PGI 19.10.