Incompatibility of flags for C++ standard with PGI v20.4 on Power9/NVIDIA V100 system

[rhager]

I am running into a problem with CMake on a Power9/NVIDIA V100 system with the PGI compiler. I use

• PGI 20.4
• Cudatoolkit 11.0
• OpenMPI 4.0.4
• Kokkos v3.1.00

I run

export NVCC_WRAPPER_DEFAULT_COMPILER=pgc++
export CXX=${KOKKOS_SRC_DIR}/bin/nvcc_wrapper
cd ${KOKKOS_BUILD_DIR}
ccmake ${KOKKOS_SRC_DIR}

The configure phase then fails with the following error:

 CMake Error at cmake/kokkos_test_cxx_std.cmake:43 (MESSAGE):
   Compiler NVIDIA should support cxx_std_11, but CMake reports feature not
   supported
 Call Stack (most recent call first):
   cmake/kokkos_test_cxx_std.cmake:58 (kokkos_set_cxx_standard_feature)
   cmake/kokkos_tribits.cmake:180 (INCLUDE)
   CMakeLists.txt:156 (KOKKOS_SETUP_BUILD_ENVIRONMENT)

The system admins came up with the following workaround:

The core issue is that cmake
detects pgi as the C++ compiler (which is correct) so it tries to use
"--c++11 -A" and similar options for turning on C++11 features (e.g.
CMAKE_CXX11_STANDARD_COMPILE_OPTION from
/usr/share/cmake/Modules/Compiler/PGI-CXX.cmake). Unfortunately you are
telling it to use nvcc (via nvcc_wrapper) which wants such options as
"--std c++11" (and NO -A).

nvcc and nvcc_wrapper do attempt to pass on --c++11 -A as options to
pgc++ but they do it inconsistently. They do use it for pgc++ which
makes the intermediate .ii file but for the next step they use cicc from
vcuda toolkit which defaults to C++14 (because it did not understand
--c++11 option) and all hell breaks loose.

I can give you a workaround:

diff --git a/bin/nvcc_wrapper b/bin/nvcc_wrapper
index 8a23d0d6..b447f131 100755
--- a/bin/nvcc_wrapper
+++ b/bin/nvcc_wrapper
@@ -194,6 +194,14 @@ do
    -rdc=*|-maxrregcount*|--maxrregcount*)
      cuda_args="$cuda_args $1"
      ;;
+  --c++17|--c++14|--c++11)
+    shared_args="$shared_args --std ${1#--}"
+    if [ "x$2" = "x-A" ]; then
+           shift
+    fi
+    ;;
+  -A)
+    ;;
    #Handle unsupported standard flags
--std=c++1y|-std=c++1y|--std=c++1z|-std=c++1z|--std=gnu++1y|-std=gnu++1y|--std=gnu++1z|-std=gnu++1z|--std=c++2a|-std=c++2a|--std=c++17|-std=c++17)
      fallback_std_flag="-std=c++14"

This resolves the configuration problem and I can build Kokkos with the CUDA backend.

[masterleinad]

I created a pull request (#3254) slightly changing what you propose. Can you please check if that works for you as well?

[rhager]

Hi @masterleinad ! The branch you suggested solves both problems I had with the 3.1.00 tag:

• The one described in this issue with the C++ standard options, and
• nvcc_wrapper using the wrong compiler flags for OpenMP with the PGI compiler (-fopenmp instead of -mp)

Thanks a lot!