Kokkos (in Trilinos) Causes Internal Compiler Error on CUDA 8.0.21-EA on POWER8

[nmhamster]

Configured with CUDA 8.0.21 and GCC 5.3.0 on POWER8 I am getting the following (see below). Just listed here for project tracking.

[  0%] Built target KokkosCore_UnitTest_Cuda
[  0%] Building CXX object packages/kokkos/core/unit_test/CMakeFiles/KokkosCore_UnitTest_Default.dir/TestDefaultDeviceType.cpp.o
nvcc_wrapper - *warning* you have set multiple optimization flags (-O*), only the first is used because nvcc can only accept a single optimization setting.
/home/sdhammo/git/trilinos-github-repo/packages/kokkos/core/src/impl/Kokkos_Atomic_Decrement.hpp(61): warning: integer conversion resulted in a change of sign

/home/sdhammo/git/trilinos-github-repo/packages/kokkos/core/src/impl/Kokkos_Atomic_Decrement.hpp(61): warning: integer conversion resulted in a change of sign

Error: Internal Compiler Error (codegen): "there was an error in verifying the lgenfe output!"

make[2]: *** [packages/kokkos/core/unit_test/CMakeFiles/KokkosCore_UnitTest_Default.dir/TestDefaultDeviceType.cpp.o] Error 2
make[1]: *** [packages/kokkos/core/unit_test/CMakeFiles/KokkosCore_UnitTest_Default.dir/all] Error 2
make: *** [all] Error 2

[crtrott]

Is that nvcc or GCC complaining. Do you know?

[nmhamster]

Good question. I am using GCC 5.3.0 on other platforms without problems but I don't know for certain.

[nmhamster]

So I think this is going through nvcc_wrapper and hitting nvcc (note the nvcc_wrapper warnings)

make[2]: Entering directory `/home/sdhammo/git/trilinos-github-repo/build-power8-gcc-cuda-8021'
[  0%] Building CXX object packages/kokkos/core/unit_test/CMakeFiles/KokkosCore_UnitTest_Default.dir/TestDefaultDeviceType.cpp.o
cd /home/sdhammo/git/trilinos-github-repo/build-power8-gcc-cuda-8021/packages/kokkos/core/unit_test && /home/projects/pwr8-rhel72/openmpi/1.10.2/gcc/5.3.0/cuda/8.0.21/bin/mpicxx    -I/home/sdhammo/git/trilinos-github-repo/build-power8-gcc-cuda-8021 -I/home/sdhammo/git/trilinos-github-repo/packages/kokkos/tpls/gtest -I/home/sdhammo/git/trilinos-github-repo/build-power8-gcc-cuda-8021/packages/kokkos/core/src -I/home/sdhammo/git/trilinos-github-repo/packages/kokkos/core/src -I/home/projects/pwr8-rhel72/cuda/8.0.21/include -I/home/sdhammo/git/trilinos-github-repo/build-power8-gcc-cuda-8021/packages/kokkos/core/unit_test -I/home/sdhammo/git/trilinos-github-repo/packages/kokkos/core/unit_test  -pedantic -Wall -Wno-long-long -Wwrite-strings   -O3 -g -fopenmp -mcpu=power8 -std=c++11 -fopenmp   -o CMakeFiles/KokkosCore_UnitTest_Default.dir/TestDefaultDeviceType.cpp.o -c /home/sdhammo/git/trilinos-github-repo/packages/kokkos/core/unit_test/TestDefaultDeviceType.cpp
nvcc_wrapper - *warning* you have set multiple optimization flags (-O*), only the first is used because nvcc can only accept a single optimization setting.
/home/sdhammo/git/trilinos-github-repo/packages/kokkos/core/src/impl/Kokkos_Atomic_Decrement.hpp(61): warning: integer conversion resulted in a change of sign

/home/sdhammo/git/trilinos-github-repo/packages/kokkos/core/src/impl/Kokkos_Atomic_Decrement.hpp(61): warning: integer conversion resulted in a change of sign

Error: Internal Compiler Error (codegen): "there was an error in verifying the lgenfe output!"

make[2]: *** [packages/kokkos/core/unit_test/CMakeFiles/KokkosCore_UnitTest_Default.dir/TestDefaultDeviceType.cpp.o] Error 2
make[2]: Leaving directory `/home/sdhammo/git/trilinos-github-repo/build-power8-gcc-cuda-8021'
make[1]: *** [packages/kokkos/core/unit_test/CMakeFiles/KokkosCore_UnitTest_Default.dir/all] Error 2
make[1]: Leaving directory `/home/sdhammo/git/trilinos-github-repo/build-power8-gcc-cuda-8021'
make: *** [all] Error 2

[nmhamster]

Looks to compile OK with just plain GCC 5.3.0 on POWER8 RHEL72

[crtrott]

Yeah I was suspecting this is nvcc failing. The codegen thingy is something I believe I saw before from it.

[nmhamster]

I thought you had already tested with CUDA 8.0.21 (just the SDK on POWER8), is this right? Did it pass?

[nmhamster]

Confirmed - Trilinos builds successfully with GCC 5.3.0 on POWER8 with RHEL7.2 (no CUDA, but OpenMP backend enabled).

[crtrott]

No I didn't test CUDA 8 on Power. I did on X86 and that works.

[hcedwar]

Retest with latest compilers

[nmhamster]

@hcedwar please hold, we will be updating compilers again very shortly.

[crtrott]

Closing this for now. With Cuda 8.0.27 everything compiles. There are still some random runtime failures.