Kokkos can not complete unit tests with CUDA UVM enabled

[piyueh]

Hi,

I'm not sure whether I can post my problems here. If here is not the right place, please let me know, thanks!

I want to build Trilinos (Github version) with MPI + CUDA + OpenMP using the package manager Tribits. It says that I need to enable UVM support in Kokkos in order to use Tpetra. However, if I have -D Kokkos_ENABLE_Cuda_UVM=ON in my configuration file, I can not complete unit tests KokkosCore_UnitTest_Cuda_MPI_1. KokkosCore_UnitTest_Cuda_MPI_1 always crashed at cuda.view_subview_auto_1d_left because of _segmentation fault_ (not because of assertion failure).

I use the wrapper nvcc_wrapper Kokkos provided and parameter sm_35 to compile codes. The version of GCC is 4.9.2. And I'm using CUDA 7. My machine has a Tesla K40c, a Tesla K20c and a GT 720. The operation system is Arch Linux.

I also tried to build Kokkos directly using the generate_makefile.bash instead using Tribits system. I tried to enable UVM by directly passing flag -DKOKKOS_USE_CUDA_UVM to the compiler. The procedure is:

$ cd ${KOKKOS_PATH}
$ mkdir build
$ cd build
$ ../generate_makefile.bash --with-cuda --kokkos-path=${KOKKOS_PATH} --compiler=nvcc_wrapper --cxxflags="-DKOKKOS_USE_CUDA_UVM"
$ make build-test
$ make test

And the test still crashed at cuda.view_subview_auto_1d_left because of segmentation fault.

Did I do anything wrong?

Thanks!

P.S. If I build Kokkos without UVM, every thing works fine. But I won't be able to build Tpetra in this case.

[ipdemes]

Hi Pi-Yueh,

Do you set "CUDA_LAUNCH_BLOCKING=1" before running tests?

[piyueh]

Thank you, Irina!
This resolved the problem !
Thank you very much/

[ipdemes]

You are welcome!