Unit test failure on Hansen KokkosCore_UnitTest_Cuda_MPI_1

[bathmatt]

I get a failure on this test.
2/319 Testing KokkosCore_UnitTest_Cuda_MPI_1***Failed

Compiled debug with -lineinfo

[ OK ] cuda.view_nested_view (3 ms)
[ RUN ] cuda.view_subview_auto_1d_left

Program received signal CUDA_EXCEPTION_15, Invalid Managed Memory Access.
TestViewSubview::test_auto_1d<Kokkos::LayoutLeft, Kokkos::Cuda> () at /home/mbetten/Trilinos/Trilinos/packages/kokkos/core/unit_test/TestViewSubview.hpp:124
124 ASSERT_TRUE(X_h(i,j) == ONE);
(cuda-gdb) p i
$1 = 0
(cuda-gdb) p j
$2 = 0
(cuda-gdb) p X_h
$3 = {<Kokkos::ViewTraits<double**, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, void>> = {}, m_ptr_on_device = 0x8900000000,
m_offset_map = {<Kokkos::Impl::Shape<8u, 2u, 0u, 0u, 1u, 1u, 1u, 1u, 1u, 1u>> = {N0 = 10, N1 = 3}, S0 = 10},
m_management = {<Kokkos::Impl::ViewDataHandle<Kokkos::ViewTraits<double**, Kokkos::LayoutLeft, Kokkos::Device<Kokkos::Serial, Kokkos::CudaUVMSpace>, void>, void>> = {}, m_traits = 0}, m_tracker = {
m_alloc_rec = 24043761}}

[bathmatt]

Runs with CUDA_LAUNCH_BLOCKING=1, thought this was the default

[crtrott]

No this is not the default. I experimented with setting it myself in initialize but that doesn't work. My suspicion is that the driver reads in the environment at process start before initialization, and thus even if I set that variable in Kokkos::initialize it won't get picked up. Also I am not sure we would want to set that.

Another thing: setting this variable should only be necessary for the Kokkos tests if you set the default memory space to be the UVM space. The tests which use UVM explicitly correctly call fences.

I am closing the issue.