clang 6.0 + cuda debug out-of-memory test failure

[ndellingwood]

The Apollo Jenkins test with clang 6.0 + cuda, with/without debug enabled, has the cuda.multi_level_scratch test failing due to insufficient memory:

02:56:37 unknown file: Failure
02:56:37 C++ exception with description "cudaMalloc( &ptr, arg_alloc_size ) error( cudaErrorMemoryAllocation): out of memory /home/jenkins/slave/workspace/Kokkos_apollo_debug/kokkos/core/src/Cuda/Kokkos_CudaSpace.cpp:175

The next test cuda.team_long_reduce in debug build also fails with the same error:

02:56:37 unknown file: Failure
02:56:37 C++ exception with description "cudaGetLastError() error( cudaErrorMemoryAllocation): out of memory /home/jenkins/slave/workspace/Kokkos_apollo_debug/kokkos/core/src/Cuda/Kokkos_CudaExec.hpp:401

[ndellingwood]

Edited the issue name to differentiate from another issue #1534.

[Char-Aznable]

Probably related issue. Clang 6 + cuda 9 in debug mode got me this error:

terminating with uncaught exception of type std::runtime_error: cudaGetDeviceCount( & m_cudaDevCount ) error( cudaErrorMemoryAllocation): out of memory /home/aznb/mycodes/kokkos/core/src/Cuda/Kokkos_Cuda_Impl.cpp:203
Traceback functionality not available
Aborted (core dumped)

in my own project, which otherwise runs fine in release mode or built with nvcc

[Char-Aznable]

@ndellingwood Do you know what the status is regarding this issue? I'm still not able to run debug with clang 6 + cuda 9.

[ndellingwood]

@Char-Aznable Are you building with an up-to-date version of Kokkos' develop branch? PR #1588 was intended to address this issue, but it's possible that it was incomplete and something snuck past testing. If you are testing against an updated develop branch we will reopen the issue. Also, if you have a small reproducer that would be helpful.

[Char-Aznable]

I'm using Kokkos latest release. But my case turns out to be a coincident of working with google AddressSanitizer and CUDA together. Just in case anyone trip over the same thing, I'm posting my minimal working example (testdebug.cpp):

#include <Kokkos_Core.hpp>
#include <Kokkos_DualView.hpp>

int main(int argc, char* argv[]) {
  Kokkos::initialize(argc,argv); {

  } Kokkos::finalize();
}

which is then built by cmake with -DCMAKE_CXX_FLAGS_DEBUG="-ggdb3 -O0 -fsanitize=address". Running ./testdebug would give the error:

terminating with uncaught exception of type std::runtime_error: cudaGetDeviceCount( & m_cudaDevCount ) error( cudaErrorMemoryAllocation): out of memory /home/aznb/mycodes/kokkos/core/src/Cuda/Kokkos_Cuda_Impl.cpp:203
Traceback functionality not available

Running instead with ASAN_OPTIONS=protect_shadow_gap=0 ./testdebug is the the correct way of doing this (see google/sanitizers#629: for an explanation)