set_scratch_size overflows

[bathmatt]

First, this is with the current version as of last night.
I've got a parallel for where I'm setting my level 1 scratch size to

TeamPolicy policy(n_leaves, Kokkos::AUTO, vector_len);
Kokkos::parallel_for(policy.set_scratch_size(1, Kokkos::PerTeam(mem_size), Kokkos::PerThread(0)),*this);

mem_size is (cuda-gdb) p mem_size
$14 = 1045456

or just over a MB. Down in the code later

#15 0x00000000004a4f8a in Kokkos::Impl::ParallelFor<MatrixBuilder, Kokkos::TeamPolicy<Kokkos::Cuda>, Kokkos::Cuda>::ParallelFor (this=0x7fffffff94c0, arg_functor=..., arg_policy=...)
    at /home/mbetten/installs/install-for-drekar/cuda-DEBUG/include/Cuda/Kokkos_Cuda_Parallel.hpp:649
649	      m_scratch_ptr[1] = cuda_resize_scratch_space(m_scratch_size[1]*(Cuda::concurrency()/(m_team_size*m_vector_size)));
(cuda-gdb) p Cuda::concurrency()
$15 = 131072
(cuda-gdb) p m_team_size
$16 = 32
(cuda-gdb) p m_vector_size
$17 = 1

This overflows the 4B memory limit.

This is because team_size is too small and there is too much concurrency. I ask what the team size recommended is and it is 256, but auto sets it to 32 with AUTO. This is an example of AUTO not working. Shouldn't the memory sizes be taken into account on team size?

[crtrott]

I'll fix the bug with the overflow. I also believe that AUTO is more likely to be right than recommended team_size, but it might be there is an issue that it thinks level 1 scratch is actually shared memory. I'll check that.

[bathmatt]

Well, AUTO may be righter but recommended works :) so give me wronger.

[ibaned]

@bathmatt by overflow do you mean the integer overflowed and wrapped around, or do you mean that the amount of memory requested exceeded the limit of cudaMalloc ?

[ibaned]

I was initially confused because your exact numbers actually didn't cause overflow for me, they were barely in the range of uint32_t, but there was still a legitimate overflow potential bug. That will be fixed by #819.