Apollo cuda.team_broadcast test fail with clang-6.0

[ndellingwood]

The Jenkins build from Apollo reported test failure with cuda.team_broadcast.

This was with the clang-6.0-Cuda_Pthread-release and clang-6.0-Cuda_Serial-release builds.

@vqd8a do you time to look into this? Runtime error is posted below:

[ RUN      ] cuda.team_broadcast
<https://jenkins-son.sandia.gov/job/Kokkos_apollo/ws/kokkos/core/unit_test/TestTeam.hpp>:1067: Failure
Value of: size_t( total )
  Actual: 960
Expected: size_t( expected_result )
Which is: 5120
<https://jenkins-son.sandia.gov/job/Kokkos_apollo/ws/kokkos/core/unit_test/TestTeam.hpp>:1067: Failure
Value of: size_t( total )
  Actual: 960
Expected: size_t( expected_result )
Which is: 5120
<https://jenkins-son.sandia.gov/job/Kokkos_apollo/ws/kokkos/core/unit_test/TestTeam.hpp>:1067: Failure
Value of: size_t( total )
  Actual: 72192
Expected: size_t( expected_result )
Which is: 385024
<https://jenkins-son.sandia.gov/job/Kokkos_apollo/ws/kokkos/core/unit_test/TestTeam.hpp>:1067: Failure
Value of: size_t( total )
  Actual: 72192
Expected: size_t( expected_result )
Which is: 385024
<https://jenkins-son.sandia.gov/job/Kokkos_apollo/ws/kokkos/core/unit_test/TestTeam.hpp>:1067: Failure
Value of: size_t( total )
  Actual: 287904000
Expected: size_t( expected_result )
Which is: 1535488000
<https://jenkins-son.sandia.gov/job/Kokkos_apollo/ws/kokkos/core/unit_test/TestTeam.hpp>:1067: Failure
Value of: size_t( total )
  Actual: 287904000
Expected: size_t( expected_result )
Which is: 1535488000

[ndellingwood]

@vqd8a here are some instructions for building on Apollo for this case:

Modules:
module load clang/6.0 cuda/9.0.69

Makefile:
generate_makefile.bash --with-cuda --with-openmp --compiler=clang++ --arch="SNB,Volta70"

No need to salloc a node or anything like that.

[vqd8a]

thanks, @ndellingwood

[crtrott]

This is actually the parallel_reduce(TeamThreadRange) which produces the wrong result in the test. The broadcast appears to be fine ...

[crtrott]

Have a fix which actually reduces one of the many reduction implementations for CUDA in Kokkos. Spotchecks running now. Will post pull request afterwards.

[vqd8a]

@crtrott thanks for fixing this. I just figured out team_broadcast working fine yesterday afternoon then you already fixed the team reduce.