Triple nested parallelism still fails on bowman

[ibaned]

https://jenkins-son.sandia.gov/job/Kokkos_SLURM_inner_test/6885/console

[crtrott]

This is definitely our big white whale ...

[hcedwar]

Suspect fixed by #1240

[crtrott]

This failed again on KNL ....

[ndellingwood]

Our arch-enemy has returned...
This showed up again on Bowman today
Link

[crtrott]

Looking into this once more. Main issue is with pthreads. I couldn't reproduce it with OpenMP yet.

This code also fails (very occasionally, like 1 in a 1000 or so):

    Kokkos::parallel_reduce(
        team_policy(nrows / chunk_size, team_size, vector_length),
        KOKKOS_LAMBDA(const member_type &teamMember, double &update) {
          const int row_start = teamMember.league_rank() * chunk_size;
          const int row_end   = row_start + chunk_size;
          Kokkos::parallel_for(
              Kokkos::TeamThreadRange(teamMember, row_start, row_end),
              [&](const int i) {
                ScalarType sum_i = 0.0;
#if 1
for(unsigned int j=0; j<ncols; j++) sum_i+=A(i,j)*x(j);
#else
                Kokkos::parallel_reduce(
                    Kokkos::ThreadVectorRange(teamMember, ncols),
                    [&](const int j, ScalarType &innerUpdate) {
                      innerUpdate += A(i, j) * x(j);
                    },
                    sum_i);
#endif
                //Kokkos::atomic_add(&control(),sum_i);
                Kokkos::atomic_add(&count(),1l);
                Kokkos::single(Kokkos::PerThread(teamMember),
                               [&]() { update += y(i) * sum_i; });
              });
        },
        result);
    Kokkos::fence();

When this fails count() is still correct so it looks like every iteration is executed. Also this obviously means its not the inner most loop ...

[ndellingwood]

Recent failure with Pthreads on Bowman, intel/compilers/17.2.174

Failure output:

2: [ RUN      ] threads.triple_nested_parallelism
2:   TestTripleNestedReduce failed solution(16777216) != result(16318464), nrows(8192) ncols(2048) league_size(64) team_size(32)
2: /home/jenkins/bowman/workspace/Kokkos_SLURM_inner_test@4/core/unit_test/TestTeamVector.hpp:993: Failure
2: Value of: result
2:   Actual: 1.63185e+07
2: Expected: solution
2: Which is: 1.67772e+07

Link to failing nightly build