Potential data race in Cuda parallel reduce

[tcclevenger]

See comment #6217 (comment). Suggested in #4855 (comment) as being a potential issue, I think this is an actual issue. At the very least we avoid doing computation that is unnecessary, and it clears a warning when running compute-sanitizer --tool=racecheck (see #6217).

[Rombur]

We probably need the same fix for HIP

[PhilMiller]

We probably need the same fix for HIP

@Rombur In which case, the fixes from #4855 would likely be applicable as well.

[masterleinad]

Have you measured if you see any impact in performance for, say https://github.com/kokkos/kokkos-tutorials/tree/main/Exercises/04/Solution?

[tcclevenger]

Have you measured if you see any impact in performance for, say https://github.com/kokkos/kokkos-tutorials/tree/main/Exercises/04/Solution?

Yes, the differences were negligible (<2% relative increase/decrease between this branch and develop commit it is based on, various problem sizes, launching p_for/reduce kernels).

[cz4rs]

Looks good to me 👍

We probably need the same fix for HIP

I propose we merge this as is and fix HIP with a separate issue / PR.

[PhilMiller]

Retest this please

[PhilMiller]

The previous Jenkins failure was long enough ago that the logs have been discarded

[PhilMiller]

Looks good to me 👍

We probably need the same fix for HIP

I propose we merge this as is and fix HIP with a separate issue / PR.

#6348

[crtrott]

Retest this please.

[PhilMiller]

CUDA builds all passed. None of the pending Jenkins builds is a CUDA backend, so this should be safe to merge.