Fix bug with CUDA's team reduction for empty ranges

[masterleinad]

Fixes #5075. This was only triggered when scratch memory was requested.

[JBludau]

looks good to me

[dalg24]

Why do you bother with a reducer?
Why do you initialize result at all?

[masterleinad]

I initialize it so that I can see that it's actually set to zero in the reduction (and not by chance uninitialized zero).
I need the reducer to trigger the bug. Otherwise needs_device_set would not be true.

[dalg24]

But that is what it would default to if you passed a scalar isn't it?

[masterleinad]

kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp

Lines 866 to 874 in ecbfc2f

	const bool is_empty_range = m_league_size == 0 || m_team_size == 0;
	const bool need_device_set = Analysis::has_init_member_function ||
	Analysis::has_final_member_function ||
	!m_result_ptr_host_accessible ||
	#ifdef KOKKOS_CUDA_ENABLE_GRAPHS
	Policy::is_graph_kernel::value ||
	#endif
	!std::is_same<ReducerType, InvalidType>::value;
	if (!is_empty_range || need_device_set) {

Using a reducer I get both has_init_member_function and has_reducer. If I only provide a scalar need_device_set is false.

init: 1
final: 0
host accessible: 1
has reducer: 1

for the reducer case for the different parts of need_device_set

init: 0
final: 0
host accessible: 1
has reducer: 0 

for the scalar case.

[jhux2]

Is the file packages/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp new since the last Kokkos promotion in Trilinos? Is there a possibility of getting a "hot-fix" for the version of Kokkos in Trilinos that I could test?

[dalg24]

Is the file packages/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp new since the last Kokkos promotion in Trilinos?

It is not new. Note that it is a private header so it does not exist as far as users are concerned.

Is there a possibility of getting a "hot-fix" for the version of Kokkos in Trilinos that I could test?

You will have to wait for Kokkos 3.7 (we expect to create a release candidate branch and start testing in two weeks) or try snapshotting on your own.

[jhux2]

@dalg24 Since it's a private header, I'm unsure how the snapshotting works. @ndellingwood, would you be able to help me with a hotfix in Trilinos? I think I can wait the two weeks.

[cz4rs]

Looks good to me.

[rgayatri23]

Retest this please

[dalg24]

Not sure why the status wasn't updated but the tests did pass
https://cloud.cees.ornl.gov/jenkins-ci/blue/organizations/jenkins/Kokkos/detail/Kokkos/8929/pipeline