Add more cuda/hip reducer tunings

[MrBurmark]

Add more cuda/hip reducer tunings

Add option to initialize on the host for reducers using atomics.
Add option to use an algorithm that avoids device scope fences.

• This PR is a feature
• It does the following (modify list as needed):
  • Adds more reducer tuning options at the request of people using reducers

TODO (incomplete moved to #1635):

• change the arguments in the reduction tuning to enum structs, or restructure away from bools
• add config flag so the hip intrinsic code can be compiled out
• add config flag so the default atomic policy can be the host or non-host policy
• Add support in the new reducer interface for tuning options, this interface should allow single reduction trees and coalesced atomics when multiple reducers are used Will do later
• finalize reducer on host with binary tree reduction
• improve user docs
• improve code docs
• improve gpu reducer test compile times, maybe break up over reduce policies or move loop over policies into the tests increased the time for cuda compiles instead

[MrBurmark]

I'm not sure why the nvcc 10.1.243 + gcc 8.3.1 test is not building, I can build it manually. The other failing tests are timeouts as the reduce tests now build with 6 different reduction policies instead of 1.

[rhornung67]

I'm not sure why the nvcc 10.1.243 + gcc 8.3.1 test is not building, I can build it manually. The other failing tests are timeouts as the reduce tests now build with 6 different reduction policies instead of 1.

That build fails for me same as the CI log shows. The build is configured with desul atomics enabled. Does that work for you?

[MrBurmark]

One option to improve compile times is to expand the reducer policies inside of the reducer tests instead of expanding that into more tests through gtest. That can be done with the newly added for_each_type function. What do you think @rhornung67?

[rhornung67]

One option to improve compile times is to expand the reducer policies inside of the reducer tests instead of expanding that into more tests through gtest. That can be done with the newly added for_each_type function. What do you think @rhornung67?

Sounds reasonable. I think test build times were more of a concern with older vendor compilers (vendors shall not be named here)

[rhornung67]

Looks great! Thank you for all your hard work on this @MrBurmark