Add deterministic random number generator

[stanmoore1]

A deterministic random number generator (i.e. it gives the same random numbers regardless of how many threads) would be helpful for debugging, especially for CUDA. I don't care how slow it is.

[crtrott]

Args, I knew you would ask for this. On CPUs it is already deterministic. I have to think about how to do that on the GPU without screwing with the whole GPU backend implementation.

[crtrott]

Would you terribly mind if the randomness of the numbers would degrade somewhat (i.e. if it doesn't reach quite as good a degree of randomness as with what it does during release mode)?

[stanmoore1]

For debugging, correlations in the random numbers don't really matter. It would also be nice for regression testing if I always got the same results on CUDA and could use a tighter tolerance.

[stanmoore1]

I need to think about this though--there may be other sources of non-determinism besides the RNG.

[timattox]

To use the RNG's deterministicly via solution #1066 , the application has to do something like this code snippet to initialize the random pool:

Kokkos::Random_XorShift64_Pool<DeviceType> rand_pool;

if (maxWorkItemCt > maxRNG) {
  rand_pool.init(seed + comm->me, maxWorkItemCt);
  maxRNG = maxWorkItemCt;
}

And then use the workItemID as the index passed to get_state(), like this:

rand_type rand_gen = rand_pool.get_state(workItemID);
// use rand_gen here for this work item.
rand_pool.free_state(rand_gen);

So as long as each workItemID is guaranteed to be active in only one thread at a time, and in the range 0 to (maxWorkItemCt - 1). Thus, when a thread comes back in the future to the same work item, it can resume using the random number sequence where the sequence left off, regardless if it is the same thread doing the work. This does require the size of the pool to be larger than before, since presumably maxWorkItemCt is greater than threadCt.

Note: Depending on how many work items you have, you could have significantly more state saved than when letting Kokkos manage the number of RNG pool items to use.

[ibaned]

The above text refers to the solution implemented in #1066.