GPU optimizations for KOKKOS package

[stanmoore1]

Purpose

Several optimizations for NVIDIA V100 GPUs, especially reducing cost of the global particle/reorder option, which allows one to reorder more often, improving performance of the move algorithm due to better data locality.

For the in.collide benchmark with 10 million particles on a V100 GPU, I see 1.45x overall speedup going from the master branch with particle/reorder 10 to this branch with particle/reorder 5.

Also use better default KOKKOS package options for CPUs vs GPUs, and change the comm classic command to comm serial. The classic option is still accepted for backwards compatibility but deprecated.

Author(s)

Stan Moore (SNL), with helpful discussions/ideas from Evan Weinberg (NVIDIA), @weinbe2

Backward Compatibility

Yes

[stanmoore1]

This also gives a small speedup on the CPU for Kokkos when using global particle/reorder.