Reworking CMake support

[benson31]

These changes are primarily to support two pain points in our current CMake:

1. When CMake added CUDA as a first-class language, it changed the way users are expected to interact with the various features of CUDA (.cu files vs "CUDA toolkit" stuff). We (I) never updated the Aluminum (or Hydrogen (see Update cmake to find_package(CUDAToolkit) Elemental#124) or LBANN) CMakeLists to address this.
2. CMake has recently added first-class language support for HIP as well. This addresses some of the issues I've been having with the ROCm CMake code path, but not all. However, things are stabilizing both on the HIP/ROCm side of the CMakery as well as on the Kitware side such that it is worthwhile to do some updates at this time.

One additional note, mentioned in a comment, is that this bumps the minimum CMake version to 3.21. This is supported locally on our machines, as well as on the external machines that I use. Additionally, Spack can build 3.21, and it's open-source and easy to build for non-Spack users. The reason for this version is that, happily, it is both the version at which HIP was added as a first-class language and the version at which Kitware fixed a bug with the FindCUDAToolkit module that affects machines using the HPC SDK.

TODO:

• Possibly update version? Maybe we do that as a separate commit, though?
• Iron out ROCm fp16 compilation issue.
• Fix the Aluminum export.

[ndryden]

Let's bump the version in a separate commit/PR.

[ndryden]

Looks good to me now.