[CI] Migrate GPU workers to use openrm.

[trivialfis]

todos:

• [WIP] Use memory pool for pinned allocation. #10932

• Add tests with HMM disabled to avoid accessing host memory by accidient.
  https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cuda
  Disable HMM for UVM

  • Create or edit /etc/modprobe.d/uvm.conf.
  • Add or update the following line:

  options nvidia_uvm uvm_disable_hmm=1
  

  • Unload and reload the nvidia_uvm kernel module or reboot the system:

  sudo modprobe -r nvidia_uvm
  sudo modprobe nvidia_uvm

[trivialfis]

@hcho3 Do you think this would be difficult to implement? I tested it on my local machine, but I'm unsure what's required to configure the kernel driver on CI.

[hcho3]

The CI is already using the open-source driver as of dmlc/xgboost-devops#39.

As for uvm_disable_hmm, is it okay if we disable HMM for all jobs? This way, we don't increase the complexity of the CI further.

[trivialfis]

is it okay if we disable HMM for all jobs? This way, we don't increase the complexity of the CI further.

That's a good point. In that case, maybe we should not disable it and instead rely on Windows tests to catch issues. (Windows doesn't support HMM)