Autodetect if a GPU can be used with CMSSW (11.3.x)

[fwyzard]

PR description:

Autodetect if a GPU can be used with CMSSW.
A GPU is deemed usable if

• it is visible to cudaGetDeviceCount();
• it can be selected by cudaSetDevice();
• it can allocate a small amount of GPU memory;
• it can run a kernel.

The last point in particular ensures that CMSSW has been built to support the GPUs "compute capability" (e.g. 7.0 for a Tesla v100, etc.).

cudaComputeCapabilities now prints (unsupported) after a GPU name if it cannot be used by CMSSW.

PR validation:

With a version of CMSSW built to support only compute capability 7.5:

$ cudaComputeCapabilities 
   0     7.0    NVIDIA Tesla V100-PCIE-32GB (unsupported)
   1     7.5    NVIDIA Tesla T4

$ cudaIsEnabled && echo true || echo false
true

After hiding the usable GPU:

$ export CUDA_VISIBLE_DEVICES=0

$ cudaComputeCapabilities 
   0     7.0    NVIDIA Tesla V100-PCIE-32GB (unsupported)

$ cudaIsEnabled && echo true || echo false
false

After hiding the unusable GPU:

$ export CUDA_VISIBLE_DEVICES=1

$ cudaComputeCapabilities 
   0     7.5    NVIDIA Tesla T4

$ cudaIsEnabled && echo true || echo false
true

After hiding all GPUs (or on a machine without GPUs):

$ export CUDA_VISIBLE_DEVICES= 

$ cudaComputeCapabilities 
cudaComputeCapabilities: no CUDA-capable device is detected

$ cudaIsEnabled && echo true || echo false
false

If this PR is a backport please specify the original PR and why you need to backport that PR:

Backport of #33561

[cmsbuild]

A new Pull Request was created by @fwyzard (Andrea Bocci) for CMSSW_11_3_X.

It involves the following packages:

HeterogeneousCore/CUDAServices

@makortel, @cmsbuild, @fwyzard can you please review it and eventually sign? Thanks.
@makortel, @rovere this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

• Backported from Autodetect if a GPU can be used with CMSSW #33561

[cmsbuild]

Pull request #33562 was updated. @makortel, @cmsbuild, @fwyzard can you please check and sign again.

[fwyzard]

enable gpu

[fwyzard]

please test

[fwyzard]

backport #33561

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-499295/14662/summary.html
COMMIT: cc325b9
CMSSW: CMSSW_11_3_X_2021-04-27-2300/slc7_amd64_gcc900
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/33562/14662/install.sh to create a dev area with all the needed externals and cmssw changes.

GPU Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 4
• DQMHistoTests: Total histograms compared: 9559
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 9559
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
• Checked 12 log files, 9 edm output root files, 4 DQM output files
• TriggerResults: no differences found

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 38
• DQMHistoTests: Total histograms compared: 2877046
• DQMHistoTests: Total failures: 7
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2877017
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 37 files compared)
• Checked 160 log files, 37 edm output root files, 38 DQM output files
• TriggerResults: no differences found

[makortel]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_11_3_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_0_X is complete. This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[qliphy]

+1