Add ROCm utilities and tests, and CUDA fallbacks #40619

fwyzard · 2023-01-26T14:05:09Z

PR description:

Add basic ROCm functions:

hipCheck() macro to check if HIP API functions are successful, or raise an exception;
cms::rocmtest::requireDevices() function to check if the system has any ROCm GPUs, or exit successfully.

Add tests for these functions and for the corresponding CUDA functions.

Add basic ROCm utilities:

rocmIsEnabled exits with success if a ROCm device is available ad configured correctly
rocmComputeCapabilities lists all ROCm devices and their GCN architecture:
```
$ rocmComputeCapabilities 
   0      gfx900    AMD Radeon Pro WX 9100
```

PR validation:

The new unit tests pass on machines with and without GPUs.

fwyzard · 2023-01-26T14:05:39Z

please test with cms-sw/cmsdist#8266

cmsbuild · 2023-01-26T14:14:31Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40619/33893

ERROR: Build errors found during clang-tidy run.

#error("Must define exactly one of __HIP_PLATFORM_AMD__ or __HIP_PLATFORM_NVIDIA__");
 ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02769/el8_amd64_gcc11/external/rocm/5.0.2-95c215630c939706b0552e3eee38861c/include/hip/hip_runtime_api.h:5501:2: error: ("Must define exactly one of __HIP_PLATFORM_AMD__ or __HIP_PLATFORM_NVIDIA__"); [clang-diagnostic-error]
#error("Must define exactly one of __HIP_PLATFORM_AMD__ or __HIP_PLATFORM_NVIDIA__");
 ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02769/el8_amd64_gcc11/external/rocm/5.0.2-95c215630c939706b0552e3eee38861c/include/hip/hip_runtime_api.h:5526:61: error: use of undeclared identifier 'hipHostMallocDefault' [clang-diagnostic-error]
                                       unsigned int flags = hipHostMallocDefault) {
                                                            ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02769/el8_amd64_gcc11/external/rocm/5.0.2-95c215630c939706b0552e3eee38861c/include/hip/hip_runtime_api.h:5532:61: error: use of undeclared identifier 'hipMemAttachGlobal' [clang-diagnostic-error]
                                       unsigned int flags = hipMemAttachGlobal) {
                                                            ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02769/el8_amd64_gcc11/external/rocm/5.0.2-95c215630c939706b0552e3eee38861c/include/hip/hip_vector_types.h:38:2: error: ("Must define exactly one of __HIP_PLATFORM_AMD__ or __HIP_PLATFORM_NVIDIA__"); [clang-diagnostic-error]
#error("Must define exactly one of __HIP_PLATFORM_AMD__ or __HIP_PLATFORM_NVIDIA__");
 ^
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02769/el8_amd64_gcc11/external/rocm/5.0.2-95c215630c939706b0552e3eee38861c/include/hip/library_types.h:48:2: error: ("Must define exactly one of __HIP_PLATFORM_AMD__ or __HIP_PLATFORM_NVIDIA__"); [clang-diagnostic-error]
#error("Must define exactly one of __HIP_PLATFORM_AMD__ or __HIP_PLATFORM_NVIDIA__");
 ^
HeterogeneousCore/ROCmUtilities/src/requireDevices.cc:12:19: error: use of undeclared identifier 'hipGetDeviceCount' [clang-diagnostic-error]
    auto status = hipGetDeviceCount(&devices);
                  ^
Suppressed 213 warnings (212 in non-user code, 1 with check filters).
--
gmake: *** [config/SCRAM/GMake/Makefile.coderules:129: code-checks] Error 2
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

fwyzard · 2023-01-26T14:18:05Z

@makortel these are the basic HIP/ROCm utilities I would add, to ease the implementation of the ROCm service and build rules.

However, we should settle on the "HIP" vs "ROCm" naming approach...

As far as I understand, "ROCm" indicates the backend and the kind of hardware, while "HIP" indicates the API.

Which makes sense: for example, one could program ROCm devices using other APIs, like OpenCL or SYCL, or use HIP to program other devices, like NVIDIA GPUs.

On the other hand, at the moment we do not plan to use HIP on other hardware than ROCm -- though HIP-over-Level 0 is an interesting idea to avoid SYCL...

So far I've used

HIP wherever the API is the most relevant aspect, e.g. hipCheck(...);
ROCm wherever the architecture is the most relevant aspect, e.g. the rocm SCRAM tool and the cms::rocmtest::requireDevices() function.

However, I guess we could also just use "HIP" everywhere (except maybe the tool).

What do you think ?

fwyzard · 2023-01-26T14:18:46Z

-code-checks

I guess the code checks do not use the flags from cms-sw/cmsdist#8266 ?

smuzaffar · 2023-01-26T14:25:43Z

code-checks with cms.week1.PR_92dfa22c/53.0-3fa13e8f691137f6726f64f123800e88

lets run the code-checks using cms-sw/cmsdist#8266 tool-conf

cmsbuild · 2023-01-26T14:54:44Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40619/33895

This PR adds an extra 16KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File HeterogeneousCore/CUDAUtilities/test/BuildFile.xml modified in PR(s): New Heterogeneous Memory Pool #37952, Use cooperative groups to populate Associations (Histograms) in Pixel Patatrack #35713

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40619/33895/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40619/33895/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

fwyzard · 2023-01-26T15:00:38Z

code-checks with cms.week1.PR_92dfa22c/53.0-3fa13e8f691137f6726f64f123800e88

cmsbuild · 2023-01-26T15:20:09Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40619/33898

This PR adds an extra 16KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File HeterogeneousCore/CUDAUtilities/test/BuildFile.xml modified in PR(s): New Heterogeneous Memory Pool #37952, Use cooperative groups to populate Associations (Histograms) in Pixel Patatrack #35713

cmsbuild · 2023-01-30T17:14:15Z

Pull request #40619 was updated. can you please check and sign again.

cmsbuild · 2023-01-30T20:28:21Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1f2297/30252/summary.html
COMMIT: bbcce06
CMSSW: CMSSW_13_0_X_2023-01-30-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/40619/30252/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@bsunanda Run3-alca236 Make small changes to a few scripts in Calibration/HcalCalibAlgos #40632
@bsunanda Phase2-hgc335L Use the right HGCalTopology builder in some of the test scripts #40633
@missirol cmsDriver.py: flush stdout buffer before call to os.execvpe #40635
@mmusich silence static analyzer warning about std::isnan / std::isinf in PackedCandidate #40608
@cms-sw [SCRAM] Fix for iftool with !name cmsdist#8272

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1f2297/30252/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1f2297/30252/git-merge-result

Comparison Summary

Summary:

You potentially added 3 lines to the logs
Reco comparison results: 12 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3555495
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3555467
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 211 log files, 162 edm output root files, 49 DQM output files
TriggerResults: no differences found

cmsbuild · 2023-01-31T16:09:26Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

fwyzard · 2023-01-31T16:34:31Z

please test with cms-sw/cmsdist#8273

cmsbuild · 2023-02-01T03:58:56Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1f2297/30279/summary.html
COMMIT: bbcce06
CMSSW: CMSSW_13_0_X_2023-01-31-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/40619/30279/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1f2297/30279/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1f2297/30279/git-merge-result

Comparison Summary

Summary:

You potentially added 6 lines to the logs
Reco comparison results: 10 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3555495
DQMHistoTests: Total failures: 9
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3555464
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 211 log files, 162 edm output root files, 49 DQM output files
TriggerResults: no differences found

perrotta · 2023-02-01T07:43:01Z

+1

cmsbuild added this to the CMSSW_13_0_X milestone Jan 26, 2023

cmsbuild added code-checks-pending heterogeneous-pending new-package-pending orp-pending pending-signatures tests-pending labels Jan 26, 2023

cmsbuild added requires-external tests-started and removed tests-pending labels Jan 26, 2023

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jan 26, 2023

cmsbuild added code-checks-pending and removed code-checks-rejected labels Jan 26, 2023

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jan 26, 2023

fwyzard force-pushed the Add_CUDA_ROCm_tests branch from 24a4acc to a5e25ed Compare January 26, 2023 15:00

cmsbuild removed the tests-started label Jan 26, 2023

cmsbuild added code-checks-pending tests-pending and removed requires-external code-checks-rejected labels Jan 26, 2023

cmsbuild added tests-approved and removed tests-started labels Jan 30, 2023

fwyzard mentioned this pull request Jan 30, 2023

Implement a first draft of the ROCmServices #40642

Merged

cmsbuild added fully-signed and removed pending-signatures new-package-pending labels Jan 31, 2023

cmsbuild added tests-started and removed tests-approved labels Jan 31, 2023

fwyzard mentioned this pull request Jan 31, 2023

Add a test for ROCm libraries and build rules #40637

Merged

cmsbuild added tests-pending and removed tests-started labels Jan 31, 2023

cmsbuild added tests-started and removed tests-pending labels Jan 31, 2023

cmsbuild added tests-approved and removed tests-started labels Feb 1, 2023

smuzaffar mentioned this pull request Feb 1, 2023

[BuildRules] set TMPDIR so that hipcc can create temp file in cmssw/tmp/package directory cms-sw/cmsdist#8279

Merged

cmsbuild added orp-approved and removed orp-pending labels Feb 1, 2023

cmsbuild merged commit bbd7b72 into cms-sw:master Feb 1, 2023

This was referenced Feb 1, 2023

Update Tensorflow to version 2.11.0 cms-sw/cmsdist#8258

Closed

[Tensorflow] Build with GPU enabled cms-sw/cmsdist#7648

Merged

Custom Run 3 PFScouting NanoAOD #40438

Merged

fwyzard deleted the Add_CUDA_ROCm_tests branch February 2, 2023 12:44

aandvalenzuela mentioned this pull request Feb 21, 2023

[LTO] Unit test testCudaCheck failing for aarch64 architecture #40834

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ROCm utilities and tests, and CUDA fallbacks #40619

Add ROCm utilities and tests, and CUDA fallbacks #40619

fwyzard commented Jan 26, 2023 •

edited

fwyzard commented Jan 26, 2023

cmsbuild commented Jan 26, 2023

fwyzard commented Jan 26, 2023

fwyzard commented Jan 26, 2023

smuzaffar commented Jan 26, 2023

cmsbuild commented Jan 26, 2023

fwyzard commented Jan 26, 2023

cmsbuild commented Jan 26, 2023

cmsbuild commented Jan 30, 2023

cmsbuild commented Jan 30, 2023

cmsbuild commented Jan 31, 2023

fwyzard commented Jan 31, 2023

cmsbuild commented Feb 1, 2023

perrotta commented Feb 1, 2023

Add ROCm utilities and tests, and CUDA fallbacks #40619

Add ROCm utilities and tests, and CUDA fallbacks #40619

Conversation

fwyzard commented Jan 26, 2023 • edited

PR description:

PR validation:

fwyzard commented Jan 26, 2023

cmsbuild commented Jan 26, 2023

fwyzard commented Jan 26, 2023

fwyzard commented Jan 26, 2023

smuzaffar commented Jan 26, 2023

cmsbuild commented Jan 26, 2023

fwyzard commented Jan 26, 2023

cmsbuild commented Jan 26, 2023

cmsbuild commented Jan 30, 2023

cmsbuild commented Jan 30, 2023

Comparison Summary

cmsbuild commented Jan 31, 2023

fwyzard commented Jan 31, 2023

cmsbuild commented Feb 1, 2023

Comparison Summary

perrotta commented Feb 1, 2023

fwyzard commented Jan 26, 2023 •

edited