Add GPU workflow to runTheMatrix #35263

srimanob · 2021-09-14T10:52:27Z

PR description:

The original PR/discussion is #33057 and dmwm/WMCore#10388 from the WMCore side.
Information on GPU support from WMCore side: https://github.com/dmwm/WMCore/wiki/GPU-Support

This PR is to add GPU workflow to runTheMatrix.
In addition, this PR migrates from optparse to argparse, with code clean up.

What to be done with this PR:
(Done)

How can I get the CUDARuntime for each release, to be the default value? @makortel @fwyzard @smuzaffar (Done)
Is GPUParams as expected from WMCore? I just use jsons.dump(the dictionary) as suggested. @amaltaro (Confirmed)

(Pending)

In this PR, I assume that the defined stepname in relval_step or upgradeWorkflowComponents will contain GPU if that step uses GPU. OK?

Thanks.

FYI @dpiparo @davidlange6 @justinasr

PR validation:

Using
runTheMatrix.py --what upgrade -l 11650.502 -t 8 --requires-gpu required --wm init
or
runTheMatrix.py --what upgrade -l 11650.502 -t 8 --gpu --wm init
you will get

'GPUParams': '{"GPUMemoryMB": 8000, "CUDACapabilities": ["6.0", '
                        '"6.1", "6.2", "7.0", "7.2", "7.5"], "CUDARuntime": '
                        '"11.2"}',
'RequiresGPU': 'required',

if this PR is a backport please specify the original PR and why you need to backport that PR:

This PR is not a backport.

cmsbuild · 2021-09-14T10:58:26Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35263/25249

This PR adds an extra 24KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/scripts/runTheMatrix.py modified in PR(s): add 11634.7 wf for 2021 ttbar mkFit to the short matrix #35235

cmsbuild · 2021-09-14T10:58:48Z

A new Pull Request was created by @srimanob (Phat Srimanobhas) for master.

It involves the following packages:

Configuration/PyReleaseValidation (pdmv, upgrade)

@jordan-martins, @chayanit, @bbilin, @wajidalikhan, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @fabiocos, @slomeo, @kpedro88 this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

srimanob · 2021-09-14T11:02:20Z

@cmsbuild please test

fwyzard · 2021-09-14T11:47:41Z

* How can I get the `CUDARuntime` for each release, to be the default value?

Good question.

One option is to read it from the version of the CUDA library bundled with CMSSW, e.g.:

ls $(scram tool tag cuda LIBDIR)/libcudart.so.*.*.* | sed -e 's/.*libcudart.so.//'

Another option is to add it as variable to SCRAM.

Another good question is: what version should we use here ? 11.2.152 or 11.2 ?

fwyzard

I'd make all options lowercase and use a dash to separate words, like --gpu-version.

Configuration/PyReleaseValidation/scripts/runTheMatrix.py

fwyzard · 2021-09-14T11:58:02Z

Configuration/PyReleaseValidation/scripts/runTheMatrix.py

@@ -305,6 +334,41 @@ def runSelected(opt):
                      default=None,
                      action='store')

+    parser.add_option('--RequiresGPU',
+                      help='if GPU is reuired or not: forbidden (default, CPU-only), optional, required. For relvals, the GPU option will be turned off for optional.',


reuired -> required
Also, why the special treatment for relvals ?

For relvals, I assume that we always control what relvals should run on, w/ or w/o (specific) GPU. So optional may not fit with the relvals. But this can be changed, no solid reason here.

fwyzard · 2021-09-14T12:02:28Z

Configuration/PyReleaseValidation/scripts/runTheMatrix.py

+                      dest='CUDADriverVersion',
+                      default='')
+
+    parser.add_option('--CUDARuntimeVersion',


I'll double check, but I don't think this should ever be set on the client side ?

Configuration/PyReleaseValidation/scripts/runTheMatrix.py

amaltaro · 2021-09-14T12:44:42Z

Besides the GPUMemoryMB comment that I made along the code, all the rest looks good to me, Phat.
I think a plus to this PR would be if the optional GPUParams would be skipped in case they are not defined (with a value different than its default). Providing them with the default empty string should be supported though.

cmsbuild · 2021-09-14T14:19:22Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b0c6cc/18586/summary.html
COMMIT: 7d8f062
CMSSW: CMSSW_12_1_X_2021-09-13-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/35263/18586/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

The workflows 140.53 have different files in step1_dasquery.log than the ones found in the baseline. You may want to check and retrigger the tests if necessary. You can check it in the "files" directory in the results of the comparisons

Summary:

No significant changes to the logs found
Reco comparison results: 1299 differences found in the comparisons
DQMHistoTests: Total files compared: 39
DQMHistoTests: Total histograms compared: 3001001
DQMHistoTests: Total failures: 3671
DQMHistoTests: Total nulls: 19
DQMHistoTests: Total successes: 2997289
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -45.703 KiB( 38 files compared)
DQMHistoSizes: changed ( 140.53 ): -44.531 KiB Hcal/DigiRunHarvesting
DQMHistoSizes: changed ( 140.53 ): -1.172 KiB RPC/DCSInfo
Checked 165 log files, 37 edm output root files, 39 DQM output files
TriggerResults: no differences found

cmsbuild · 2021-09-14T15:06:31Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35263/25256

This PR adds an extra 24KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/scripts/runTheMatrix.py modified in PR(s): add 11634.7 wf for 2021 ttbar mkFit to the short matrix #35235

cmsbuild · 2021-09-15T15:57:00Z

Pull request #35263 was updated. @jordan-martins, @makortel, @chayanit, @bbilin, @wajidalikhan, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen, @fwyzard can you please check and sign again.

srimanob · 2021-09-15T16:00:44Z

@cmsbuild please test

cmsbuild · 2021-09-15T18:55:29Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b0c6cc/18634/summary.html
COMMIT: de6a246
CMSSW: CMSSW_12_1_X_2021-09-14-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/35263/18634/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 39
DQMHistoTests: Total histograms compared: 3000833
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3000805
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
Checked 165 log files, 37 edm output root files, 39 DQM output files
TriggerResults: no differences found

srimanob · 2021-09-17T11:11:11Z

@fwyzard @kpedro88
Do you have additional comments to this PR? Thanks.

Maybe we can merge, and open the next PR to improve runThematrix.

fwyzard · 2021-09-17T11:40:09Z

Maybe we can merge, and open the next PR to improve runThematrix.

sure

fwyzard · 2021-09-17T11:40:14Z

+heterogeneous

srimanob · 2021-09-17T13:13:45Z

+Upgrade

qliphy · 2021-09-21T08:52:51Z

@cms-sw/pdmv-l2 Do you have any comment?

kskovpen · 2021-09-21T08:57:02Z

+1

cmsbuild · 2021-09-21T08:57:23Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

qliphy · 2021-09-21T09:40:31Z

+1

Norraphat added 2 commits September 14, 2021 12:38

add GPU to runTheMatrix

cc07c62

add GPU to runTheMatrix

7d8f062

cmsbuild added this to the CMSSW_12_1_X milestone Sep 14, 2021

cmsbuild added code-checks-pending orp-pending pdmv-pending pending-signatures tests-pending upgrade-pending labels Sep 14, 2021

srimanob mentioned this pull request Sep 14, 2021

Add support for GPU parameters at ReReco spec level dmwm/WMCore#10388

Closed

cmsbuild added code-checks-approved and removed code-checks-pending labels Sep 14, 2021

cmsbuild added tests-started and removed tests-pending labels Sep 14, 2021

fwyzard reviewed Sep 14, 2021

View reviewed changes

amaltaro reviewed Sep 14, 2021

View reviewed changes

Configuration/PyReleaseValidation/scripts/runTheMatrix.py Outdated Show resolved Hide resolved

cmsbuild added tests-approved and removed tests-started labels Sep 14, 2021

update runTheMatrix following comments

5cec229

cmsbuild added code-checks-pending tests-pending and removed tests-approved code-checks-approved labels Sep 14, 2021

cmsbuild added tests-started and removed tests-pending labels Sep 15, 2021

cmsbuild added tests-approved and removed tests-started labels Sep 15, 2021

cmsbuild added heterogeneous-approved and removed heterogeneous-pending labels Sep 17, 2021

cmsbuild added upgrade-approved and removed upgrade-pending labels Sep 17, 2021

cmsbuild added fully-signed pdmv-approved and removed pending-signatures pdmv-pending labels Sep 21, 2021

cmsbuild added orp-approved and removed orp-pending labels Sep 21, 2021

cmsbuild merged commit 5d10e5e into cms-sw:master Sep 21, 2021

This was referenced Sep 21, 2021

Protect storage accounting UDP messages from NaN #35362

Merged

Remove unused magnetic field geometry tags from all GTs #35363

Merged

smuzaffar mentioned this pull request Oct 7, 2021

runTheMatrix.py --command option broken #35565

Closed

srimanob mentioned this pull request Jul 4, 2023

Migrate from optparse to argparse #29338

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GPU workflow to runTheMatrix #35263

Add GPU workflow to runTheMatrix #35263

srimanob commented Sep 14, 2021 •

edited

cmsbuild commented Sep 14, 2021

cmsbuild commented Sep 14, 2021

srimanob commented Sep 14, 2021

fwyzard commented Sep 14, 2021

fwyzard left a comment

fwyzard Sep 14, 2021

srimanob Sep 14, 2021

fwyzard Sep 14, 2021

amaltaro commented Sep 14, 2021

cmsbuild commented Sep 14, 2021

cmsbuild commented Sep 14, 2021

cmsbuild commented Sep 15, 2021

srimanob commented Sep 15, 2021

cmsbuild commented Sep 15, 2021

srimanob commented Sep 17, 2021

fwyzard commented Sep 17, 2021

fwyzard commented Sep 17, 2021

srimanob commented Sep 17, 2021

qliphy commented Sep 21, 2021

kskovpen commented Sep 21, 2021

cmsbuild commented Sep 21, 2021

qliphy commented Sep 21, 2021

Add GPU workflow to runTheMatrix #35263

Add GPU workflow to runTheMatrix #35263

Conversation

srimanob commented Sep 14, 2021 • edited

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented Sep 14, 2021

cmsbuild commented Sep 14, 2021

srimanob commented Sep 14, 2021

fwyzard commented Sep 14, 2021

fwyzard left a comment

Choose a reason for hiding this comment

fwyzard Sep 14, 2021

Choose a reason for hiding this comment

srimanob Sep 14, 2021

Choose a reason for hiding this comment

fwyzard Sep 14, 2021

Choose a reason for hiding this comment

amaltaro commented Sep 14, 2021

cmsbuild commented Sep 14, 2021

Comparison Summary

cmsbuild commented Sep 14, 2021

cmsbuild commented Sep 15, 2021

srimanob commented Sep 15, 2021

cmsbuild commented Sep 15, 2021

Comparison Summary

srimanob commented Sep 17, 2021

fwyzard commented Sep 17, 2021

fwyzard commented Sep 17, 2021

srimanob commented Sep 17, 2021

qliphy commented Sep 21, 2021

kskovpen commented Sep 21, 2021

cmsbuild commented Sep 21, 2021

qliphy commented Sep 21, 2021

srimanob commented Sep 14, 2021 •

edited