Update to CUDA 11.4.4 #7641

fwyzard · 2022-02-22T13:58:17Z

Update to CUDA 11.4.4:

CUDA runtime version 11.4.148
NVIDIA drivers version 470.82.01

Fixes various Apache Log4J vulnerabilities.

See https://docs.nvidia.com/cuda/archive/11.4.4/cuda-toolkit-release-notes/index.html for the full CUDA 11.4.x release notes and change log.

Update to CUDA 11.4.4: * CUDA runtime version 11.4.148 * NVIDIA drivers version 470.82.01 Fixes various Apache Log4J vulnerabilities. See https://docs.nvidia.com/cuda/archive/11.4.4/cuda-toolkit-release-notes/index.html for the full CUDA 11.4.x release notes and change log.

fwyzard · 2022-02-22T13:58:23Z

enable gpu

fwyzard · 2022-02-22T13:58:27Z

please test

cmsbuild · 2022-02-22T13:58:39Z

A new Pull Request was created by @fwyzard (Andrea Bocci) for branch IB/CMSSW_12_3_X/master.

@smuzaffar, @iarspider can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @qliphy you are the release manager for this.
cms-bot commands are listed here

fwyzard · 2022-02-22T13:59:24Z

@smuzaffar, I don't know where Log4J is used in the CUDA tools, but should we backport this to 12.2.x as well ?

fwyzard · 2022-02-22T13:59:34Z

type bugfix

smuzaffar · 2022-02-22T14:11:35Z

@fwyzard Looking at https://nvidia.custhelp.com/app/answers/detail/a_id/5294 , the only product which are affected by this are NETQ and "VGPU SOFTWARE LICENSE SERVER" . I think we are not using any of these so no need to backport

cmsbuild · 2022-02-23T00:28:11Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b874d9/22578/summary.html
COMMIT: 9a8bc1e
CMSSW: CMSSW_12_3_X_2022-02-22-1100/slc7_amd64_gcc10
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7641/22578/install.sh to create a dev area with all the needed externals and cmssw changes.

GPU Comparison Summary

Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 0 differences found in the comparisons
Reco comparison had 3 failed jobs
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 19811
DQMHistoTests: Total failures: 10
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 19801
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
Checked 12 log files, 9 edm output root files, 4 DQM output files
TriggerResults: found differences in 3 / 3 workflows

Comparison Summary

Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3965137
DQMHistoTests: Total failures: 10979
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3954135
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 10144.568 KiB( 48 files compared)
DQMHistoSizes: changed ( 11634.0,... ): 1690.762 KiB HLT/Muon
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 204 log files, 45 edm output root files, 49 DQM output files
TriggerResults: found differences in 7 / 48 workflows

smuzaffar · 2022-02-23T07:08:09Z

please test for slc7_aarch64_gcc11

smuzaffar · 2022-02-23T07:08:29Z

please test for slc7_ppc64le_gcc11

cmsbuild · 2022-02-23T15:52:23Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b874d9/22600/summary.html
COMMIT: 9a8bc1e
CMSSW: CMSSW_12_3_X_2022-02-21-2300/slc7_ppc64le_gcc11
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7641/22600/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test testFWCoreUtilities had ERRORS
---> test DRNTest had ERRORS
---> test materialBudgetTrackerPlots had ERRORS

cmsbuild · 2022-02-23T15:55:18Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b874d9/22601/summary.html
COMMIT: 9a8bc1e
CMSSW: CMSSW_12_3_X_2022-02-21-2300/slc7_aarch64_gcc11
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7641/22601/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation warning when building: See details on the summary page.

cmsbuild · 2022-02-24T00:44:35Z

-1

Failed Tests: UnitTests RelVals AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b874d9/22624/summary.html
COMMIT: 9a8bc1e
CMSSW: CMSSW_12_3_X_2022-02-22-2300/slc7_aarch64_gcc11
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7641/22624/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b874d9/22624/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b874d9/22624/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test testFWCoreUtilities had ERRORS
---> test TestFWCoreServicesDriver had ERRORS
---> test test_edmPickEvents had ERRORS
---> test TestDQMOnlineClient-visualization_secondInstance had ERRORS
and more ...

RelVals

4.53

----- Begin Fatal Exception 24-Feb-2022 00:35:07 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 194533 lumi: 329 event: 462355458 stream: 0
   [1] Running path 'dqmofflineOnPAT_1_step'
   [2] Prefetching for module SingleTopTChannelLeptonDQM_miniAOD/'singleTopElectronMediumDQM_miniAOD'
   [3] Prefetching for module PATMuonSlimmer/'slimmedMuons'
   [4] Prefetching for module PATMuonSelector/'selectedPatMuons'
   [5] Prefetching for module PATMuonProducer/'patMuons'
   [6] Prefetching for module MuonProducer/'muons'
   [7] Prefetching for module PFProducer/'particleFlowTmp'
   [8] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [9] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [10] Prefetching for module PFConversionProducer/'pfConversions'
   [11] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

140.56

----- Begin Fatal Exception 24-Feb-2022 00:48:36 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 326479 lumi: 7 event: 1579493 stream: 0
   [1] Running path 'dqmoffline_8_step'
   [2] Prefetching for module SMPDQM/'SMPDQM'
   [3] Prefetching for module MuonProducer/'muons'
   [4] Prefetching for module PFProducer/'particleFlowTmp'
   [5] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [6] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [7] Prefetching for module PFConversionProducer/'pfConversions'
   [8] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

136.874

----- Begin Fatal Exception 24-Feb-2022 01:04:07 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 319450 lumi: 76 event: 106007323 stream: 0
   [1] Running path 'dqmoffline_10_step'
   [2] Prefetching for module SMPDQM/'SMPDQM'
   [3] Prefetching for module MuonProducer/'muons'
   [4] Prefetching for module PFProducer/'particleFlowTmp'
   [5] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [6] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [7] Prefetching for module PFConversionProducer/'pfConversions'
   [8] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

AddOn Tests

fastsim

----- Begin Fatal Exception 23-Feb-2022 23:12:09 CET-----------------------
An exception of category 'Vertex' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 65 stream: 0
   [1] Running path 'prevalidation_step'
   [2] Prefetching for module MultiTrackValidator/'trackValidator'
   [3] Prefetching for module JetTracksAssociationToTrackRefs/'cutsRecoTracksAK4PFJets'
   [4] Prefetching for module JetTracksAssociatorExplicit/'ak4JetTracksAssociatorExplicitAll'
   [5] Prefetching for module FastjetJetProducer/'ak4PFJets'
   [6] Prefetching for module PFLinker/'particleFlow'
   [7] Prefetching for module PFProducer/'particleFlowTmp'
   [8] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [9] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [10] Prefetching for module PFConversionProducer/'pfConversions'
   [11] Calling method for module ConversionProducer/'allConversions'
Exception Message:
Refitted track not found in list
----- End Fatal Exception -------------------------------------------------

smuzaffar · 2022-02-24T09:32:06Z

+externals

cmsbuild · 2022-02-24T09:32:26Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_12_3_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

cmsbuild added externals-pending orp-pending pending-signatures tests-started labels Feb 22, 2022

cmsbuild added the bug-fix label Feb 22, 2022

cmsbuild added tests-approved and removed tests-started labels Feb 23, 2022

smuzaffar merged commit 6d480bb into cms-sw:IB/CMSSW_12_3_X/master Feb 24, 2022

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Feb 24, 2022

This was referenced Feb 24, 2022

XRootD: Update to version 5.4.0 and build with scitokens-cpp #7594

Merged

New dasgoclient version #7646

Merged

fwyzard deleted the IB/CMSSW_12_3_X/master_cuda_11.4.4 branch April 1, 2022 11:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to CUDA 11.4.4 #7641

Update to CUDA 11.4.4 #7641

fwyzard commented Feb 22, 2022

fwyzard commented Feb 22, 2022

fwyzard commented Feb 22, 2022

cmsbuild commented Feb 22, 2022

fwyzard commented Feb 22, 2022

fwyzard commented Feb 22, 2022

smuzaffar commented Feb 22, 2022 •

edited

cmsbuild commented Feb 23, 2022

smuzaffar commented Feb 23, 2022

smuzaffar commented Feb 23, 2022

cmsbuild commented Feb 23, 2022

cmsbuild commented Feb 23, 2022

cmsbuild commented Feb 24, 2022

smuzaffar commented Feb 24, 2022

cmsbuild commented Feb 24, 2022

Update to CUDA 11.4.4 #7641

Update to CUDA 11.4.4 #7641

Conversation

fwyzard commented Feb 22, 2022

fwyzard commented Feb 22, 2022

fwyzard commented Feb 22, 2022

cmsbuild commented Feb 22, 2022

fwyzard commented Feb 22, 2022

fwyzard commented Feb 22, 2022

smuzaffar commented Feb 22, 2022 • edited

cmsbuild commented Feb 23, 2022

GPU Comparison Summary

Comparison Summary

smuzaffar commented Feb 23, 2022

smuzaffar commented Feb 23, 2022

cmsbuild commented Feb 23, 2022

Unit Tests

cmsbuild commented Feb 23, 2022

External Build

cmsbuild commented Feb 24, 2022

Unit Tests

RelVals

AddOn Tests

smuzaffar commented Feb 24, 2022

cmsbuild commented Feb 24, 2022

smuzaffar commented Feb 22, 2022 •

edited