Revert to CUDA 10.1 Update 1 (10.1.168), add support for Power #5418

fwyzard · 2019-12-12T07:45:56Z

CUDA 10.1 Update 2 and later show a problem with running under MPS (NVIDIA Multi Process Server) or using CUDA Dynamic Parallelism within CMSSW.
While awaiting feedback from NVIDIA, the only solution seems to revert to the latest working version, which was 10.1 Update 1.

Drop the Nsight Compute and Nsigh System tools from the CUDA package, because they are released much more often as external packages.

Add support for CUDA on IBM Power architecture (ppc64le) on Linux.

fwyzard · 2019-12-12T07:46:07Z

please test

cmsbuild · 2019-12-12T07:46:21Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/3933/console Started: 2019/12/12 08:47

cmsbuild · 2019-12-12T07:46:22Z

A new Pull Request was created by @fwyzard (Andrea Bocci) for branch IB/CMSSW_11_1_X/master.

@cmsbuild, @smuzaffar, @mrodozov can you please review it and eventually sign? Thanks.
cms-bot commands are listed here

cmsbuild · 2019-12-12T07:55:10Z

Pull request #5418 was updated.

fwyzard · 2019-12-12T08:16:40Z

please test

fwyzard · 2019-12-12T08:17:04Z

@mrodozov @smuzaffar how do I ask to run the tests on Power ?

cmsbuild · 2019-12-12T08:17:43Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/3934/console Started: 2019/12/12 09:21

cmsbuild · 2019-12-12T08:17:45Z

Pull request #5418 was updated.

fwyzard · 2019-12-12T08:42:22Z

please test for slc7_ppc64le_gcc820

fwyzard · 2019-12-12T08:42:30Z

please test for slc7_aarch64_gcc820

cmsbuild · 2019-12-12T08:42:48Z

The tests are being triggered in jenkins.
Test Parameters:

ARCHITECTURE_FILTER = slc7_aarch64_gcc820
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/3937/console Started: 2019/12/12 09:46
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/3938/console Started: 2019/12/12 09:46
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/3940/console Started: 2019/12/12 10:39
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/3943/console Started: 2019/12/12 11:48

fwyzard · 2019-12-12T11:05:43Z

The CUDA part on Power seems to have completed fine ?

cmsbuild · 2019-12-12T11:48:48Z

-1

Tested at: c559c69

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-58adba/3937/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests

Unit Tests:

I found errors in the following unit tests:

---> test test_pycurl had ERRORS

cmsbuild · 2019-12-12T11:48:55Z

Comparison job queued.

cmsbuild · 2019-12-12T12:04:05Z

-1

Tested at: c559c69

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-58adba/3943/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests RelVals AddOn

Unit Tests:

I found errors in the following unit tests:

---> test test_calo_rechit had ERRORS

RelVals:

When I ran the RelVals I found an error in the following workflows:
136.7611 step2

runTheMatrix-results/136.7611_RunJetHT2016E_reminiaod+RunJetHT2016E_reminiaod+REMINIAOD_data2016_HIPM+HARVESTDR2_REMINIAOD_data2016_HIPM/step2_RunJetHT2016E_reminiaod+RunJetHT2016E_reminiaod+REMINIAOD_data2016_HIPM+HARVESTDR2_REMINIAOD_data2016_HIPM.log

136.8311 step2

runTheMatrix-results/136.8311_RunJetHT2017F_reminiaod+RunJetHT2017F_reminiaod+REMINIAOD_data2017+HARVEST2017_REMINIAOD_data2017/step2_RunJetHT2017F_reminiaod+RunJetHT2017F_reminiaod+REMINIAOD_data2017+HARVEST2017_REMINIAOD_data2017.log

1325.7 step2

runTheMatrix-results/1325.7_TTbar_13_94Xv2NanoAODINPUT+TTbar_13_94Xv2NanoAODINPUT+NANOEDMMC2017_94XMiniAODv2+HARVESTNANOAODMC2017_94XMiniAODv2/step2_TTbar_13_94Xv2NanoAODINPUT+TTbar_13_94Xv2NanoAODINPUT+NANOEDMMC2017_94XMiniAODv2+HARVESTNANOAODMC2017_94XMiniAODv2.log

5.1 step1

runTheMatrix-results/5.1_TTbar+TTbarFS+HARVESTFS/step1_TTbar+TTbarFS+HARVESTFS.log

135.4 step1

runTheMatrix-results/135.4_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS/step1_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS.log

1000.0 step2

runTheMatrix-results/1000.0_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT/step2_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT.log

101.0 step1

runTheMatrix-results/101.0_SingleElectronE120EHCAL+SingleElectronE120EHCAL/step1_SingleElectronE120EHCAL+SingleElectronE120EHCAL.log

1001.0 step2

runTheMatrix-results/1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVDSIPIXELCALRUN1+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5/step2_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVDSIPIXELCALRUN1+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5.log

140.56 step2

runTheMatrix-results/140.56_RunHI2018+RunHI2018+RECOHID18+HARVESTDHI18/step2_RunHI2018+RunHI2018+RECOHID18+HARVESTDHI18.log

4.53 step3

runTheMatrix-results/4.53_RunPhoton2012B+RunPhoton2012B+HLTD+RECODR1reHLT+HARVESTDR1reHLT/step3_RunPhoton2012B+RunPhoton2012B+HLTD+RECODR1reHLT+HARVESTDR1reHLT.log

136.85 step3

runTheMatrix-results/136.85_RunEGamma2018A+RunEGamma2018A+HLTDR2_2018+RECODR2_2018reHLT_skimEGamma_Offline_L1TEgDQM+HARVEST2018_L1TEgDQM/step3_RunEGamma2018A+RunEGamma2018A+HLTDR2_2018+RECODR2_2018reHLT_skimEGamma_Offline_L1TEgDQM+HARVEST2018_L1TEgDQM.log

136.731 step3

runTheMatrix-results/136.731_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2/step3_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2.log

136.788 step3

runTheMatrix-results/136.788_RunSinglePh2017B+RunSinglePh2017B+HLTDR2_2017+RECODR2_2017reHLT_skimSinglePh_Prompt+HARVEST2017/step3_RunSinglePh2017B+RunSinglePh2017B+HLTDR2_2017+RECODR2_2017reHLT_skimSinglePh_Prompt+HARVEST2017.log

1306.0 step3

runTheMatrix-results/1306.0_SingleMuPt1_UP15+SingleMuPt1_UP15+DIGIUP15+RECOUP15+HARVESTUP15/step3_SingleMuPt1_UP15+SingleMuPt1_UP15+DIGIUP15+RECOUP15+HARVESTUP15.log

1330.0 step3

runTheMatrix-results/1330.0_ZMM_13+ZMM_13+DIGIUP15+RECOUP15_L1TMuDQM+HARVESTUP15_L1TMuDQM+NANOUP15/step3_ZMM_13+ZMM_13+DIGIUP15+RECOUP15_L1TMuDQM+HARVESTUP15_L1TMuDQM+NANOUP15.log

4.22 step5

runTheMatrix-results/4.22_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC/step5_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC.log

8.0 step5

runTheMatrix-results/8.0_BeamHalo+BeamHalo+DIGICOS+RECOCOS+ALCABH+HARVESTCOS/step5_BeamHalo+BeamHalo+DIGICOS+RECOCOS+ALCABH+HARVESTCOS.log

10042.0 step3

runTheMatrix-results/10042.0_ZMM_13+ZMM_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017+ALCAFull_2017+NanoFull_2017/step3_ZMM_13+ZMM_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017+ALCAFull_2017+NanoFull_2017.log

7.3 step5

runTheMatrix-results/7.3_CosmicsSPLoose_UP18+CosmicsSPLoose_UP18+DIGICOS_UP18+RECOCOS_UP18+ALCACOS_UP18+HARVESTCOS_UP18/step5_CosmicsSPLoose_UP18+CosmicsSPLoose_UP18+DIGICOS_UP18+RECOCOS_UP18+ALCACOS_UP18+HARVESTCOS_UP18.log

9.0 step3

runTheMatrix-results/9.0_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST/step3_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST.log

25.0 step3

runTheMatrix-results/25.0_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT/step3_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT.log

158.0 step3

runTheMatrix-results/158.0_HydjetQ_B12_5020GeV_2018_ppReco+HydjetQ_B12_5020GeV_2018_ppReco+DIGIHI2018PPRECO+RECOHI2018PPRECO+ALCARECOHI2018PPRECO+HARVESTHI2018PPRECO/step3_HydjetQ_B12_5020GeV_2018_ppReco+HydjetQ_B12_5020GeV_2018_ppReco+DIGIHI2018PPRECO+RECOHI2018PPRECO+ALCARECOHI2018PPRECO+HARVESTHI2018PPRECO.log

140.53 step3

runTheMatrix-results/140.53_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI/step3_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI.log

11634.0 step3

runTheMatrix-results/11634.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2021_GenSimFull+DigiFull_2021+RecoFull_2021+HARVESTFull_2021+ALCAFull_2021/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2021_GenSimFull+DigiFull_2021+RecoFull_2021+HARVESTFull_2021+ALCAFull_2021.log

10024.0 step3

runTheMatrix-results/10024.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017+ALCAFull_2017+NanoFull_2017/step3_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017+ALCAFull_2017+NanoFull_2017.log

10824.0 step3

runTheMatrix-results/10824.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2018_GenSimFull+DigiFull_2018+RecoFull_2018+HARVESTFull_2018+ALCAFull_2018+NanoFull_2018/step3_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2018_GenSimFull+DigiFull_2018+RecoFull_2018+HARVESTFull_2018+ALCAFull_2018+NanoFull_2018.log

12434.0 step3

runTheMatrix-results/12434.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023_GenSimFull+DigiFull_2023+RecoFull_2023+HARVESTFull_2023+ALCAFull_2023/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023_GenSimFull+DigiFull_2023+RecoFull_2023+HARVESTFull_2023+ALCAFull_2023.log

25202.0 step3

runTheMatrix-results/25202.0_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25+NANOUP15_PU25/step3_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25+NANOUP15_PU25.log

20034.0 step3

runTheMatrix-results/20034.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D35_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D35+RecoFullGlobal_2026D35+HARVESTFullGlobal_2026D35/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D35_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D35+RecoFullGlobal_2026D35+HARVESTFullGlobal_2026D35.log

10224.0 step3

runTheMatrix-results/10224.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017PU_GenSimFull+DigiFullPU_2017PU+RecoFullPU_2017PU+HARVESTFullPU_2017PU+NanoFull_2017PU/step3_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017PU_GenSimFull+DigiFullPU_2017PU+RecoFullPU_2017PU+HARVESTFullPU_2017PU+NanoFull_2017PU.log

20434.0 step3

runTheMatrix-results/20434.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D41_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D41+RecoFullGlobal_2026D41+HARVESTFullGlobal_2026D41/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D41_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D41+RecoFullGlobal_2026D41+HARVESTFullGlobal_2026D41.log

23234.0 step3

runTheMatrix-results/23234.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D49_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D49+RecoFullGlobal_2026D49+HARVESTFullGlobal_2026D49/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D49_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D49+RecoFullGlobal_2026D49+HARVESTFullGlobal_2026D49.log

21234.0 step3

runTheMatrix-results/21234.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D44_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D44+RecoFullGlobal_2026D44+HARVESTFullGlobal_2026D44/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D44_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D44+RecoFullGlobal_2026D44+HARVESTFullGlobal_2026D44.log

250202.181 step4

runTheMatrix-results/250202.181_TTbar_13UP18+TTbar_13UP18+PREMIXUP18_PU25+DIGIPRMXLOCALUP18_PU25+RECOPRMXUP18_PU25+HARVESTUP18_PU25/step4_TTbar_13UP18+TTbar_13UP18+PREMIXUP18_PU25+DIGIPRMXLOCALUP18_PU25+RECOPRMXUP18_PU25+HARVESTUP18_PU25.log

AddOn:

I found errors in the following addon tests:

cmsDriver.py TTbar_8TeV_TuneCUETP8M1_cfi --conditions auto:run1_mc --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot Realistic8TeVCollision : FAILED - time: date Thu Dec 12 12:52:39 2019-date Thu Dec 12 12:52:30 2019 s - exit: 256
cmsRun /cvmfs/cms-ib.cern.ch/week0/slc7_ppc64le_gcc820/cms/cmssw/CMSSW_11_1_X_2019-12-11-2300/src/PhysicsTools/PatAlgos/test/IntegrationTest_cfg.py : FAILED - time: date Thu Dec 12 12:52:36 2019-date Thu Dec 12 12:52:34 2019 s - exit: 23040
cmsDriver.py RelVal -s HLT:Fake2,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_Fake2 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016 --processName=HLTRECO --filein file:RelVal_Raw_Fake2_DATA.root --fileout file:RelVal_Raw_Fake2_DATA_HLT_RECO.root : FAILED - time: date Thu Dec 12 12:53:58 2019-date Thu Dec 12 12:52:36 2019 s - exit: 256
cmsDriver.py RelVal -s HLT:Fake1,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_Fake1 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_25ns --processName=HLTRECO --filein file:RelVal_Raw_Fake1_MC.root --fileout file:RelVal_Raw_Fake1_MC_HLT_RECO.root : FAILED - time: date Thu Dec 12 12:57:27 2019-date Thu Dec 12 12:52:41 2019 s - exit: 256
cmsDriver.py RelVal -s HLT:2018,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_2018 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018 --processName=HLTRECO --filein file:RelVal_Raw_2018_MC.root --fileout file:RelVal_Raw_2018_MC_HLT_RECO.root : FAILED - time: date Thu Dec 12 13:01:01 2019-date Thu Dec 12 12:52:42 2019 s - exit: 256
cmsDriver.py RelVal -s HLT:HIon,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_HIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018_pp_on_AA --processName=HLTRECO --filein file:RelVal_Raw_HIon_DATA.root --fileout file:RelVal_Raw_HIon_DATA_HLT_RECO.root : FAILED - time: date Thu Dec 12 12:56:25 2019-date Thu Dec 12 12:52:43 2019 s - exit: 256
cmsDriver.py RelVal -s HLT:GRun,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_GRun --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run3 --processName=HLTRECO --filein file:RelVal_Raw_GRun_MC.root --fileout file:RelVal_Raw_GRun_MC_HLT_RECO.root : FAILED - time: date Thu Dec 12 13:00:15 2019-date Thu Dec 12 12:52:48 2019 s - exit: 256
cmsDriver.py TTbar_13TeV_TuneCUETP8M1_cfi --conditions auto:run2_mc_l1stage1 --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot NominalCollision2015 --era Run2_25ns : FAILED - time: date Thu Dec 12 12:52:58 2019-date Thu Dec 12 12:52:50 2019 s - exit: 256
cmsDriver.py RelVal -s HLT:2018,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_2018 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018 --processName=HLTRECO --filein file:RelVal_Raw_2018_DATA.root --fileout file:RelVal_Raw_2018_DATA_HLT_RECO.root : FAILED - time: date Thu Dec 12 12:58:03 2019-date Thu Dec 12 12:52:53 2019 s - exit: 256
cmsDriver.py RelVal -s HLT:PRef,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_PRef --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run3 --processName=HLTRECO --filein file:RelVal_Raw_PRef_MC.root --fileout file:RelVal_Raw_PRef_MC_HLT_RECO.root : FAILED - time: date Thu Dec 12 12:58:04 2019-date Thu Dec 12 12:52:55 2019 s - exit: 256
cmsDriver.py TTbar_13TeV_TuneCUETP8M1_cfi --conditions auto:run2_mc --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot NominalCollision2015 --era Run2_2016 : FAILED - time: date Thu Dec 12 12:53:07 2019-date Thu Dec 12 12:52:59 2019 s - exit: 256
cmsDriver.py RelVal -s HLT:PIon,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_PIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018 --processName=HLTRECO --filein file:RelVal_Raw_PIon_DATA.root --fileout file:RelVal_Raw_PIon_DATA_HLT_RECO.root : FAILED - time: date Thu Dec 12 12:54:42 2019-date Thu Dec 12 12:53:00 2019 s - exit: 256
cmsDriver.py RelVal -s HLT:Fake,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run1_mc_Fake --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --processName=HLTRECO --filein file:RelVal_Raw_Fake_MC.root --fileout file:RelVal_Raw_Fake_MC_HLT_RECO.root : FAILED - time: date Thu Dec 12 12:57:17 2019-date Thu Dec 12 12:53:02 2019 s - exit: 256
cmsDriver.py RelVal -s HLT:HIon,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_HIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018_pp_on_AA --processName=HLTRECO --filein file:RelVal_Raw_HIon_MC.root --fileout file:RelVal_Raw_HIon_MC_HLT_RECO.root : FAILED - time: date Thu Dec 12 12:59:20 2019-date Thu Dec 12 12:53:05 2019 s - exit: 256
cmsDriver.py RelVal -s HLT:Fake2,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_Fake2 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016 --processName=HLTRECO --filein file:RelVal_Raw_Fake2_MC.root --fileout file:RelVal_Raw_Fake2_MC_HLT_RECO.root : FAILED - time: date Thu Dec 12 12:57:52 2019-date Thu Dec 12 12:53:07 2019 s - exit: 256
cmsDriver.py RelVal -s HLT:PRef,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_PRef --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018 --processName=HLTRECO --filein file:RelVal_Raw_PRef_DATA.root --fileout file:RelVal_Raw_PRef_DATA_HLT_RECO.root : FAILED - time: date Thu Dec 12 12:55:17 2019-date Thu Dec 12 12:53:09 2019 s - exit: 256
cmsDriver.py RelVal -s HLT:Fake,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run1_data_Fake --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --processName=HLTRECO --filein file:RelVal_Raw_Fake_DATA.root --fileout file:RelVal_Raw_Fake_DATA_HLT_RECO.root : FAILED - time: date Thu Dec 12 12:54:24 2019-date Thu Dec 12 12:53:14 2019 s - exit: 256
cmsDriver.py RelVal -s HLT:GRun,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_GRun --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018 --processName=HLTRECO --filein file:RelVal_Raw_GRun_DATA.root --fileout file:RelVal_Raw_GRun_DATA_HLT_RECO.root : FAILED - time: date Thu Dec 12 12:58:23 2019-date Thu Dec 12 12:53:19 2019 s - exit: 256
cmsDriver.py RelVal -s HLT:Fake1,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_Fake1 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_25ns --processName=HLTRECO --filein file:RelVal_Raw_Fake1_DATA.root --fileout file:RelVal_Raw_Fake1_DATA_HLT_RECO.root : FAILED - time: date Thu Dec 12 12:54:36 2019-date Thu Dec 12 12:53:24 2019 s - exit: 256
cmsDriver.py RelVal -s HLT:PIon,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_PIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run3 --processName=HLTRECO --filein file:RelVal_Raw_PIon_MC.root --fileout file:RelVal_Raw_PIon_MC_HLT_RECO.root : FAILED - time: date Thu Dec 12 12:58:00 2019-date Thu Dec 12 12:53:25 2019 s - exit: 256

cmsbuild · 2019-12-12T12:04:09Z

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

smuzaffar · 2019-12-12T12:33:44Z

Yes, cuda part of cmssw build fine for ppc64le

cmsbuild · 2019-12-12T13:36:24Z

-1

Tested at: c559c69

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-58adba/3940/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests RelVals

Unit Tests:

I found errors in the following unit tests:

---> test import-tables had ERRORS
---> test testTables had ERRORS

RelVals:

cmsbuild · 2019-12-12T13:36:26Z

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

cmsbuild · 2019-12-12T13:47:51Z

+1
Tested at: c559c69
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-58adba/3934/summary.html

cmsbuild · 2019-12-12T13:47:55Z

Comparison job queued.

cmsbuild · 2019-12-12T15:15:35Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-58adba/3934/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2800084
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2799742
DQMHistoTests: Total skipped: 341
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
Checked 147 log files, 16 edm output root files, 34 DQM output files

smuzaffar · 2019-12-12T15:55:12Z

+externals
@fwyzard , so cuda builds for all archs (x86_64, aarch64, ppc64le) but cuda gcc combination only works for x86_64 and ppc64le. aarch64 cuda version 10.0 does not support gcc8.

cmsbuild · 2019-12-12T15:55:40Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_11_1_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

fwyzard · 2019-12-12T16:00:12Z

aarch64 cuda version 10.0 does not support gcc8.

I know... but there is no newer version officially available, yet :-(

Remove Nsight Compute and Systems from the CUDA packages

eed2e4a

cmsbuild added comparison-pending externals-pending orp-pending pending-signatures tests-pending labels Dec 12, 2019

cmsbuild added tests-started and removed tests-pending labels Dec 12, 2019

fwyzard force-pushed the revert_cuda_10.1.168 branch from 5b43c86 to 0d7b06c Compare December 12, 2019 07:54

cmsbuild added tests-pending and removed tests-started labels Dec 12, 2019

fwyzard added 2 commits December 12, 2019 09:15

Revert to CUDA 10.1 Update 1 (10.1.168) on x86_64

45d7f99

Add support for CUDA on IBM Power architecture (ppc64le)

c559c69

fwyzard force-pushed the revert_cuda_10.1.168 branch from 0d7b06c to c559c69 Compare December 12, 2019 08:16

cmsbuild added tests-started and removed tests-pending labels Dec 12, 2019

fwyzard changed the title ~~Revert to CUDA 10.1 Update 1 (10.1.168) on x86_64~~ Revert to CUDA 10.1 Update 1 (10.1.168), add support for Power Dec 12, 2019

fwyzard mentioned this pull request Dec 12, 2019

Revert to CUDA 10.1 Update 1 (10.1.168), add support for Power (11.0.x backport) #5419

Merged

cmsbuild removed the tests-started label Dec 12, 2019

cmsbuild added comparison-pending and removed comparison-notrun labels Dec 12, 2019

cmsbuild added comparison-notrun and removed comparison-pending labels Dec 12, 2019

cmsbuild added comparison-pending tests-approved and removed comparison-notrun tests-rejected labels Dec 12, 2019

cmsbuild added comparison-available and removed comparison-pending labels Dec 12, 2019

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Dec 12, 2019

smuzaffar merged commit 5a803be into cms-sw:IB/CMSSW_11_1_X/master Dec 12, 2019

fwyzard deleted the revert_cuda_10.1.168 branch April 16, 2020 13:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revert to CUDA 10.1 Update 1 (10.1.168), add support for Power #5418

Revert to CUDA 10.1 Update 1 (10.1.168), add support for Power #5418

fwyzard commented Dec 12, 2019 •

edited

fwyzard commented Dec 12, 2019

cmsbuild commented Dec 12, 2019 •

edited

cmsbuild commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

fwyzard commented Dec 12, 2019

fwyzard commented Dec 12, 2019

cmsbuild commented Dec 12, 2019 •

edited

cmsbuild commented Dec 12, 2019

fwyzard commented Dec 12, 2019

fwyzard commented Dec 12, 2019

cmsbuild commented Dec 12, 2019 •

edited

fwyzard commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

smuzaffar commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

smuzaffar commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

fwyzard commented Dec 12, 2019

Revert to CUDA 10.1 Update 1 (10.1.168), add support for Power #5418

Revert to CUDA 10.1 Update 1 (10.1.168), add support for Power #5418

Conversation

fwyzard commented Dec 12, 2019 • edited

fwyzard commented Dec 12, 2019

cmsbuild commented Dec 12, 2019 • edited

cmsbuild commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

fwyzard commented Dec 12, 2019

fwyzard commented Dec 12, 2019

cmsbuild commented Dec 12, 2019 • edited

cmsbuild commented Dec 12, 2019

fwyzard commented Dec 12, 2019

fwyzard commented Dec 12, 2019

cmsbuild commented Dec 12, 2019 • edited

fwyzard commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

smuzaffar commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

smuzaffar commented Dec 12, 2019

cmsbuild commented Dec 12, 2019

fwyzard commented Dec 12, 2019

fwyzard commented Dec 12, 2019 •

edited

cmsbuild commented Dec 12, 2019 •

edited

cmsbuild commented Dec 12, 2019 •

edited

cmsbuild commented Dec 12, 2019 •

edited