raise Exception only if there are two workflows having the same name #33732

silviodonato · 2021-05-14T10:35:21Z

Premise

When two workflows having the same name are found, runTheMatrix.py gives the warning:

==> duplicate name found for  36634.0_TTbar_14TeV+2026D81+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal
    keeping  :  (36634.0, 'TTbar_14TeV+2026D81+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal', ['cmsDriver.py TTbar_14TeV_TuneCP5_cfi  --conditions auto:phase2_realistic_T26 -n 10 --era Phase2C11I13T26M9 --eventcontent FEVTDEBUG --relval 9000,100 -s GEN,SIM --datatier GEN-SIM --beamspot HLLHC14TeV --geometry Extended2026D81', 'cmsDriver.py step2  --conditions auto:phase2_realistic_T26 -s DIGI:pdigi_valid,L1TrackTrigger,L1,DIGI2RAW,HLT:@fake2 --datatier GEN-SIM-DIGI-RAW -n 10 --geometry Extended2026D81 --era Phase2C11I13T26M9 --eventcontent FEVTDEBUGHLT', 'cmsDriver.py step3  --conditions auto:phase2_realistic_T26 -s RAW2DIGI,L1Reco,RECO,RECOSIM,PAT,VALIDATION:@phase2Validation+@miniAODValidation,DQM:@phase2+@miniAODDQM --datatier GEN-SIM-RECO,MINIAODSIM,DQMIO -n 10 --geometry Extended2026D81 --era Phase2C11I13T26M9 --eventcontent FEVTDEBUGHLT,MINIAODSIM,DQM', 'cmsDriver.py step4  --conditions auto:phase2_realistic_T26 -s HARVESTING:@phase2Validation+@phase2+@miniAODValidation+@miniAODDQM --scenario pp --filetype DQM --geometry Extended2026D81 --era Phase2C11I13T26M9 --mc '], ['TTbar_14TeV_TuneCP5_2026D81_GenSimHLBeamSpot14', 'DigiTrigger_2026D81', 'RecoGlobal_2026D81', 'HARVESTGlobal_2026D81'])
    ignoring :  (36634.0, 'TTbar_14TeV+2026D81+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal', ['cmsDriver.py TTbar_14TeV_TuneCP5_cfi  --conditions auto:phase2_realistic_T26 -n 10 --era Phase2C11I13T26M9 --eventcontent FEVTDEBUG --relval 9000,100 -s GEN,SIM --datatier GEN-SIM --beamspot HLLHC14TeV --geometry Extended2026D81', 'cmsDriver.py step2  --conditions auto:phase2_realistic_T26 -s DIGI:pdigi_valid,L1TrackTrigger,L1,DIGI2RAW,HLT:@fake2 --datatier GEN-SIM-DIGI-RAW -n 10 --geometry Extended2026D81 --era Phase2C11I13T26M9 --eventcontent FEVTDEBUGHLT', 'cmsDriver.py step3  --conditions auto:phase2_realistic_T26 -s RAW2DIGI,L1Reco,RECO,RECOSIM,PAT,VALIDATION:@phase2Validation+@miniAODValidation,DQM:@phase2+@miniAODDQM --datatier GEN-SIM-RECO,MINIAODSIM,DQMIO -n 10 --geometry Extended2026D81 --era Phase2C11I13T26M9 --eventcontent FEVTDEBUGHLT,MINIAODSIM,DQM', 'cmsDriver.py step4  --conditions auto:phase2_realistic_T26 -s HARVESTING:@phase2Validation+@phase2+@miniAODValidation+@miniAODDQM --scenario pp --filetype DQM --geometry Extended2026D81 --era Phase2C11I13T26M9 --mc '], ['TTbar_14TeV_TuneCP5_2026D81_GenSimHLBeamSpot14', 'DigiTrigger_2026D81', 'RecoGlobal_2026D81',

This happens if you try to run on runTheMatrix.py --showMatrix --what upgrade,2026, because 2026 is a subset of upgrade.
The possibility to run on multiple matrices was added by #33338. This PR added also a protection to prevent you from running on a workflow number that is used in multiple workflows.

PR description:

This PR allows you to run on workflow numbers (eg. 23461.97) which are defined multiple times if and only if the duplicated workflows have the same name (eg. 23461.97_PREMIXUP26D49_PU25+2026D49PU_PMXS1+PREMIX_PremixHLBeamSpotPU)

PR validation:

Previous behavior (ERROR if there are wfs defined multiple times)

$ runTheMatrix.py --showMatrix --what upgrade,2026 -l 23461.97
==> duplicate name found for  23461.97_PREMIXUP26D49_PU25+2026D49PU_PMXS1+PREMIX_PremixHLBeamSpotPU
    keeping  :  (23461.97, 'PREMIXUP26D49_PU25+2026D49PU_PMXS1+PREMIX_PremixHLBeamSpotPU', ['cmsDriver.py step1  --conditions auto:phase2_realistic_T15 --pileup_input das:/RelValMinBias_14TeV/CMSSW_11_3_0_pre4-113X_mcRun4_realistic_v4_2026D49noPU_rsb-v1/GEN-SIM -n 10 --era Phase2C9 --eventcontent PREMIX --procModifiers premix_stage1 --relval 100000,100 -s GEN,SIM,DIGI:pdigi_valid --pileup AVE_200_BX_25ns --datatier PREMIX --beamspot HLLHC --geometry Extended2026D49 --evt_type SingleNuE10_cfi'], ['PREMIX_2026D49PU_PremixHLBeamSpotPU_PMXS1'])
    ignoring :  (23461.97, 'PREMIXUP26D49_PU25+2026D49PU_PMXS1+PREMIX_PremixHLBeamSpotPU', ['cmsDriver.py step1  --conditions auto:phase2_realistic_T15 --pileup_input das:/RelValMinBias_14TeV/CMSSW_11_3_0_pre4-113X_mcRun4_realistic_v4_2026D49noPU_rsb-v1/GEN-SIM -n 10 --era Phase2C9 --eventcontent PREMIX --procModifiers premix_stage1 --relval 100000,100 -s GEN,SIM,DIGI:pdigi_valid --pileup AVE_200_BX_25ns --datatier PREMIX --beamspot HLLHC --geometry Extended2026D49 --evt_type SingleNuE10_cfi'], ['PREMIX_2026D49PU_PremixHLBeamSpotPU_PMXS1'])
    
[...]

Traceback (most recent call last):
  File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_0_pre1/bin/slc7_amd64_gcc900/runTheMatrix.py", line 544, in <module>
    ret = runSelected(opt)
  File "/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_0_pre1/bin/slc7_amd64_gcc900/runTheMatrix.py", line 33, in runSelected
    if len(duplicates)>0: raise ValueError('Duplicated workflows: '+', '.join(map(str,list(duplicates))))
ValueError: Duplicated workflows: 23461.97

New behavior (WARNING if there are wfs defined multiple times having the same wf name)

$ runTheMatrix.py --showMatrix --what upgrade,2026 -l 23461.97
==> duplicate name found for  23461.97_PREMIXUP26D49_PU25+2026D49PU_PMXS1+PREMIX_PremixHLBeamSpotPU
    keeping  :  (23461.97, 'PREMIXUP26D49_PU25+2026D49PU_PMXS1+PREMIX_PremixHLBeamSpotPU', ['cmsDriver.py step1  --conditions auto:phase2_realistic_T15 --pileup_input das:/RelValMinBias_14TeV/CMSSW_11_3_0_pre4-113X_mcRun4_realistic_v4_2026D49noPU_rsb-v1/GEN-SIM -n 10 --era Phase2C9 --eventcontent PREMIX --procModifiers premix_stage1 --relval 100000,100 -s GEN,SIM,DIGI:pdigi_valid --pileup AVE_200_BX_25ns --datatier PREMIX --beamspot HLLHC --geometry Extended2026D49 --evt_type SingleNuE10_cfi'], ['PREMIX_2026D49PU_PremixHLBeamSpotPU_PMXS1'])
    ignoring :  (23461.97, 'PREMIXUP26D49_PU25+2026D49PU_PMXS1+PREMIX_PremixHLBeamSpotPU', ['cmsDriver.py step1  --conditions auto:phase2_realistic_T15 --pileup_input das:/RelValMinBias_14TeV/CMSSW_11_3_0_pre4-113X_mcRun4_realistic_v4_2026D49noPU_rsb-v1/GEN-SIM -n 10 --era Phase2C9 --eventcontent PREMIX --procModifiers premix_stage1 --relval 100000,100 -s GEN,SIM,DIGI:pdigi_valid --pileup AVE_200_BX_25ns --datatier PREMIX --beamspot HLLHC --geometry Extended2026D49 --evt_type SingleNuE10_cfi'], ['PREMIX_2026D49PU_PremixHLBeamSpotPU_PMXS1'])
    
[...]

WARNING: Workflow 23461.97_PREMIXUP26D49_PU25+2026D49PU_PMXS1+PREMIX_PremixHLBeamSpotPU is defined multiple time.

New behavior (ERROR if there are wfs defined multiple times having different wf name)

[...]
Traceback (most recent call last):
  File "/afs/cern.ch/work/s/sdonato/ORP/CMSSW_12_0_X_2021-05-13-2300/bin/slc7_amd64_gcc900/runTheMatrix.py", line 551, in <module>
    ret = runSelected(opt)
  File "/afs/cern.ch/work/s/sdonato/ORP/CMSSW_12_0_X_2021-05-13-2300/bin/slc7_amd64_gcc900/runTheMatrix.py", line 38, in runSelected
    raise ValueError('There are multiple workflows matching the wf number %s: \n'%duplicate + '\n'.join(duplicate_wfs) + "\nPlease remove the duplicated workflow.")
ValueError: There are multiple workflows matching the wf number 23234.0: 
23234.0_TTbar_14TeV+2026D60+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal
23234.0_TTbar_14TeV+2026D49+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal
Please remove the duplicated workflow.

silviodonato · 2021-05-14T10:35:28Z

please test

cmsbuild · 2021-05-14T10:41:50Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-33732/22672

This PR adds an extra 16KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/scripts/runTheMatrix.py modified in PR(s): Add GPUParams to runTheMatrix workflows #33538

cmsbuild · 2021-05-14T10:42:14Z

A new Pull Request was created by @silviodonato (Silvio Donato) for master.

It involves the following packages:

Configuration/PyReleaseValidation

@chayanit, @srimanob, @wajidalikhan, @kpedro88, @jordan-martins can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @fabiocos, @slomeo this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2021-05-14T12:58:50Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c482cb/15092/summary.html
COMMIT: 3c03d7b
CMSSW: CMSSW_12_0_X_2021-05-13-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/33732/15092/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2648242
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2648219
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 36 files compared)
Checked 155 log files, 37 edm output root files, 37 DQM output files
TriggerResults: no differences found

silviodonato · 2021-05-18T09:36:14Z

test parameters:

relvals_options = --what standard,highstats,pileup,generator,production,ged,2017,2026,machine,premix,upgrade

silviodonato · 2021-05-18T09:37:43Z

please test workflow 37725.0

cmsbuild · 2021-05-18T10:54:25Z

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c482cb/15151/summary.html
COMMIT: 3c03d7b
CMSSW: CMSSW_12_0_X_2021-05-17-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/33732/15151/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

10024.010024.0_TTbar_13+2017+TTbar_13TeV_TuneCUETP8M1_GenSim+Digi+RecoFakeHLT+HARVESTFakeHLT+ALCA+Nano/step3_TTbar_13+2017+TTbar_13TeV_TuneCUETP8M1_GenSim+Digi+RecoFakeHLT+HARVESTFakeHLT+ALCA+Nano.log
10042.010042.0_ZMM_13+2017+ZMM_13TeV_TuneCUETP8M1_GenSim+Digi+RecoFakeHLT+HARVESTFakeHLT+ALCA+Nano/step3_ZMM_13+2017+ZMM_13TeV_TuneCUETP8M1_GenSim+Digi+RecoFakeHLT+HARVESTFakeHLT+ALCA+Nano.log
10224.010224.0_TTbar_13+2017PU+TTbar_13TeV_TuneCUETP8M1_GenSim+DigiPU+RecoFakeHLTPU+HARVESTFakeHLTPU+Nano/step2_TTbar_13+2017PU+TTbar_13TeV_TuneCUETP8M1_GenSim+DigiPU+RecoFakeHLTPU+HARVESTFakeHLTPU+Nano.log

Expand to see more relval errors ...

silviodonato · 2021-05-24T07:40:53Z

please test workflow 37725.0

cmsbuild · 2021-05-24T08:54:43Z

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c482cb/15267/summary.html
COMMIT: 3c03d7b
CMSSW: CMSSW_12_0_X_2021-05-23-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/33732/15267/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

10024.010024.0_TTbar_13+2017+TTbar_13TeV_TuneCUETP8M1_GenSim+Digi+RecoFakeHLT+HARVESTFakeHLT+ALCA+Nano/step3_TTbar_13+2017+TTbar_13TeV_TuneCUETP8M1_GenSim+Digi+RecoFakeHLT+HARVESTFakeHLT+ALCA+Nano.log
10042.010042.0_ZMM_13+2017+ZMM_13TeV_TuneCUETP8M1_GenSim+Digi+RecoFakeHLT+HARVESTFakeHLT+ALCA+Nano/step3_ZMM_13+2017+ZMM_13TeV_TuneCUETP8M1_GenSim+Digi+RecoFakeHLT+HARVESTFakeHLT+ALCA+Nano.log
10224.010224.0_TTbar_13+2017PU+TTbar_13TeV_TuneCUETP8M1_GenSim+DigiPU+RecoFakeHLTPU+HARVESTFakeHLTPU+Nano/step2_TTbar_13+2017PU+TTbar_13TeV_TuneCUETP8M1_GenSim+DigiPU+RecoFakeHLTPU+HARVESTFakeHLTPU+Nano.log

Expand to see more relval errors ...

chayanit · 2021-05-24T13:20:12Z

Hi @silviodonato , I'm not sure how we can have duplicate workflows with the same IDs?

silviodonato · 2021-05-24T15:32:29Z

@chayanit this is needed for running multiple matrices in the PR test. The most common case happens when somebody wants to extend the PR test to a workflow that is included upgrade matrix. In this case you have to add the upgrade matrix on top of the other matrices used standard PR tests (2026,standard,pileup, ...).
Since all the workflows in 2026 matrix (eg. 36634.0) are a subset of the workflows in the upgrade matrix, they will appear as duplicated because the workflows (eg. 36634.0) are both in upgrade and in 2026

silviodonato · 2021-05-24T17:21:45Z

move to a different solution #33823

chayanit · 2021-05-25T03:58:51Z

ah I see thanks @silviodonato

silviodonato added 2 commits May 14, 2021 11:51

raise Exception only if there are two workflows having the same name

ac1b7de

add comments

3c03d7b

cmsbuild added this to the CMSSW_12_0_X milestone May 14, 2021

cmsbuild added code-checks-pending orp-pending pdmv-pending pending-signatures tests-started upgrade-pending labels May 14, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels May 14, 2021

cmsbuild added tests-approved and removed tests-started labels May 14, 2021

cmsbuild added tests-started and removed tests-approved labels May 18, 2021

cmsbuild added tests-rejected and removed tests-started labels May 18, 2021

cmsbuild added tests-started and removed tests-rejected labels May 24, 2021

cmsbuild added tests-rejected and removed tests-started labels May 24, 2021

silviodonato mentioned this pull request May 24, 2021

[runTheMatrix.py] Adding cleanedupgrade matrix for PR tests #33823

Merged

silviodonato closed this May 24, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

raise Exception only if there are two workflows having the same name #33732

raise Exception only if there are two workflows having the same name #33732

silviodonato commented May 14, 2021

silviodonato commented May 14, 2021

cmsbuild commented May 14, 2021

cmsbuild commented May 14, 2021

cmsbuild commented May 14, 2021

silviodonato commented May 18, 2021

silviodonato commented May 18, 2021

cmsbuild commented May 18, 2021

silviodonato commented May 24, 2021

cmsbuild commented May 24, 2021

chayanit commented May 24, 2021

silviodonato commented May 24, 2021

silviodonato commented May 24, 2021

chayanit commented May 25, 2021

raise Exception only if there are two workflows having the same name #33732

raise Exception only if there are two workflows having the same name #33732

Conversation

silviodonato commented May 14, 2021

Premise

PR description:

PR validation:

silviodonato commented May 14, 2021

cmsbuild commented May 14, 2021

cmsbuild commented May 14, 2021

cmsbuild commented May 14, 2021

Comparison Summary

silviodonato commented May 18, 2021

silviodonato commented May 18, 2021

cmsbuild commented May 18, 2021

RelVals

silviodonato commented May 24, 2021

cmsbuild commented May 24, 2021

RelVals

chayanit commented May 24, 2021

silviodonato commented May 24, 2021

silviodonato commented May 24, 2021

chayanit commented May 25, 2021