Add a script to split a CMSSW configuration into individual cfi files #27305

fwyzard · 2019-06-21T14:23:32Z

PR description:

This PR adds the necessary hooks under FWCore/ParameterSet/python, and a script under FWCore/ParameterSet/scripts, to split a CMSSW python configuration into individual *_cfi.py files.

Missing features

SwitchProducer and Subprocess are not supported;
refToPSet, SequencePlaceholder and TaskPlaceholder are not handled specifically, but that is probably the correct approach.

PR validation:

As a validation, the latest HLT menu has been split into individual modules with edmConfigSplit, and combined back together with edmConfigDump; the resulting configuration was identical.

cmsbuild · 2019-06-21T14:23:59Z

The code-checks are being triggered in jenkins.

fwyzard · 2019-06-21T14:24:13Z

@Dr15Jones @makortel @wddgit comments are welcome

cmsbuild · 2019-06-21T14:29:33Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-27305/10531

This PR adds an extra 88KB to repository

cmsbuild · 2019-06-21T14:30:01Z

A new Pull Request was created by @fwyzard (Andrea Bocci) for master.

It involves the following packages:

FWCore/ParameterSet

@cmsbuild, @smuzaffar, @Dr15Jones can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @wddgit this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

fabiocos · 2019-06-22T08:58:43Z

please test

cmsbuild · 2019-06-22T08:59:14Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/1107/console Started: 2019/06/22 10:59

Dr15Jones · 2019-06-22T11:12:50Z

@fwyzard my first comment would be to please add tests to the bottom of Config.py. That way you'd also be sure Tasks function :) .

Dr15Jones · 2019-06-22T11:14:01Z

@fwyzard I'm unclear as to the use case you are wanting to solve?

cmsbuild · 2019-06-22T11:37:08Z

-1

Tested at: e2c49c5

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-837552/1107/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests RelVals AddOn

Unit Tests:

I found errors in the following unit tests:

---> test TestConfigDP had ERRORS
---> test runtestPhysicsToolsNanoAOD had ERRORS
---> test testRecoMETMETProducers had ERRORS
---> test testTauEmbeddingProducers had ERRORS
---> test TestFWCoreParameterSetDriver had ERRORS

RelVals:

When I ran the RelVals I found an error in the following workflows:
4.53 step2

runTheMatrix-results/4.53_RunPhoton2012B+RunPhoton2012B+HLTD+RECODR1reHLT+HARVESTDR1reHLT/step2_RunPhoton2012B+RunPhoton2012B+HLTD+RECODR1reHLT+HARVESTDR1reHLT.log

7.3 step1

runTheMatrix-results/7.3_CosmicsSPLoose_UP18+CosmicsSPLoose_UP18+DIGICOS_UP18+RECOCOS_UP18+ALCACOS_UP18+HARVESTCOS_UP18/step1_CosmicsSPLoose_UP18+CosmicsSPLoose_UP18+DIGICOS_UP18+RECOCOS_UP18+ALCACOS_UP18+HARVESTCOS_UP18.log

8.0 step1

runTheMatrix-results/8.0_BeamHalo+BeamHalo+DIGICOS+RECOCOS+ALCABH+HARVESTCOS/step1_BeamHalo+BeamHalo+DIGICOS+RECOCOS+ALCABH+HARVESTCOS.log

9.0 step1

runTheMatrix-results/9.0_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST/step1_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST.log

25.0 step1

runTheMatrix-results/25.0_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT/step1_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT.log

136.731 step2

runTheMatrix-results/136.731_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2/step2_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2.log

136.788 step2

runTheMatrix-results/136.788_RunSinglePh2017B+RunSinglePh2017B+HLTDR2_2017+RECODR2_2017reHLT_skimSinglePh_Prompt+HARVEST2017/step2_RunSinglePh2017B+RunSinglePh2017B+HLTDR2_2017+RECODR2_2017reHLT_skimSinglePh_Prompt+HARVEST2017.log

4.22 step2

runTheMatrix-results/4.22_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC/step2_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC.log

158.0 step1

runTheMatrix-results/158.0_HydjetQ_B12_5020GeV_2018_ppReco+HydjetQ_B12_5020GeV_2018_ppReco+DIGIHI2018PPRECO+RECOHI2018PPRECO+ALCARECOHI2018PPRECO+HARVESTHI2018PPRECO/step1_HydjetQ_B12_5020GeV_2018_ppReco+HydjetQ_B12_5020GeV_2018_ppReco+DIGIHI2018PPRECO+RECOHI2018PPRECO+ALCARECOHI2018PPRECO+HARVESTHI2018PPRECO.log

1306.0 step1

runTheMatrix-results/1306.0_SingleMuPt1_UP15+SingleMuPt1_UP15+DIGIUP15+RECOUP15+HARVESTUP15/step1_SingleMuPt1_UP15+SingleMuPt1_UP15+DIGIUP15+RECOUP15+HARVESTUP15.log

1330.0 step1

runTheMatrix-results/1330.0_ZMM_13+ZMM_13+DIGIUP15+RECOUP15_L1TMuDQM+HARVESTUP15_L1TMuDQM/step1_ZMM_13+ZMM_13+DIGIUP15+RECOUP15_L1TMuDQM+HARVESTUP15_L1TMuDQM.log

5.1 step1

runTheMatrix-results/5.1_TTbar+TTbarFS+HARVESTFS/step1_TTbar+TTbarFS+HARVESTFS.log

25202.0 step1

runTheMatrix-results/25202.0_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25/step1_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25.log

10024.0 step1

runTheMatrix-results/10024.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017+ALCAFull_2017/step1_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017+ALCAFull_2017.log

10042.0 step1

runTheMatrix-results/10042.0_ZMM_13+ZMM_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017+ALCAFull_2017/step1_ZMM_13+ZMM_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017+ALCAFull_2017.log

135.4 step1

runTheMatrix-results/135.4_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS/step1_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS.log

10224.0 step1

runTheMatrix-results/10224.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017PU_GenSimFull+DigiFullPU_2017PU+RecoFullPU_2017PU+HARVESTFullPU_2017PU/step1_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017PU_GenSimFull+DigiFullPU_2017PU+RecoFullPU_2017PU+HARVESTFullPU_2017PU.log

10824.0 step1

runTheMatrix-results/10824.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2018_GenSimFull+DigiFull_2018+RecoFull_2018+HARVESTFull_2018+ALCAFull_2018+NanoFull_2018/step1_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2018_GenSimFull+DigiFull_2018+RecoFull_2018+HARVESTFull_2018+ALCAFull_2018+NanoFull_2018.log

136.85 step2

runTheMatrix-results/136.85_RunEGamma2018A+RunEGamma2018A+HLTDR2_2018+RECODR2_2018reHLT_skimEGamma_Offline_L1TEgDQM+HARVEST2018_L1TEgDQM/step2_RunEGamma2018A+RunEGamma2018A+HLTDR2_2018+RECODR2_2018reHLT_skimEGamma_Offline_L1TEgDQM+HARVEST2018_L1TEgDQM.log

101.0 step1

runTheMatrix-results/101.0_SingleElectronE120EHCAL+SingleElectronE120EHCAL/step1_SingleElectronE120EHCAL+SingleElectronE120EHCAL.log

11624.0 step1

runTheMatrix-results/11624.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2021_GenSimFull+DigiFull_2021+RecoFull_2021+HARVESTFull_2021+ALCAFull_2021/step1_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2021_GenSimFull+DigiFull_2021+RecoFull_2021+HARVESTFull_2021+ALCAFull_2021.log

136.7611 step2

runTheMatrix-results/136.7611_RunJetHT2016E_reminiaod+RunJetHT2016E_reminiaod+REMINIAOD_data2016_HIPM+HARVESTDR2_REMINIAOD_data2016_HIPM/step2_RunJetHT2016E_reminiaod+RunJetHT2016E_reminiaod+REMINIAOD_data2016_HIPM+HARVESTDR2_REMINIAOD_data2016_HIPM.log

20034.0 step1

runTheMatrix-results/20034.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D17_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D17+RecoFullGlobal_2023D17+HARVESTFullGlobal_2023D17/step1_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D17_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D17+RecoFullGlobal_2023D17+HARVESTFullGlobal_2023D17.log

21234.0 step1

runTheMatrix-results/21234.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D21_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D21+RecoFullGlobal_2023D21+HARVESTFullGlobal_2023D21/step1_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D21_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D21+RecoFullGlobal_2023D21+HARVESTFullGlobal_2023D21.log

27434.0 step1

runTheMatrix-results/27434.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D35_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D35+RecoFullGlobal_2023D35+HARVESTFullGlobal_2023D35/step1_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D35_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D35+RecoFullGlobal_2023D35+HARVESTFullGlobal_2023D35.log

136.8311 step2

runTheMatrix-results/136.8311_RunJetHT2017F_reminiaod+RunJetHT2017F_reminiaod+REMINIAOD_data2017+HARVEST2017_REMINIAOD_data2017/step2_RunJetHT2017F_reminiaod+RunJetHT2017F_reminiaod+REMINIAOD_data2017+HARVEST2017_REMINIAOD_data2017.log

29034.0 step1

runTheMatrix-results/29034.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D41_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D41+RecoFullGlobal_2023D41+HARVESTFullGlobal_2023D41/step1_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D41_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D41+RecoFullGlobal_2023D41+HARVESTFullGlobal_2023D41.log

250202.181 step1

runTheMatrix-results/250202.181_TTbar_13UP18+TTbar_13UP18+PREMIXUP18_PU25+DIGIPRMXLOCALUP18_PU25+RECOPRMXUP18_PU25+HARVESTUP18_PU25/step1_TTbar_13UP18+TTbar_13UP18+PREMIXUP18_PU25+DIGIPRMXLOCALUP18_PU25+RECOPRMXUP18_PU25+HARVESTUP18_PU25.log

1325.7 step2

runTheMatrix-results/1325.7_TTbar_13_94Xv2NanoAODINPUT+TTbar_13_94Xv2NanoAODINPUT+NANOEDMMC2017_94XMiniAODv2+HARVESTNANOAODMC2017_94XMiniAODv2/step2_TTbar_13_94Xv2NanoAODINPUT+TTbar_13_94Xv2NanoAODINPUT+NANOEDMMC2017_94XMiniAODv2+HARVESTNANOAODMC2017_94XMiniAODv2.log

140.56 step2

runTheMatrix-results/140.56_RunHI2018+RunHI2018+RECOHID18+HARVESTDHI18/step2_RunHI2018+RunHI2018+RECOHID18+HARVESTDHI18.log

140.53 step2

runTheMatrix-results/140.53_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI/step2_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI.log

1000.0 step2

runTheMatrix-results/1000.0_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT/step2_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT.log

1001.0 step2

runTheMatrix-results/1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVDSIPIXELCALRUN1+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5/step2_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVDSIPIXELCALRUN1+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5.log

AddOn:

I found errors in the following addon tests:

cmsDriver.py TTbar_8TeV_TuneCUETP8M1_cfi --conditions auto:run1_mc --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot Realistic8TeVCollision : FAILED - time: date Sat Jun 22 13:35:36 2019-date Sat Jun 22 13:35:26 2019 s - exit: 256
cmsDriver.py RelVal -s L1REPACK:Full --data --scenario=pp -n 10 --conditions auto:run2_hlt_Fake2 --relval 9000,50 --datatier "RAW" --eventcontent RAW --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2016 --fileout file:RelVal_Raw_Fake2_DATA.root --filein /store/data/Run2016B/JetHT/RAW/v1/000/272/762/00000/C666CDE2-E013-E611-B15A-02163E011DBE.root : FAILED - time: date Sat Jun 22 13:35:46 2019-date Sat Jun 22 13:35:32 2019 s - exit: 256
cmsRun /build/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_11_0_X_2019-06-21-2300/src/HLTrigger/Configuration/test/OnLine_HLT_Fake2.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Sat Jun 22 13:35:46 2019-date Sat Jun 22 13:35:32 2019 s - exit: 21504
cmsDriver.py RelVal -s HLT:Fake2,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_Fake2 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016 --processName=HLTRECO --filein file:RelVal_Raw_Fake2_DATA.root --fileout file:RelVal_Raw_Fake2_DATA_HLT_RECO.root : FAILED - time: date Sat Jun 22 13:35:46 2019-date Sat Jun 22 13:35:32 2019 s - exit: 256
cmsDriver.py RelVal -s L1REPACK:Full --data --scenario=pp -n 10 --conditions auto:run2_hlt_PRef --relval 9000,50 --datatier "RAW" --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2018 --eventcontent RAW --fileout file:RelVal_Raw_PRef_DATA.root --filein /store/data/Run2017A/HLTPhysics4/RAW/v1/000/295/606/00000/36DE5E0A-3645-E711-8FA1-02163E01A43B.root : FAILED - time: date Sat Jun 22 13:35:53 2019-date Sat Jun 22 13:35:37 2019 s - exit: 256
cmsRun /build/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_11_0_X_2019-06-21-2300/src/HLTrigger/Configuration/test/OnLine_HLT_PRef.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Sat Jun 22 13:35:53 2019-date Sat Jun 22 13:35:37 2019 s - exit: 21504
cmsDriver.py RelVal -s HLT:PRef,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_PRef --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018 --processName=HLTRECO --filein file:RelVal_Raw_PRef_DATA.root --fileout file:RelVal_Raw_PRef_DATA_HLT_RECO.root : FAILED - time: date Sat Jun 22 13:35:53 2019-date Sat Jun 22 13:35:37 2019 s - exit: 256
cmsDriver.py RelVal -s L1REPACK:Full --data --scenario=pp -n 10 --conditions auto:run2_hlt_HIon --relval 9000,50 --datatier "RAW" --eventcontent RAW --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2018_pp_on_AA --fileout file:RelVal_Raw_HIon_DATA.root --filein /store/data/Run2018D/HIMinimumBias0/RAW/v1/000/325/112/00000/660F62BB-9932-D645-A4A4-0BBBDA3963E8.root --customise_commands='from FWCore.ParameterSet.MassReplace import massSearchReplaceAnyInputTag; massSearchReplaceAnyInputTag(process.SimL1Emulator,"rawDataCollector","rawDataRepacker",False,True)' : FAILED - time: date Sat Jun 22 13:36:02 2019-date Sat Jun 22 13:35:38 2019 s - exit: 256
cmsRun /build/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_11_0_X_2019-06-21-2300/src/HLTrigger/Configuration/test/OnLine_HLT_HIon.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Sat Jun 22 13:36:02 2019-date Sat Jun 22 13:35:38 2019 s - exit: 21504
cmsDriver.py RelVal -s HLT:HIon,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_HIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018_pp_on_AA --processName=HLTRECO --filein file:RelVal_Raw_HIon_DATA.root --fileout file:RelVal_Raw_HIon_DATA_HLT_RECO.root : FAILED - time: date Sat Jun 22 13:36:02 2019-date Sat Jun 22 13:35:38 2019 s - exit: 256
cmsDriver.py TTbar_Tauola_13TeV_TuneCUETP8M1_cfi -s GEN,SIM,DIGI,L1,DIGI2RAW --mc --scenario=pp -n 10 --conditions auto:run2_mc_GRun --relval 9000,50 --datatier "GEN-SIM-RAW" --eventcontent RAWSIM --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2018 --fileout file:RelVal_Raw_GRun_MC.root : FAILED - time: date Sat Jun 22 13:36:24 2019-date Sat Jun 22 13:35:42 2019 s - exit: 256
cmsRun /build/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_11_0_X_2019-06-21-2300/src/HLTrigger/Configuration/test/OnLine_HLT_GRun.py realData=False globalTag=@ inputFiles=@ : FAILED - time: date Sat Jun 22 13:36:24 2019-date Sat Jun 22 13:35:42 2019 s - exit: 21504
cmsDriver.py RelVal -s HLT:GRun,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_GRun --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018 --processName=HLTRECO --filein file:RelVal_Raw_GRun_MC.root --fileout file:RelVal_Raw_GRun_MC_HLT_RECO.root : FAILED - time: date Sat Jun 22 13:36:24 2019-date Sat Jun 22 13:35:42 2019 s - exit: 256
cmsDriver.py TTbar_13TeV_TuneCUETP8M1_cfi --conditions auto:run2_mc_l1stage1 --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot NominalCollision2015 --era Run2_25ns : FAILED - time: date Sat Jun 22 13:35:58 2019-date Sat Jun 22 13:35:47 2019 s - exit: 256
cmsDriver.py TTbar_Tauola_13TeV_TuneCUETP8M1_cfi -s GEN,SIM,DIGI,L1,DIGI2RAW --mc --scenario=pp -n 10 --conditions auto:run2_mc_PRef --relval 9000,50 --datatier "GEN-SIM-RAW" --eventcontent RAWSIM --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2018 --fileout file:RelVal_Raw_PRef_MC.root : FAILED - time: date Sat Jun 22 13:36:10 2019-date Sat Jun 22 13:35:52 2019 s - exit: 256
cmsRun /build/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_11_0_X_2019-06-21-2300/src/HLTrigger/Configuration/test/OnLine_HLT_PRef.py realData=False globalTag=@ inputFiles=@ : FAILED - time: date Sat Jun 22 13:36:10 2019-date Sat Jun 22 13:35:52 2019 s - exit: 21504
cmsDriver.py RelVal -s HLT:PRef,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_PRef --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018 --processName=HLTRECO --filein file:RelVal_Raw_PRef_MC.root --fileout file:RelVal_Raw_PRef_MC_HLT_RECO.root : FAILED - time: date Sat Jun 22 13:36:10 2019-date Sat Jun 22 13:35:52 2019 s - exit: 256
cmsDriver.py TTbar_13TeV_TuneCUETP8M1_cfi --conditions auto:run2_mc --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot NominalCollision2015 --era Run2_2016 : FAILED - time: date Sat Jun 22 13:36:03 2019-date Sat Jun 22 13:35:53 2019 s - exit: 256
cmsDriver.py RelVal -s L1REPACK:Full --data --scenario=pp -n 10 --conditions auto:run2_hlt_PIon --relval 9000,50 --datatier "RAW" --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2018 --eventcontent RAW --fileout file:RelVal_Raw_PIon_DATA.root --filein /store/data/Run2017A/HLTPhysics4/RAW/v1/000/295/606/00000/36DE5E0A-3645-E711-8FA1-02163E01A43B.root : FAILED - time: date Sat Jun 22 13:36:12 2019-date Sat Jun 22 13:35:56 2019 s - exit: 256
cmsRun /build/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_11_0_X_2019-06-21-2300/src/HLTrigger/Configuration/test/OnLine_HLT_PIon.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Sat Jun 22 13:36:12 2019-date Sat Jun 22 13:35:56 2019 s - exit: 21504
cmsDriver.py RelVal -s HLT:PIon,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_PIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018 --processName=HLTRECO --filein file:RelVal_Raw_PIon_DATA.root --fileout file:RelVal_Raw_PIon_DATA_HLT_RECO.root : FAILED - time: date Sat Jun 22 13:36:12 2019-date Sat Jun 22 13:35:56 2019 s - exit: 256
cmsDriver.py TTbar_Tauola_8TeV_TuneCUETP8M1_cfi -s GEN,SIM,DIGI,L1,DIGI2RAW --mc --scenario=pp -n 10 --conditions auto:run1_mc_Fake --relval 9000,50 --datatier "GEN-SIM-RAW" --eventcontent RAWSIM --customise=HLTrigger/Configuration/CustomConfigs.L1T --fileout file:RelVal_Raw_Fake_MC.root : FAILED - time: date Sat Jun 22 13:36:12 2019-date Sat Jun 22 13:35:58 2019 s - exit: 256
cmsRun /build/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_11_0_X_2019-06-21-2300/src/HLTrigger/Configuration/test/OnLine_HLT_Fake.py realData=False globalTag=@ inputFiles=@ : FAILED - time: date Sat Jun 22 13:36:12 2019-date Sat Jun 22 13:35:58 2019 s - exit: 21504
cmsDriver.py RelVal -s HLT:Fake,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run1_mc_Fake --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --processName=HLTRECO --filein file:RelVal_Raw_Fake_MC.root --fileout file:RelVal_Raw_Fake_MC_HLT_RECO.root : FAILED - time: date Sat Jun 22 13:36:12 2019-date Sat Jun 22 13:35:58 2019 s - exit: 256
cmsDriver.py TTbar_Tauola_13TeV_TuneCUETP8M1_cfi -s GEN,SIM,DIGI,L1,DIGI2RAW --mc --scenario=pp -n 10 --conditions auto:run2_mc_HIon --relval 9000,50 --datatier "GEN-SIM-RAW" --eventcontent RAWSIM --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2018_pp_on_AA --fileout file:RelVal_Raw_HIon_MC.root : FAILED - time: date Sat Jun 22 13:36:24 2019-date Sat Jun 22 13:35:59 2019 s - exit: 256
cmsRun /build/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_11_0_X_2019-06-21-2300/src/HLTrigger/Configuration/test/OnLine_HLT_HIon.py realData=False globalTag=@ inputFiles=@ : FAILED - time: date Sat Jun 22 13:36:24 2019-date Sat Jun 22 13:35:59 2019 s - exit: 21504
cmsDriver.py RelVal -s HLT:HIon,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_HIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018_pp_on_AA --processName=HLTRECO --filein file:RelVal_Raw_HIon_MC.root --fileout file:RelVal_Raw_HIon_MC_HLT_RECO.root : FAILED - time: date Sat Jun 22 13:36:24 2019-date Sat Jun 22 13:35:59 2019 s - exit: 256
cmsDriver.py TTbar_Tauola_13TeV_TuneCUETP8M1_cfi -s GEN,SIM,DIGI,L1,DIGI2RAW --mc --scenario=pp -n 10 --conditions auto:run2_mc_Fake2 --relval 9000,50 --datatier "GEN-SIM-RAW" --eventcontent RAWSIM --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2016 --fileout file:RelVal_Raw_Fake2_MC.root : FAILED - time: date Sat Jun 22 13:36:16 2019-date Sat Jun 22 13:36:00 2019 s - exit: 256
cmsRun /build/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_11_0_X_2019-06-21-2300/src/HLTrigger/Configuration/test/OnLine_HLT_Fake2.py realData=False globalTag=@ inputFiles=@ : FAILED - time: date Sat Jun 22 13:36:16 2019-date Sat Jun 22 13:36:00 2019 s - exit: 21504
cmsDriver.py RelVal -s HLT:Fake2,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_Fake2 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016 --processName=HLTRECO --filein file:RelVal_Raw_Fake2_MC.root --fileout file:RelVal_Raw_Fake2_MC_HLT_RECO.root : FAILED - time: date Sat Jun 22 13:36:16 2019-date Sat Jun 22 13:36:00 2019 s - exit: 256
cmsDriver.py TTbar_Tauola_13TeV_TuneCUETP8M1_cfi -s GEN,SIM,DIGI,L1,DIGI2RAW --mc --scenario=pp -n 10 --conditions auto:run2_mc_Fake1 --relval 9000,50 --datatier "GEN-SIM-RAW" --eventcontent RAWSIM --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_25ns --fileout file:RelVal_Raw_Fake1_MC.root : FAILED - time: date Sat Jun 22 13:36:20 2019-date Sat Jun 22 13:36:05 2019 s - exit: 256
cmsRun /build/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_11_0_X_2019-06-21-2300/src/HLTrigger/Configuration/test/OnLine_HLT_Fake1.py realData=False globalTag=@ inputFiles=@ : FAILED - time: date Sat Jun 22 13:36:20 2019-date Sat Jun 22 13:36:05 2019 s - exit: 21504
cmsDriver.py RelVal -s HLT:Fake1,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_Fake1 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_25ns --processName=HLTRECO --filein file:RelVal_Raw_Fake1_MC.root --fileout file:RelVal_Raw_Fake1_MC_HLT_RECO.root : FAILED - time: date Sat Jun 22 13:36:20 2019-date Sat Jun 22 13:36:05 2019 s - exit: 256
cmsDriver.py RelVal -s L1REPACK:GT1 --data --scenario=pp -n 10 --conditions auto:run1_hlt_Fake --relval 9000,50 --datatier "RAW" --eventcontent RAW --customise=HLTrigger/Configuration/CustomConfigs.L1T --fileout file:RelVal_Raw_Fake_DATA.root --filein /store/data/Run2012A/MuEG/RAW/v1/000/191/718/14932935-E289-E111-830C-5404A6388697.root : FAILED - time: date Sat Jun 22 13:36:22 2019-date Sat Jun 22 13:36:09 2019 s - exit: 256
cmsRun /build/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_11_0_X_2019-06-21-2300/src/HLTrigger/Configuration/test/OnLine_HLT_Fake.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Sat Jun 22 13:36:22 2019-date Sat Jun 22 13:36:09 2019 s - exit: 21504
cmsDriver.py RelVal -s HLT:Fake,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run1_data_Fake --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --processName=HLTRECO --filein file:RelVal_Raw_Fake_DATA.root --fileout file:RelVal_Raw_Fake_DATA_HLT_RECO.root : FAILED - time: date Sat Jun 22 13:36:22 2019-date Sat Jun 22 13:36:09 2019 s - exit: 256
cmsDriver.py RelVal -s L1REPACK:Full --data --scenario=pp -n 10 --conditions auto:run2_hlt_GRun --relval 9000,50 --datatier "RAW" --eventcontent RAW --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2018 --fileout file:RelVal_Raw_GRun_DATA.root --filein /store/data/Run2017A/HLTPhysics4/RAW/v1/000/295/606/00000/36DE5E0A-3645-E711-8FA1-02163E01A43B.root : FAILED - time: date Sat Jun 22 13:36:47 2019-date Sat Jun 22 13:36:10 2019 s - exit: 256
cmsRun /build/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_11_0_X_2019-06-21-2300/src/HLTrigger/Configuration/test/OnLine_HLT_GRun.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Sat Jun 22 13:36:47 2019-date Sat Jun 22 13:36:10 2019 s - exit: 21504
cmsDriver.py RelVal -s HLT:GRun,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_GRun --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018 --processName=HLTRECO --filein file:RelVal_Raw_GRun_DATA.root --fileout file:RelVal_Raw_GRun_DATA_HLT_RECO.root : FAILED - time: date Sat Jun 22 13:36:47 2019-date Sat Jun 22 13:36:10 2019 s - exit: 256
cmsDriver.py RelVal -s L1REPACK:GCTGT --data --scenario=pp -n 10 --conditions auto:run2_hlt_Fake1 --relval 9000,50 --datatier "RAW" --eventcontent RAW --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_25ns --fileout file:RelVal_Raw_Fake1_DATA.root --filein /store/data/Run2015D/MuonEG/RAW/v1/000/256/677/00000/80950A90-745D-E511-92FD-02163E011C5D.root : FAILED - time: date Sat Jun 22 13:36:25 2019-date Sat Jun 22 13:36:11 2019 s - exit: 256
cmsRun /build/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_11_0_X_2019-06-21-2300/src/HLTrigger/Configuration/test/OnLine_HLT_Fake1.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Sat Jun 22 13:36:25 2019-date Sat Jun 22 13:36:11 2019 s - exit: 21504
cmsDriver.py RelVal -s HLT:Fake1,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_Fake1 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_25ns --processName=HLTRECO --filein file:RelVal_Raw_Fake1_DATA.root --fileout file:RelVal_Raw_Fake1_DATA_HLT_RECO.root : FAILED - time: date Sat Jun 22 13:36:25 2019-date Sat Jun 22 13:36:11 2019 s - exit: 256
cmsDriver.py TTbar_Tauola_13TeV_TuneCUETP8M1_cfi -s GEN,SIM,DIGI,L1,DIGI2RAW --mc --scenario=pp -n 10 --conditions auto:run2_mc_PIon --relval 9000,50 --datatier "GEN-SIM-RAW" --eventcontent RAWSIM --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2018 --fileout file:RelVal_Raw_PIon_MC.root : FAILED - time: date Sat Jun 22 13:36:32 2019-date Sat Jun 22 13:36:16 2019 s - exit: 256
cmsRun /build/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_11_0_X_2019-06-21-2300/src/HLTrigger/Configuration/test/OnLine_HLT_PIon.py realData=False globalTag=@ inputFiles=@ : FAILED - time: date Sat Jun 22 13:36:32 2019-date Sat Jun 22 13:36:16 2019 s - exit: 21504
cmsDriver.py RelVal -s HLT:PIon,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_PIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018 --processName=HLTRECO --filein file:RelVal_Raw_PIon_MC.root --fileout file:RelVal_Raw_PIon_MC_HLT_RECO.root : FAILED - time: date Sat Jun 22 13:36:32 2019-date Sat Jun 22 13:36:16 2019 s - exit: 256

cmsbuild · 2019-06-22T11:37:10Z

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

fwyzard · 2019-06-22T13:36:15Z

@fwyzard my first comment would be to please add tests to the bottom of Config.py. That way you'd also be sure Tasks function :) .

Sure, I'll see how to add those.

@fwyzard I'm unclear as to the use case you are wanting to solve?

In multiple occasions in the past years, and recently working on the GPU reconstruction and on the Phase 2 HLT, I had the need to break a configuration into its component pieces: modules, esmodules, paths & sequences, etc.

For example, keeping a separate set of cfi fiiles in a git repository could be an alternative to ConfDB...

I plat to add few options to the script:

generate the main cfg as a file instead of to standard output
generate all cfi files in a subdirectory
generate the cfi files in a different subdirectory per type (modules vs esmodules vs sequences etc.) if I can get the resulting python fragments to work

Once caveat so far: parsing the lot of cfi files is much slower than parsing the original configuration; eventually I will try to have a look at where the extra time gets spent.

fabiocos · 2019-10-30T09:08:54Z

please test

cmsbuild · 2019-10-30T09:09:23Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/3241/console Started: 2019/10/30 10:10

cmsbuild · 2019-10-30T14:01:30Z

+1
Tested at: 1ef6b19
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-837552/3241/summary.html

cmsbuild · 2019-10-30T14:01:34Z

Comparison job queued.

cmsbuild · 2019-10-30T15:42:35Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-837552/3241/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2961036
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2960694
DQMHistoTests: Total skipped: 341
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
Checked 147 log files, 16 edm output root files, 34 DQM output files

fwyzard · 2019-10-30T16:08:13Z

Unfortunately I will not be able to add the unit tests before the end of November (CHEP + holidays...).
Can this be merged as is ?

Dr15Jones · 2019-10-30T16:13:57Z

+1

cmsbuild · 2019-10-30T16:14:32Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

fabiocos · 2019-10-31T15:37:10Z

@fwyzard fine for me, I will integrate it today, they you may further update it later on

fabiocos · 2019-10-31T18:03:11Z

+1

cmsbuild added this to the CMSSW_11_0_X milestone Jun 21, 2019

cmsbuild added code-checks-pending comparison-pending core-pending orp-pending pending-signatures tests-pending labels Jun 21, 2019

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 21, 2019

cmsbuild added tests-started and removed tests-pending labels Jun 22, 2019

cmsbuild added comparison-notrun tests-rejected and removed comparison-pending tests-started labels Jun 22, 2019

This was referenced Jun 27, 2019

Ecal DPG - Deploy new software (based on BDT) to recover isolated dead channels… #27175

Merged

Make cms.optional etal work with python 3 #27399

Merged

Implement core support for concurrent IOVs #26592

Merged

fabiocos closed this Jul 1, 2019

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 30, 2019

cmsbuild added comparison-pending tests-pending and removed comparison-available tests-rejected labels Oct 30, 2019

cmsbuild added tests-started and removed tests-pending labels Oct 30, 2019

cmsbuild added tests-approved and removed tests-started labels Oct 30, 2019

cmsbuild added comparison-available and removed comparison-pending labels Oct 30, 2019

cmsbuild added core-approved fully-signed and removed core-pending pending-signatures labels Oct 30, 2019

cmsbuild added orp-approved and removed orp-pending labels Oct 31, 2019

cmsbuild merged commit 3ef807a into cms-sw:master Oct 31, 2019

fwyzard deleted the edmConfigSplit branch May 14, 2020 08:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a script to split a CMSSW configuration into individual cfi files #27305

Add a script to split a CMSSW configuration into individual cfi files #27305

fwyzard commented Jun 21, 2019 •

edited

cmsbuild commented Jun 21, 2019

fwyzard commented Jun 21, 2019

cmsbuild commented Jun 21, 2019

cmsbuild commented Jun 21, 2019

fabiocos commented Jun 22, 2019

cmsbuild commented Jun 22, 2019 •

edited

Dr15Jones commented Jun 22, 2019

Dr15Jones commented Jun 22, 2019

cmsbuild commented Jun 22, 2019

cmsbuild commented Jun 22, 2019

fwyzard commented Jun 22, 2019

fabiocos commented Oct 30, 2019

cmsbuild commented Oct 30, 2019 •

edited

cmsbuild commented Oct 30, 2019

cmsbuild commented Oct 30, 2019

cmsbuild commented Oct 30, 2019

fwyzard commented Oct 30, 2019

Dr15Jones commented Oct 30, 2019

cmsbuild commented Oct 30, 2019

fabiocos commented Oct 31, 2019

fabiocos commented Oct 31, 2019

Add a script to split a CMSSW configuration into individual cfi files #27305

Add a script to split a CMSSW configuration into individual cfi files #27305

Conversation

fwyzard commented Jun 21, 2019 • edited

PR description:

PR validation:

cmsbuild commented Jun 21, 2019

fwyzard commented Jun 21, 2019

cmsbuild commented Jun 21, 2019

cmsbuild commented Jun 21, 2019

fabiocos commented Jun 22, 2019

cmsbuild commented Jun 22, 2019 • edited

Dr15Jones commented Jun 22, 2019

Dr15Jones commented Jun 22, 2019

cmsbuild commented Jun 22, 2019

cmsbuild commented Jun 22, 2019

fwyzard commented Jun 22, 2019

fabiocos commented Oct 30, 2019

cmsbuild commented Oct 30, 2019 • edited

cmsbuild commented Oct 30, 2019

cmsbuild commented Oct 30, 2019

cmsbuild commented Oct 30, 2019

fwyzard commented Oct 30, 2019

Dr15Jones commented Oct 30, 2019

cmsbuild commented Oct 30, 2019

fabiocos commented Oct 31, 2019

fabiocos commented Oct 31, 2019

fwyzard commented Jun 21, 2019 •

edited

cmsbuild commented Jun 22, 2019 •

edited

cmsbuild commented Oct 30, 2019 •

edited