Clean up temporary 2017 workflows and eras, add "2017 design" workflow #15994

makortel · 2016-09-27T09:41:26Z

This PR in an attempt to clean up all the temporary 2017 workflows and eras introduced in pre12. The old "2017AllNew" workflow is made as the new "2017" workflow. It is possible that I missed something, but at least everything in this PR needs to be done.

In addition, a "2017 design" workflow is added for further detailed (tracking) studies with the new geometry (also after 810pre13).

Tested in 8_1_0_pre12 (rebased on top of ~~CMSSW_8_1_X_2016-09-28-2300~~ ~~CMSSW_8_1_X_2016-10-02-2300~~ ~~CMSSW_8_1_X_2016-10-06-1100~~ ~~CMSSW_8_1_X_2016-10-12-1100~~ CMSSW_8_1_X_2016-10-16-1100), no changes expected between IB 108XX.0, and IB+PR 100XX.0 (or phase0/phase2). Between IB 100XX.0 and IB+PR 100XX.0 changes can be expected everywhere depending on tracker or HCAL (details being a topic of the validation of pre12).

@schneiml @veszpv @kpedro88

cmsbuild · 2016-09-27T09:41:43Z

A new Pull Request was created by @makortel (Matti Kortelainen) for CMSSW_8_1_X.

It involves the following packages:

Configuration/Eras
Configuration/PyReleaseValidation
Configuration/StandardSequences
RecoPixelVertexing/PixelLowPtUtilities

@cvuosalo, @franzoni, @fabozzi, @cmsbuild, @srimanob, @slava77, @hengne, @davidlange6 can you please review it and eventually sign? Thanks.
@ghellwig, @GiacomoSguazzoni, @rovere, @VinInn, @Martin-Grunewald, @dgulhan this is something you requested to watch as well.
@slava77, @smuzaffar you are the release manager for this.

cms-bot commands are list here #13028

VinInn · 2016-09-27T09:49:10Z

@cmsbuild , please test

cmsbuild · 2016-09-27T09:49:26Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/15389/console

cmsbuild · 2016-09-27T12:22:19Z

-1

Tested at: 49a5a10

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-15994/15389/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests RelVals AddOn

Unit Tests:

I found errors in the following unit tests:

---> test TestConfigDP had ERRORS

RelVals:

When I ran the RelVals I found an error in the following worklfows:
5.1 step1

runTheMatrix-results/5.1_TTbar+TTbarFS+HARVESTFS/step1_TTbar+TTbarFS+HARVESTFS.log

4.22 step2

runTheMatrix-results/4.22_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC/step2_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC.log

135.4 step1

runTheMatrix-results/135.4_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS/step1_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS.log

140.53 step2

runTheMatrix-results/140.53_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI/step2_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI.log

101.0 step1

runTheMatrix-results/101.0_SingleElectronE120EHCAL+SingleElectronE120EHCAL/step1_SingleElectronE120EHCAL+SingleElectronE120EHCAL.log

8.0 step3

runTheMatrix-results/8.0_BeamHalo+BeamHalo+DIGICOS+RECOCOS+ALCABH+HARVESTCOS/step3_BeamHalo+BeamHalo+DIGICOS+RECOCOS+ALCABH+HARVESTCOS.log

1000.0 step2

runTheMatrix-results/1000.0_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT/step2_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT.log

1001.0 step2

runTheMatrix-results/1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5/step2_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5.log

1003.0 step2

runTheMatrix-results/1003.0_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM/step2_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM.log

1306.0 step3

runTheMatrix-results/1306.0_SingleMuPt1_UP15+SingleMuPt1_UP15+DIGIUP15+RECOUP15+HARVESTUP15/step3_SingleMuPt1_UP15+SingleMuPt1_UP15+DIGIUP15+RECOUP15+HARVESTUP15.log

10021.0 step3

runTheMatrix-results/10021.0_TenMuE_0_200+TenMuE_0_200_pythia8_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017/step3_TenMuE_0_200+TenMuE_0_200_pythia8_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017.log

1330.0 step3

runTheMatrix-results/1330.0_ZMM_13+ZMM_13+DIGIUP15+RECOUP15+HARVESTUP15/step3_ZMM_13+ZMM_13+DIGIUP15+RECOUP15+HARVESTUP15.log

136.731 step3

runTheMatrix-results/136.731_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2/step3_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2.log

9.0 step3

runTheMatrix-results/9.0_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST/step3_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST.log

25.0 step3

runTheMatrix-results/25.0_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT/step3_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT.log

20024.0 step3

runTheMatrix-results/20024.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2023D1_GenSimFull+DigiFull_2023D1+RecoFullGlobal_2023D1+HARVESTFullGlobal_2023D1/step3_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2023D1_GenSimFull+DigiFull_2023D1+RecoFullGlobal_2023D1+HARVESTFullGlobal_2023D1.log

10024.0 step3

runTheMatrix-results/10024.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017/step3_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017.log

50202.0 step3

runTheMatrix-results/50202.0_TTbar_13+TTbar_13+DIGIUP15_PU50+RECOUP15_PU50+HARVESTUP15_PU50/step3_TTbar_13+TTbar_13+DIGIUP15_PU50+RECOUP15_PU50+HARVESTUP15_PU50.log

25202.0 step3

runTheMatrix-results/25202.0_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25/step3_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25.log

20824.0 step3

runTheMatrix-results/20824.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2023D3_GenSimFull+DigiFull_2023D3+RecoFullGlobal_2023D3+HARVESTFullGlobal_2023D3/step3_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2023D3_GenSimFull+DigiFull_2023D3+RecoFullGlobal_2023D3+HARVESTFullGlobal_2023D3.log

AddOn:

I found errors in the following addon tests:

cmsDriver.py TTbar_8TeV_TuneCUETP8M1_cfi --conditions auto:run1_mc --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,HLT:@Fake,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot Realistic8TeVCollision : FAILED - time: date Tue Sep 27 14:03:16 2016-date Tue Sep 27 14:03:14 2016 s - exit: 256
cmsDriver.py RelVal -s HLT:PRef,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_PRef --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_PRef_DATA.root --fileout file:RelVal_Raw_PRef_DATA_HLT_RECO.root : FAILED - time: date Tue Sep 27 14:20:44 2016-date Tue Sep 27 14:03:20 2016 s - exit: 256
cmsDriver.py RelVal -s HLT:HIon,RAW2DIGI,L1Reco,RECO --data --scenario=HeavyIons -n 10 --conditions auto:run2_data_HIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016,Run2_HI --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_HIon_DATA.root --fileout file:RelVal_Raw_HIon_DATA_HLT_RECO.root : FAILED - time: date Tue Sep 27 14:09:50 2016-date Tue Sep 27 14:03:22 2016 s - exit: 256
cmsDriver.py RelVal -s HLT:GRun,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_GRun --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_GRun_MC.root --fileout file:RelVal_Raw_GRun_MC_HLT_RECO.root : FAILED - time: date Tue Sep 27 14:16:34 2016-date Tue Sep 27 14:03:27 2016 s - exit: 256
cmsDriver.py TTbar_13TeV_TuneCUETP8M1_cfi --conditions auto:run2_mc --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,HLT:@relval25ns,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot NominalCollision2015 --era Run2_25ns --magField 38T_PostLS1 : FAILED - time: date Tue Sep 27 14:03:30 2016-date Tue Sep 27 14:03:28 2016 s - exit: 256
cmsDriver.py RelVal -s HLT:PRef,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_PRef --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_PRef_MC.root --fileout file:RelVal_Raw_PRef_MC_HLT_RECO.root : FAILED - time: date Tue Sep 27 14:15:42 2016-date Tue Sep 27 14:03:29 2016 s - exit: 256
cmsDriver.py TTbar_13TeV_TuneCUETP8M1_cfi --conditions auto:run2_mc --fast -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,EI,HLT:@relval2016,VALIDATION --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot NominalCollision2015 --era Run2_2016 --magField 38T_PostLS1 : FAILED - time: date Tue Sep 27 14:03:33 2016-date Tue Sep 27 14:03:31 2016 s - exit: 256
cmsDriver.py RelVal -s HLT:PIon,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_PIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_PIon_DATA.root --fileout file:RelVal_Raw_PIon_DATA_HLT_RECO.root : FAILED - time: date Tue Sep 27 14:09:34 2016-date Tue Sep 27 14:03:35 2016 s - exit: 256
cmsDriver.py RelVal -s HLT:Fake,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run1_mc_Fake --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --processName=HLTRECO --filein file:RelVal_Raw_Fake_MC.root --fileout file:RelVal_Raw_Fake_MC_HLT_RECO.root : FAILED - time: date Tue Sep 27 14:12:25 2016-date Tue Sep 27 14:03:38 2016 s - exit: 256
cmsDriver.py RelVal -s HLT:HIon,RAW2DIGI,L1Reco,RECO --mc --scenario=HeavyIons -n 10 --conditions auto:run2_mc_HIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016,Run2_HI --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_HIon_MC.root --fileout file:RelVal_Raw_HIon_MC_HLT_RECO.root : FAILED - time: date Tue Sep 27 14:15:48 2016-date Tue Sep 27 14:03:39 2016 s - exit: 256
cmsDriver.py RelVal -s HLT:Fake1,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_Fake1 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_25ns --processName=HLTRECO --filein file:RelVal_Raw_Fake1_MC.root --fileout file:RelVal_Raw_Fake1_MC_HLT_RECO.root : FAILED - time: date Tue Sep 27 14:14:57 2016-date Tue Sep 27 14:04:13 2016 s - exit: 256
cmsDriver.py RelVal -s HLT:Fake,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run1_data_Fake --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --processName=HLTRECO --filein file:RelVal_Raw_Fake_DATA.root --fileout file:RelVal_Raw_Fake_DATA_HLT_RECO.root : FAILED - time: date Tue Sep 27 14:12:47 2016-date Tue Sep 27 14:09:59 2016 s - exit: 256
cmsDriver.py RelVal -s HLT:GRun,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_GRun --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_25ns --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_GRun_DATA.root --fileout file:RelVal_Raw_GRun_DATA_HLT_RECO.root : FAILED - time: date Tue Sep 27 14:16:43 2016-date Tue Sep 27 14:10:23 2016 s - exit: 256
cmsDriver.py RelVal -s HLT:Fake1,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_Fake1 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_25ns --processName=HLTRECO --filein file:RelVal_Raw_Fake1_DATA.root --fileout file:RelVal_Raw_Fake1_DATA_HLT_RECO.root : FAILED - time: date Tue Sep 27 14:14:34 2016-date Tue Sep 27 14:12:28 2016 s - exit: 256
cmsDriver.py RelVal -s HLT:PIon,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_PIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_PIon_MC.root --fileout file:RelVal_Raw_PIon_MC_HLT_RECO.root : FAILED - time: date Tue Sep 27 14:19:54 2016-date Tue Sep 27 14:12:53 2016 s - exit: 256

cmsbuild · 2016-09-27T13:47:50Z

Pull request #15994 was updated. @cvuosalo, @franzoni, @fabozzi, @cmsbuild, @srimanob, @slava77, @hengne, @davidlange6 can you please check and sign again.

makortel · 2016-09-27T13:48:28Z

Didn't clean up my developer area before running tests, and missed this import to a removed file.

slava77 · 2016-09-27T13:55:01Z

@cmsbuild please test

cmsbuild · 2016-09-27T13:55:15Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/15394/console

kpedro88 · 2016-09-27T15:50:24Z

The changes in this PR look fine to me from the HCAL side. (At some point we need to get the "final" 2017 geometry into the database and GT, but that can be handled separately once #15985 is resolved.)

cmsbuild · 2016-09-27T17:43:03Z

+1
Tested at: 31b33a3
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-15994/15394/summary.html

cmsbuild · 2016-09-27T19:06:28Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-15994/15394/summary.html

cmsbuild · 2016-10-17T14:39:41Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/15779/console

cmsbuild · 2016-10-17T16:04:22Z

+1
Tested at: 77eecdb
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-15994/15779/summary.html

mmusich · 2016-10-17T16:07:46Z

+1
depending on if this is merged fast or not, there are other Hcal conditions fixes in the pipeline.

abdoulline · 2016-10-17T16:09:37Z

... indeed, a bunch of HCAL conditions (including those for 2017) will

be submitted shortly...

On Mon, 17 Oct 2016, Marco Musich wrote:

+1
depending on if this is merged fast or not, there are other Hcal conditions fixes in the
pipeline.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the
thread.[AEx02r8tM8cbT-VNcxMWRcsoUCUApZj8ks5q051VgaJpZM4KHdOE.gif]

kpedro88 · 2016-10-17T16:25:47Z

so this will not be included in pre13 after all? #16232

mmusich · 2016-10-17T16:52:32Z

seems not: making conditions management a nightmare...

cmsbuild · 2016-10-17T17:39:46Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-15994/15779/summary.html

@slava77 comparisons for the following workflows were not done due to missing matrix map:

20024.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2023D1_GenSimHLBeamSpotFull+DigiFull_2023D1+RecoFullGlobal_2023D1+HARVESTFullGlobal_2023D1
22424.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2023D3Timing_GenSimHLBeamSpotFull+DigiFull_2023D3Timing+RecoFullGlobal_2023D3Timing+HARVESTFullGlobal_2023D3Timing

makortel · 2016-10-17T19:23:09Z

@mmusich Out of curiosity, would it help at all if we get this PR merged soon?

Anyway, I guess for pre13 RelVals we're interested only in 108xx workflows.

mmusich · 2016-10-18T07:35:41Z

@makortel indeed it would help, but the problem is that since this PR is changing the "meaning" of the phase1_2017_realistic key (including all the 2017 Hcal conditions, that were not included so far because of the geometry mismatch issue), we have a bunch of stuff now included in the queue:

$ conddb diff 81X_upgrade2017_realistic_Candidate_2016_10_05_11_41_20 81X_upgrade2017_realistic_v16
[2016-10-18 09:32:53,442] INFO: Connecting to pro [frontier://PromptProd/CMS_CONDITIONS]
Record                       Label     pro::81X_upgrade2017_realistic_Candidate_2016_10_05_11_41_20 Tag  pro::81X_upgrade2017_realistic_v16 Tag     
---------------------------  --------  ----------------------------------------------------------------  -----------------------------------------  
GeometryFileRcd              Extended  XMLFILE_Geometry_81YV7_Extended2017_mc                            XMLFILE_Geometry_81YV8_Extended2017dev_mc  
HcalChannelQualityRcd        -         HcalChannelQuality_v2.20_mc                                       HcalChannelQuality_2017_v1.0_mc            
HcalElectronicsMapRcd        -         HcalElectronicsMap_v8.02_mc                                       HcalElectronicsMap_2017_v1.0_mc            
HcalFlagHFDigiTimeParamsRcd  -         HcalFlagHFDigiTimeParams_v1.0_mc                                  HcalFlagHFDigiTimeParams_2017_v1.0_mc      
HcalFrontEndMapRcd           -         HcalFrontEndMap_v2.00_mc                                          HcalFrontEndMap_2017_v1.0_mc               
HcalGainsRcd                 -         HcalGains_v4.50_mc                                                HcalGains_2017_v1.0_mc                     
HcalL1TriggerObjectsRcd      -         HcalL1TriggerObjects_v1.00_mc                                     HcalL1TriggerObjects_2017_v1.0_mc          
HcalLUTCorrsRcd              -         HcalLUTCorrs_v1.01_mc                                             HcalLUTCorrs_2017_v1.0_mc                  
HcalLongRecoParamsRcd        -         HcalLongRecoParams_v1.0_mc                                        HcalLongRecoParams_2017_v1.0_mc            
HcalLutMetadataRcd           -         HcalLutMetadata_v4.00_mc                                          HcalLutMetadata_2017_v2.0_mc               
HcalMCParamsRcd              -         HcalMCParams_v5.0_mc                                              HcalMCParams_2017_v1.0_mc                  
HcalPFCorrsRcd               -         HcalPFCorrs_v3.00_mc                                              HcalPFCorrs_2017_v1.0_mc                   
HcalParametersRcd            -         HCALParameters_Geometry_81YV7                                     HCALParameters_Geometry_81YV8              
HcalPedestalWidthsRcd        -         HcalPedestalWidths_ADC_v8.00_mc                                   HcalPedestalWidths_2017_v1.0_mc            
HcalPedestalsRcd             -         HcalPedestals_ADC_v8.00_mc                                        HcalPedestals_2017_v1.0_mc                 
HcalQIEDataRcd               -         HcalQIEData_NormalMode_v8.00_mc                                   HcalQIEData_2017_v2.0_mc                   
HcalQIETypesRcd              -         HcalQIETypes_v1.00_mc                                             HcalQIETypes_2017_v1.0_mc                  
HcalRecoParamsRcd            -         HcalRecoParams_v9.0_mc                                            HcalRecoParams_2017_v2.0_mc                
HcalRespCorrsRcd             -         HcalRespCorrs_v5.00_mc                                            HcalRespCorrs_2017_v1.0_mc                 
HcalSiPMCharacteristicsRcd   -         HcalSiPMCharacteristics_v2.0_mc                                   HcalSiPMCharacteristics_2017_v2.0_mc       
HcalSiPMParametersRcd        -         HcalSiPMParameters_v2.0_mc                                        HcalSiPMParameters_2017_v2.0_mc            
HcalTPChannelParametersRcd   -         HcalTPChannelParameters_v1.0_mc                                   HcalTPChannelParameters_2017_v1.0_mc       
HcalTPParametersRcd          -         HcalTPParameters_v1.0_mc                                          HcalTPParameters_2017_v1.0_mc              
HcalTimeCorrsRcd             -         HcalTimeCorrs_v1.00_mc                                            HcalTimeCorrs_2017_v1.0_mc                 
HcalZSThresholdsRcd          -         HcalZSThresholds_v3.0_mc                                          HcalZSThresholds_2017_v1.0_mc              
L1TUtmTriggerMenuRcd         -         L1Menu_Collisions2016_v6r5_ugt_1board_xml                         L1Menu_Collisions2016_v6_xml               
PCaloTowerRcd                -         CTRECO_Geometry_81YV7                                             CTRECO_Geometry_81YV8                      
PHcalRcd                     -         HCALRECO_Geometry_81YV7                                           HCALRECO_Geometry_81YV8

that I will need to roll back and forth each time I will need to update the queue for any other reason (in the "current" meaning of phase1_2017_realistic HCal is still 2016), until this PR is merged

civanch · 2016-10-18T15:21:46Z

+1

fabozzi · 2016-10-18T15:30:09Z

+1

smuzaffar · 2016-11-03T21:05:11Z

Configuration/PyReleaseValidation/python/relval_2017.py

-#2017 WFs to run in IB (TenMuE_0_200, TTbar, ZEE, MinBias, TTbar PU, ZEE PU)
-numWFIB = [10021.0,10024.0,10025.0,10026.0,10023.0,10224.0,10225.0]
+#2017 WFs to run in IB (TenMuE_0_200, TTbar, ZEE, MinBias, TTbar PU, ZEE PU, TTbar design)
+numWFIB = [10021.0,10024.0,10025.0,10026.0,10023.0,10224.0,10225.0,10424.0]


@makortel , In 81X IBs, this workflow 10424.0 is failing as it is trying to get /RelValTTbar_13/1/GEN-SIM data. Can you please look why it is asking for wrong dataset?

my guess is tht https://github.com/makortel/cmssw/blob/77eecdbe8a344f294ced034a29773c8d8a6d0d27/Configuration/PyReleaseValidation/python/relval_steps.py#L1876 is where it is setting invalid dataset for TTbar_13TeV_TuneCUETP8M1_2017Design_GenSimFull

Umm... I need to take a look. Thanks for letting me know. Apparently the workflow has never worked with -i all...

@smuzaffar it's a combination of the line you referenced and https://github.com/makortel/cmssw/blob/77eecdbe8a344f294ced034a29773c8d8a6d0d27/Configuration/PyReleaseValidation/python/relval_steps.py#L1652-L1653

Would it make sense on
https://github.com/makortel/cmssw/blob/77eecdbe8a344f294ced034a29773c8d8a6d0d27/Configuration/PyReleaseValidation/python/relval_steps.py#L1875
that instead of "temporarily excluding 2023" with an explicit if, enable the input query only if defaultDataSets is not empty for that upgradeKey?

That would be better if you can make it work consistently... the actual use of defaultDataSets is a bit complicated.

Yeah, I noticed. I'm giving a try anyway (but it won't simplify the overall complexity).

#16460 should fix the issue.

ianna · 2016-11-03T21:50:15Z

Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py

@@ -62,33 +61,24 @@

 upgradeProperties[2017] = {
    '2017' : {
-        'Geom' : 'DB:Extended',
+        'Geom' : 'Extended2017new',


@makortel - this does not exist any more, please, use DB:Extended

@ianna don't worry, this is an old PR

ianna · 2016-11-03T21:50:40Z

Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py

-        'ScenToRun' : ['GenSimFull','DigiFull','RecoFull','ALCAFull','HARVESTFull'],
-    },
-    '2017AllNew' : {
+    '2017Design' : {


@makortel - same here

cmsbuild added this to the Next CMSSW_8_1_X milestone Sep 27, 2016

cmsbuild added comparison-pending operations-pending orp-pending pdmv-pending pending-signatures reconstruction-pending tests-pending labels Sep 27, 2016

cmsbuild added tests-started and removed tests-pending labels Sep 27, 2016

cmsbuild added tests-rejected tests-pending and removed tests-started tests-rejected labels Sep 27, 2016

cmsbuild added tests-started and removed tests-pending labels Sep 27, 2016

schneiml mentioned this pull request Sep 27, 2016

Remove old FPIX; make New FPIX default (RFC) #15986

Closed

cmsbuild added tests-approved and removed tests-started labels Sep 27, 2016

cmsbuild added the tests-pending label Oct 17, 2016

cmsbuild added tests-started and removed tests-pending labels Oct 17, 2016

cmsbuild added tests-approved and removed tests-started labels Oct 17, 2016

cmsbuild added alca-approved and removed alca-pending labels Oct 17, 2016

cmsbuild added comparison-available and removed comparison-pending labels Oct 17, 2016

cmsbuild added simulation-approved and removed simulation-pending labels Oct 18, 2016

cmsbuild added pdmv-approved and removed pdmv-pending labels Oct 18, 2016

davidlange6 merged commit a95432d into cms-sw:CMSSW_8_1_X Oct 18, 2016

makortel deleted the cleanup2017Workflows branch October 21, 2016 07:11

smuzaffar reviewed Nov 3, 2016

View reviewed changes

ianna reviewed Nov 3, 2016

View reviewed changes

makortel mentioned this pull request Nov 3, 2016

Fix 2017 design workflow for '-i all' #16460

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clean up temporary 2017 workflows and eras, add "2017 design" workflow #15994

Clean up temporary 2017 workflows and eras, add "2017 design" workflow #15994

makortel commented Sep 27, 2016 •

edited

cmsbuild commented Sep 27, 2016

VinInn commented Sep 27, 2016

cmsbuild commented Sep 27, 2016 •

edited

cmsbuild commented Sep 27, 2016

cmsbuild commented Sep 27, 2016

makortel commented Sep 27, 2016

slava77 commented Sep 27, 2016

cmsbuild commented Sep 27, 2016 •

edited

kpedro88 commented Sep 27, 2016

cmsbuild commented Sep 27, 2016

cmsbuild commented Sep 27, 2016

cmsbuild commented Oct 17, 2016 •

edited

cmsbuild commented Oct 17, 2016

mmusich commented Oct 17, 2016

abdoulline commented Oct 17, 2016

kpedro88 commented Oct 17, 2016

mmusich commented Oct 17, 2016

cmsbuild commented Oct 17, 2016

makortel commented Oct 17, 2016

mmusich commented Oct 18, 2016

civanch commented Oct 18, 2016

fabozzi commented Oct 18, 2016

smuzaffar Nov 3, 2016

smuzaffar Nov 3, 2016

makortel Nov 3, 2016

kpedro88 Nov 3, 2016

makortel Nov 3, 2016

kpedro88 Nov 3, 2016

makortel Nov 3, 2016

makortel Nov 3, 2016

ianna Nov 3, 2016

kpedro88 Nov 3, 2016

ianna Nov 3, 2016

Clean up temporary 2017 workflows and eras, add "2017 design" workflow #15994

Clean up temporary 2017 workflows and eras, add "2017 design" workflow #15994

Conversation

makortel commented Sep 27, 2016 • edited

cmsbuild commented Sep 27, 2016

VinInn commented Sep 27, 2016

cmsbuild commented Sep 27, 2016 • edited

cmsbuild commented Sep 27, 2016

cmsbuild commented Sep 27, 2016

makortel commented Sep 27, 2016

slava77 commented Sep 27, 2016

cmsbuild commented Sep 27, 2016 • edited

kpedro88 commented Sep 27, 2016

cmsbuild commented Sep 27, 2016

cmsbuild commented Sep 27, 2016

cmsbuild commented Oct 17, 2016 • edited

cmsbuild commented Oct 17, 2016

mmusich commented Oct 17, 2016

abdoulline commented Oct 17, 2016

kpedro88 commented Oct 17, 2016

mmusich commented Oct 17, 2016

cmsbuild commented Oct 17, 2016

makortel commented Oct 17, 2016

mmusich commented Oct 18, 2016

civanch commented Oct 18, 2016

fabozzi commented Oct 18, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

makortel commented Sep 27, 2016 •

edited

cmsbuild commented Sep 27, 2016 •

edited

cmsbuild commented Sep 27, 2016 •

edited

cmsbuild commented Oct 17, 2016 •

edited