Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add combined premixing stage1+stage2 workflows for 2016-18 #22956

Merged
merged 2 commits into from Apr 23, 2018
Merged

Add combined premixing stage1+stage2 workflows for 2016-18 #22956

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4056692
Add combined premixing stage1+stage2 workflows for 2016,17,18
makortel Apr 12, 2018
cd28089
Add stage1+stage2 workflow for ZMM as well
makortel Apr 13, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
10 changes: 7 additions & 3 deletions Configuration/PyReleaseValidation/python/WorkFlowRunner.py
View file
Open in desktop
Expand Up @@ -137,7 +137,9 @@ def closeCmd(i,ID):
cmd += com
if self.noRun:
cmd +=' --no_exec'
if inFile: #in case previous step used DAS query (either filelist of das:)
# in case previous step used DAS query (either filelist of das:)
# not to be applied for premixing stage1 to allow combiend stage1+stage2 workflow
if inFile and not 'premix_stage1' in cmd:
cmd += ' --filein '+inFile
inFile=None
if lumiRangeFile: #DAS query can also restrict lumi range
Expand All @@ -147,8 +149,10 @@ def closeCmd(i,ID):
if 'HARVESTING' in cmd and not 134==self.wf.numId and not '--filein' in cmd:
cmd+=' --filein file:step%d_inDQM.root --fileout file:step%d.root '%(istep-1,istep)
else:
if istep!=1 and not '--filein' in cmd:
cmd+=' --filein file:step%s.root '%(istep-1,)
# Disable input for premix stage1 to allow combined stage1+stage2 workflow
# Bit of a hack but works
if istep!=1 and not '--filein' in cmd and not 'premix_stage1' in cmd:
cmd+=' --filein file:step%s.root '%(istep-1,)
if not '--fileout' in com:
cmd+=' --fileout file:step%s.root '%(istep,)
if self.jobReport:
Expand Down
4 changes: 4 additions & 0 deletions Configuration/PyReleaseValidation/python/relval_premix.py
View file
Open in desktop
Expand Up @@ -19,6 +19,7 @@
# 25ns pile up overlay using premix
workflows[250200]=['',['ZEE_13','DIGIPRMXUP15_PU25','RECOPRMXUP15_PU25','HARVESTUP15_PU25']]
workflows[250202]=['',['TTbar_13','DIGIPRMXUP15_PU25','RECOPRMXUP15_PU25','HARVESTUP15_PU25']]
workflows[250202.1]=['',['TTbar_13','PREMIXUP15_PU25','DIGIPRMXLOCALUP15_PU25','RECOPRMXUP15_PU25','HARVESTUP15_PU25']]
workflows[250203]=['',['H125GGgluonfusion_13','DIGIPRMXUP15_PU25','RECOPRMXUP15_PU25','HARVESTUP15_PU25']]
workflows[250204]=['',['QQH1352T_13','DIGIPRMXUP15_PU25','RECOPRMXUP15_PU25','HARVESTUP15_PU25']]
workflows[250205]=['',['ZTT_13','DIGIPRMXUP15_PU25','RECOPRMXUP15_PU25','HARVESTUP15_PU25']]
Expand All @@ -28,6 +29,7 @@

workflows[250200.17]=['',['ZEE_13UP17','DIGIPRMXUP17_PU25','RECOPRMXUP17_PU25','HARVESTUP17_PU25']]
workflows[250202.17]=['',['TTbar_13UP17','DIGIPRMXUP17_PU25','RECOPRMXUP17_PU25','HARVESTUP17_PU25']]
workflows[250202.171]=['',['TTbar_13UP17','PREMIXUP17_PU25','DIGIPRMXLOCALUP17_PU25','RECOPRMXUP17_PU25','HARVESTUP17_PU25']]
workflows[250203.17]=['',['H125GGgluonfusion_13UP17','DIGIPRMXUP17_PU25','RECOPRMXUP17_PU25','HARVESTUP17_PU25']]
workflows[250204.17]=['',['QQH1352T_13UP17','DIGIPRMXUP17_PU25','RECOPRMXUP17_PU25','HARVESTUP17_PU25']]
workflows[250205.17]=['',['ZTT_13UP17','DIGIPRMXUP17_PU25','RECOPRMXUP17_PU25','HARVESTUP17_PU25']]
Expand All @@ -38,10 +40,12 @@

workflows[250200.18]=['',['ZEE_13UP18','DIGIPRMXUP18_PU25','RECOPRMXUP18_PU25_L1TEgDQM','HARVESTUP18_PU25_L1TEgDQM']]
workflows[250202.18]=['',['TTbar_13UP18','DIGIPRMXUP18_PU25','RECOPRMXUP18_PU25','HARVESTUP18_PU25']]
workflows[250202.181]=['',['TTbar_13UP18','PREMIXUP18_PU25','DIGIPRMXLOCALUP18_PU25','RECOPRMXUP18_PU25','HARVESTUP18_PU25']]
workflows[250203.18]=['',['H125GGgluonfusion_13UP18','DIGIPRMXUP18_PU25','RECOPRMXUP18_PU25','HARVESTUP18_PU25']]
workflows[250204.18]=['',['QQH1352T_13UP18','DIGIPRMXUP18_PU25','RECOPRMXUP18_PU25','HARVESTUP18_PU25']]
workflows[250205.18]=['',['ZTT_13UP18','DIGIPRMXUP18_PU25','RECOPRMXUP18_PU25','HARVESTUP18_PU25']]
workflows[250206.18]=['',['ZMM_13UP18','DIGIPRMXUP18_PU25','RECOPRMXUP18_PU25_L1TMuDQM','HARVESTUP18_PU25_L1TMuDQM']]
workflows[250206.181]=['',['ZMM_13UP18','PREMIXUP18_PU25','DIGIPRMXLOCALUP18_PU25','RECOPRMXUP18_PU25_L1TMuDQM','HARVESTUP18_PU25_L1TMuDQM']]
workflows[250207.18]=['',['NuGun_UP18','DIGIPRMXUP18_PU25','RECOPRMXUP18_PU25','HARVESTUP18_PU25']]
workflows[250208.18]=['',['SMS-T1tttt_mGl-1500_mLSP-100_13UP18','DIGIPRMXUP18_PU25','RECOPRMXUP18_PU25','HARVESTUP18_PU25']]

Expand Down
19 changes: 19 additions & 0 deletions Configuration/PyReleaseValidation/python/relval_steps.py
View file
Open in desktop
Expand Up @@ -1275,6 +1275,18 @@ def lhegensim2017(fragment,howMuch):
'--procModifiers': 'premix_stage2',
'--era' : 'Run2_2016'
}
# Specifying explicitly the --filein is not nice but that was the
# easiest way to "skip" the output of step2 (=premixing stage1) for
# filein (as it goes to pileup_input). It works (a bit accidentally
# though) also for "-i all" because in that case the --filein for DAS
# input is after this one in the list of command line arguments to
# cmsDriver, and gets then used in practice.
digiPremixLocalPileup = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this going to work in a relval setup?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume you mean that this works for the single workflow but not necessarily when injecting the production of the overall sample. I think this is a question for PdmV, how is this managed there?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davidlange6
I do not think these kinds of premix workflows can run as a relval, because in a taskchain all the output datasets from the workflow steps are being produced at the same time (and not one after the other). So the mixing step will search for an input PU dataset that is still in production. In addition to this, I am not sure if it is possible to specify in WMAgent an input PU taken from a previous step of the workflow.

But I think these workflows are only needed for testing a PR or to run in IB, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I think these workflows are only needed for testing a PR or to run in IB, right?

That (PR+IB) is indeed my intention. I'd think the current separate stages approach works indeed better for RelVal (and production).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, let's take this anyway in mind

"--filein": "file:step1.root",
"--pileup_input": "file:step2.root"
}
digiPremixLocalPileupUp2015Defaults25ns = merge([digiPremixLocalPileup,
digiPremixUp2015Defaults25ns])
digiPremixUp2015Defaults50ns=merge([{'-s':'DIGI:pdigi_valid,DATAMIX,L1,DIGI2RAW,HLT:@relval50ns'},
{'--conditions':'auto:run2_mc_50ns'},
{'--pileup_input' : 'das:/RelValPREMIXUP15_PU50/%s/GEN-SIM-DIGI-RAW'%baseDataSetRelease[6]},
Expand All @@ -1291,6 +1303,8 @@ def lhegensim2017(fragment,howMuch):
'--procModifiers': 'premix_stage2',
'--era' : 'Run2_2017'
}
digiPremixLocalPileupUp2017Defaults25ns = merge([digiPremixLocalPileup,
digiPremixUp2017Defaults25ns])


digiPremixUp2018Defaults25ns = {
Expand All @@ -1303,12 +1317,17 @@ def lhegensim2017(fragment,howMuch):
'--procModifiers': 'premix_stage2',
'--era' : 'Run2_2018'
}
digiPremixLocalPileupUp2018Defaults25ns = merge([digiPremixLocalPileup,
digiPremixUp2018Defaults25ns])


steps['DIGIPRMXUP15_PU25']=merge([digiPremixUp2015Defaults25ns])
steps['DIGIPRMXLOCALUP15_PU25']=merge([digiPremixLocalPileupUp2015Defaults25ns])
steps['DIGIPRMXUP15_PU50']=merge([digiPremixUp2015Defaults50ns])
steps['DIGIPRMXUP17_PU25']=merge([digiPremixUp2017Defaults25ns])
steps['DIGIPRMXLOCALUP17_PU25']=merge([digiPremixLocalPileupUp2017Defaults25ns])
steps['DIGIPRMXUP18_PU25']=merge([digiPremixUp2018Defaults25ns])
steps['DIGIPRMXLOCALUP18_PU25']=merge([digiPremixLocalPileupUp2018Defaults25ns])

premixProd25ns = {'-s' : 'DIGI,DATAMIX,L1,DIGI2RAW,HLT:@relval2016',
'--eventcontent' : 'PREMIXRAW',
Expand Down