ProdLike workflow for premixing

[kpedro88]

PR description:

A premixing production-like workflow is added for profiling purposes.

PR validation:

Ran new workflow 23434.9921 successfully.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-33037/21336

• This PR adds an extra 24KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/PyReleaseValidation/python/relval_2026.py modified in PR(s): Adding D76 to the short matrix for PR test, disable ME0 when phase2_GE0 modifier is used #33007
  • File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983, Patatrack integration - Pixel workflows (12/N) #31854

[cmsbuild]

A new Pull Request was created by @kpedro88 (Kevin Pedro) for master.

It involves the following packages:

Configuration/PyReleaseValidation

@jordan-martins, @chayanit, @wajidalikhan, @kpedro88, @cmsbuild, @srimanob can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @fabiocos, @slomeo this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[kpedro88]

test parameters:
workflow = 23434.9921

[kpedro88]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3c0d6d/13209/summary.html
COMMIT: d6cab1d
CMSSW: CMSSW_11_3_X_2021-03-02-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/33037/13209/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-3c0d6d/23434.9921_TTbar_14TeV+2026D49PU_PMXS1S2ProdLike+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+PREMIX_PremixHLBeamSpot14PU+DigiTriggerPU+RecoGlobalPU+MiniAODPU

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 37
• DQMHistoTests: Total histograms compared: 2750983
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2750960
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 36 files compared)
• Checked 156 log files, 37 edm output root files, 37 DQM output files

[chayanit]

@kpedro88 , @srimanob
Since the classes in upgrade become larger and larger (

cmssw/Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py

Lines 150 to 1078 in d6cab1d

	class UpgradeWorkflow_baseline(UpgradeWorkflow):
	def setup_(self, step, stepName, stepDict, k, properties):
	cust=properties.get('Custom', None)
	era=properties.get('Era', None)
	modifier=properties.get('ProcessModifier',None)
	if cust is not None: stepDict[stepName][k]['--customise']=cust
	if era is not None: stepDict[stepName][k]['--era']=era
	if modifier is not None: stepDict[stepName][k]['--procModifier']=modifier
	def condition(self, fragment, stepList, key, hasHarvest):
	return True
	upgradeWFs['baseline'] = UpgradeWorkflow_baseline(
	steps = [
	'GenSim',
	'GenSimHLBeamSpot',
	'GenSimHLBeamSpot14',
	'Digi',
	'DigiTrigger',
	'RecoLocal',
	'Reco',
	'RecoFakeHLT',
	'RecoGlobal',
	'HARVEST',
	'HARVESTFakeHLT',
	'FastSim',
	'HARVESTFast',
	'HARVESTGlobal',
	'ALCA',
	'Nano',
	'MiniAOD',
	],
	PU = [
	'DigiTrigger',
	'RecoLocal',
	'RecoGlobal',
	'Digi',
	'Reco',
	'RecoFakeHLT',
	'HARVEST',
	'HARVESTFakeHLT',
	'HARVESTGlobal',
	'MiniAOD',
	'Nano',
	],
	suffix = '',
	offset = 0.0,
	)
	# some commonalities among tracking WFs
	class UpgradeWorkflowTracking(UpgradeWorkflow):
	def condition(self, fragment, stepList, key, hasHarvest):
	result = (fragment=="TTbar_13" or fragment=="TTbar_14TeV") and not 'PU' in key and hasHarvest and self.condition_(fragment, stepList, key, hasHarvest)
	if result:
	# skip ALCA and Nano
	skipList = [s for s in stepList if (("ALCA" in s) or ("Nano" in s))]
	for skip in skipList:
	stepList.remove(skip)
	return result
	def condition_(self, fragment, stepList, key, hasHarvest):
	return True
	class UpgradeWorkflow_trackingOnly(UpgradeWorkflowTracking):
	def setup_(self, step, stepName, stepDict, k, properties):
	if 'Reco' in step: stepDict[stepName][k] = merge([self.step3, stepDict[step][k]])
	elif 'HARVEST' in step: stepDict[stepName][k] = merge([{'-s': 'HARVESTING:@trackingOnlyValidation+@trackingOnlyDQM'}, stepDict[step][k]])
	upgradeWFs['trackingOnly'] = UpgradeWorkflow_trackingOnly(
	steps = [
	'Reco',
	'HARVEST',
	'RecoGlobal',
	'HARVESTGlobal',
	],
	PU = [],
	suffix = '_trackingOnly',
	offset = 0.1,
	)
	upgradeWFs['trackingOnly'].step3 = {
	'-s': 'RAW2DIGI,RECO:reconstruction_trackingOnly,VALIDATION:@trackingOnlyValidation,DQM:@trackingOnlyDQM',
	'--datatier':'GEN-SIM-RECO,DQMIO',
	'--eventcontent':'RECOSIM,DQM',
	}
	# used outside of upgrade WFs
	step3_trackingOnly = upgradeWFs['trackingOnly'].step3
	class UpgradeWorkflow_trackingRun2(UpgradeWorkflowTracking):
	def setup_(self, step, stepName, stepDict, k, properties):
	if 'Reco' in step and stepDict[step][k]['--era']=='Run2_2017':
	stepDict[stepName][k] = merge([{'--era': 'Run2_2017_trackingRun2'}, stepDict[step][k]])
	def condition_(self, fragment, stepList, key, hasHarvest):
	return '2017' in key
	upgradeWFs['trackingRun2'] = UpgradeWorkflow_trackingRun2(
	steps = [
	'Reco',
	],
	PU = [],
	suffix = '_trackingRun2',
	offset = 0.2,
	)
	class UpgradeWorkflow_trackingOnlyRun2(UpgradeWorkflowTracking):
	def setup_(self, step, stepName, stepDict, k, properties):
	if 'Reco' in step and stepDict[step][k]['--era']=='Run2_2017':
	stepDict[stepName][k] = merge([{'--era': 'Run2_2017_trackingRun2'}, self.step3, stepDict[step][k]])
	elif 'HARVEST' in step: stepDict[stepName][k] = merge([{'-s': 'HARVESTING:@trackingOnlyValidation+@trackingOnlyDQM'}, stepDict[step][k]])
	def condition_(self, fragment, stepList, key, hasHarvest):
	return '2017' in key
	upgradeWFs['trackingOnlyRun2'] = UpgradeWorkflow_trackingOnlyRun2(
	steps = [
	'Reco',
	'HARVEST',
	],
	PU = [],
	suffix = '_trackingOnlyRun2',
	offset = 0.3,
	)
	upgradeWFs['trackingOnlyRun2'].step3 = upgradeWFs['trackingOnly'].step3
	class UpgradeWorkflow_trackingLowPU(UpgradeWorkflowTracking):
	def setup_(self, step, stepName, stepDict, k, properties):
	if 'Reco' in step and stepDict[step][k]['--era']=='Run2_2017':
	stepDict[stepName][k] = merge([{'--era': 'Run2_2017_trackingLowPU'}, stepDict[step][k]])
	def condition_(self, fragment, stepList, key, hasHarvest):
	return '2017' in key
	upgradeWFs['trackingLowPU'] = UpgradeWorkflow_trackingLowPU(
	steps = [
	'Reco',
	],
	PU = [],
	suffix = '_trackingLowPU',
	offset = 0.4,
	)
	class UpgradeWorkflow_pixelTrackingOnly(UpgradeWorkflowTracking):
	def setup_(self, step, stepName, stepDict, k, properties):
	if 'Reco' in step: stepDict[stepName][k] = merge([self.step3, stepDict[step][k]])
	elif 'HARVEST' in step: stepDict[stepName][k] = merge([{'-s': 'HARVESTING:@trackingOnlyValidation+@pixelTrackingOnlyDQM'}, stepDict[step][k]])
	def condition_(self, fragment, stepList, key, hasHarvest):
	return '2017' in key or '2018' in key or '2021' in key
	upgradeWFs['pixelTrackingOnly'] = UpgradeWorkflow_pixelTrackingOnly(
	steps = [
	'Reco',
	'HARVEST',
	'RecoGlobal',
	'HARVESTGlobal',
	],
	PU = [],
	suffix = '_pixelTrackingOnly',
	offset = 0.5,
	)
	upgradeWFs['pixelTrackingOnly'].step3 = {
	'-s': 'RAW2DIGI:RawToDigi_pixelOnly,RECO:reconstruction_pixelTrackingOnly,VALIDATION:@pixelTrackingOnlyValidation,DQM:@pixelTrackingOnlyDQM',
	'--datatier': 'GEN-SIM-RECO,DQMIO',
	'--eventcontent': 'RECOSIM,DQM',
	}
	class UpgradeWorkflow_trackingMkFit(UpgradeWorkflowTracking):
	def setup_(self, step, stepName, stepDict, k, properties):
	if 'Reco' in step: stepDict[stepName][k] = merge([self.step3, stepDict[step][k]])
	def condition_(self, fragment, stepList, key, hasHarvest):
	return '2017' in key or '2021' in key
	upgradeWFs['trackingMkFit'] = UpgradeWorkflow_trackingMkFit(
	steps = [
	'Reco',
	'RecoGlobal',
	],
	PU = [],
	suffix = '_trackingMkFit',
	offset = 0.7,
	)
	upgradeWFs['trackingMkFit'].step3 = {
	'--procModifiers': 'trackingMkFit'
	}
	#DeepCore seeding for JetCore iteration workflow
	class UpgradeWorkflow_seedingDeepCore(UpgradeWorkflow):
	def setup_(self, step, stepName, stepDict, k, properties):
	if 'Reco' in step or 'HARVEST' in step: stepDict[stepName][k] = merge([{'--procModifiers': 'seedingDeepCore'}, stepDict[step][k]])
	def condition(self, fragment, stepList, key, hasHarvest):
	result = (fragment=="QCD_Pt_1800_2400_14" or fragment=="TTbar_14TeV" ) and ('2021' in key or '2024' in key) and hasHarvest
	if result:
	# skip ALCA and Nano
	skipList = [s for s in stepList if (("ALCA" in s) or ("Nano" in s))]
	for skip in skipList:
	stepList.remove(skip)
	return result
	upgradeWFs['seedingDeepCore'] = UpgradeWorkflow_seedingDeepCore(
	steps = [
	'Reco',
	'HARVEST',
	'RecoGlobal',
	'HARVESTGlobal',
	],
	PU = [],
	suffix = '_seedingDeepCore',
	offset = 0.17,
	)
	# Vector Hits workflows
	class UpgradeWorkflow_vectorHits(UpgradeWorkflow):
	def setup_(self, step, stepName, stepDict, k, properties):
	stepDict[stepName][k] = merge([{'--procModifiers': 'vectorHits'}, stepDict[step][k]])
	def condition(self, fragment, stepList, key, hasHarvest):
	return fragment=="TTbar_14TeV" and '2026' in key
	upgradeWFs['vectorHits'] = UpgradeWorkflow_vectorHits(
	steps = [
	'RecoGlobal',
	],
	PU = [
	'RecoGlobal',
	],
	suffix = '_vectorHits',
	offset = 0.9,
	)
	# Track DNN workflows
	class UpgradeWorkflow_trackdnn(UpgradeWorkflow):
	def setup_(self, step, stepName, stepDict, k, properties):
	stepDict[stepName][k] = merge([{'--procModifiers': 'trackdnn'}, stepDict[step][k]])
	def condition(self, fragment, stepList, key, hasHarvest):
	return fragment=="TTbar_14TeV" and '2021' in key
	upgradeWFs['trackdnn'] = UpgradeWorkflow_trackdnn(
	steps = [
	'Reco',
	],
	PU = [
	'Reco',
	],
	suffix = '_trackdnn',
	offset = 0.91,
	)
	# MLPF workflows
	class UpgradeWorkflow_mlpf(UpgradeWorkflow):
	def setup_(self, step, stepName, stepDict, k, properties):
	if 'Reco' in step:
	stepDict[stepName][k] = merge([self.step3, stepDict[step][k]])
	def condition(self, fragment, stepList, key, hasHarvest):
	return fragment=="TTbar_14TeV" and '2021' in key
	upgradeWFs['mlpf'] = UpgradeWorkflow_mlpf(
	steps = [
	'Reco',
	],
	PU = [
	'Reco',
	],
	suffix = '_mlpf',
	offset = 0.13,
	)
	upgradeWFs['mlpf'].step3 = {
	'--datatier': 'GEN-SIM-RECO,RECOSIM,MINIAODSIM,DQMIO',
	'--eventcontent': 'FEVTDEBUGHLT,RECOSIM,MINIAODSIM,DQM',
	'--procModifiers': 'mlpf'
	}
	# Patatrack workflows
	class UpgradeWorkflowPatatrack(UpgradeWorkflow):
	def condition(self, fragment, stepList, key, hasHarvest):
	is_2018_ttbar = ('2018' in key and fragment=="TTbar_13")
	is_2021_ttbar = ('2021' in key and fragment=="TTbar_14TeV")
	is_2018_zmumu = ('2018' in key and fragment=="ZMM_13")
	is_2021_zmumu = ('2021' in key and fragment=="ZMM_14")
	result = any((is_2018_ttbar, is_2021_ttbar, is_2018_zmumu, is_2021_zmumu)) and hasHarvest and self.condition_(fragment, stepList, key, hasHarvest)
	if result:
	# skip ALCA and Nano
	skipList = [s for s in stepList if (("ALCA" in s) or ("Nano" in s))]
	for skip in skipList:
	stepList.remove(skip)
	return result
	def condition_(self, fragment, stepList, key, hasHarvest):
	return True
	class UpgradeWorkflowPatatrack_PixelOnlyCPU(UpgradeWorkflowPatatrack):
	def setup_(self, step, stepName, stepDict, k, properties):
	if 'Reco' in step:
	stepDict[stepName][k] = merge([self.step3, stepDict[step][k]])
	elif 'HARVEST' in step:
	stepDict[stepName][k] = merge([{'-s': 'HARVESTING:@trackingOnlyValidation+@pixelTrackingOnlyDQM'}, stepDict[step][k]])
	def condition_(self, fragment, stepList, key, hasHarvest):
	return '2018' in key or '2021' in key
	upgradeWFs['PatatrackPixelOnlyCPU'] = UpgradeWorkflowPatatrack_PixelOnlyCPU(
	steps = [
	'Reco',
	'HARVEST',
	'RecoFakeHLT',
	'HARVESTFakeHLT',
	'RecoGlobal',
	'HARVESTGlobal',
	],
	PU = [],
	suffix = 'Patatrack_PixelOnlyCPU',
	offset = 0.501,
	)
	upgradeWFs['PatatrackPixelOnlyCPU'].step3 = {
	'-s': 'RAW2DIGI:RawToDigi_pixelOnly,RECO:reconstruction_pixelTrackingOnly,VALIDATION:@pixelTrackingOnlyValidation,DQM:@pixelTrackingOnlyDQM',
	'--datatier': 'GEN-SIM-RECO,DQMIO',
	'--eventcontent': 'RECOSIM,DQM',
	'--procModifiers': 'pixelNtupleFit'
	}
	class UpgradeWorkflowPatatrack_PixelOnlyGPU(UpgradeWorkflowPatatrack):
	def setup_(self, step, stepName, stepDict, k, properties):
	if 'Reco' in step:
	stepDict[stepName][k] = merge([self.step3, stepDict[step][k]])
	elif 'HARVEST' in step:
	stepDict[stepName][k] = merge([{'-s': 'HARVESTING:@trackingOnlyValidation+@pixelTrackingOnlyDQM'}, stepDict[step][k]])
	def condition_(self, fragment, stepList, key, hasHarvest):
	return '2018' in key or '2021' in key
	upgradeWFs['PatatrackPixelOnlyGPU'] = UpgradeWorkflowPatatrack_PixelOnlyGPU(
	steps = [
	'Reco',
	'HARVEST',
	'RecoFakeHLT',
	'HARVESTFakeHLT',
	'RecoGlobal',
	'HARVESTGlobal',
	],
	PU = [],
	suffix = 'Patatrack_PixelOnlyGPU',
	offset = 0.502,
	)
	upgradeWFs['PatatrackPixelOnlyGPU'].step3 = {
	'-s': 'RAW2DIGI:RawToDigi_pixelOnly,RECO:reconstruction_pixelTrackingOnly,VALIDATION:@pixelTrackingOnlyValidation,DQM:@pixelTrackingOnlyDQM',
	'--datatier': 'GEN-SIM-RECO,DQMIO',
	'--eventcontent': 'RECOSIM,DQM',
	'--procModifiers': 'gpu'
	}
	class UpgradeWorkflowPatatrack_ECALOnlyCPU(UpgradeWorkflowPatatrack):
	def setup_(self, step, stepName, stepDict, k, properties):
	if 'Reco' in step:
	stepDict[stepName][k] = merge([self.step3, stepDict[step][k]])
	elif 'HARVEST' in step:
	stepDict[stepName][k] = merge([{'-s': 'HARVESTING:@ecalOnlyValidation+@ecal'}, stepDict[step][k]])
	def condition_(self, fragment, stepList, key, hasHarvest):
	return '2018' in key or '2021' in key
	upgradeWFs['PatatrackECALOnlyCPU'] = UpgradeWorkflowPatatrack_ECALOnlyCPU(
	steps = [
	'Reco',
	'HARVEST',
	'RecoFakeHLT',
	'HARVESTFakeHLT',
	'RecoGlobal',
	'HARVESTGlobal',
	],
	PU = [],
	suffix = 'Patatrack_ECALOnlyCPU',
	offset = 0.511,
	)
	upgradeWFs['PatatrackECALOnlyCPU'].step3 = {
	'-s': 'RAW2DIGI:RawToDigi_ecalOnly,RECO:reconstruction_ecalOnly,VALIDATION:@ecalOnlyValidation,DQM:@ecalOnly',
	'--datatier': 'GEN-SIM-RECO,DQMIO',
	'--eventcontent': 'RECOSIM,DQM',
	}
	class UpgradeWorkflowPatatrack_ECALOnlyGPU(UpgradeWorkflowPatatrack):
	def setup_(self, step, stepName, stepDict, k, properties):
	if 'Reco' in step:
	stepDict[stepName][k] = merge([self.step3, stepDict[step][k]])
	elif 'HARVEST' in step:
	stepDict[stepName][k] = merge([{'-s': 'HARVESTING:@ecalOnlyValidation+@ecal'}, stepDict[step][k]])
	def condition_(self, fragment, stepList, key, hasHarvest):
	return '2018' in key or '2021' in key
	upgradeWFs['PatatrackECALOnlyGPU'] = UpgradeWorkflowPatatrack_ECALOnlyGPU(
	steps = [
	'Reco',
	'HARVEST',
	'RecoFakeHLT',
	'HARVESTFakeHLT',
	'RecoGlobal',
	'HARVESTGlobal',
	],
	PU = [],
	suffix = 'Patatrack_ECALOnlyGPU',
	offset = 0.512,
	)
	upgradeWFs['PatatrackECALOnlyGPU'].step3 = {
	'-s': 'RAW2DIGI:RawToDigi_ecalOnly,RECO:reconstruction_ecalOnly,VALIDATION:@ecalOnlyValidation,DQM:@ecalOnly',
	'--datatier': 'GEN-SIM-RECO,DQMIO',
	'--eventcontent': 'RECOSIM,DQM',
	'--procModifiers': 'gpu'
	}
	class UpgradeWorkflowPatatrack_HCALOnlyCPU(UpgradeWorkflowPatatrack):
	def setup_(self, step, stepName, stepDict, k, properties):
	if 'Reco' in step:
	stepDict[stepName][k] = merge([self.step3, stepDict[step][k]])
	elif 'HARVEST' in step:
	stepDict[stepName][k] = merge([{'-s': 'HARVESTING:@hcalOnlyValidation+@hcalOnly+@hcal2Only'}, stepDict[step][k]])
	def condition_(self, fragment, stepList, key, hasHarvest):
	return '2018' in key or '2021' in key
	upgradeWFs['PatatrackHCALOnlyCPU'] = UpgradeWorkflowPatatrack_HCALOnlyCPU(
	steps = [
	'Reco',
	'HARVEST',
	'RecoFakeHLT',
	'HARVESTFakeHLT',
	'RecoGlobal',
	'HARVESTGlobal',
	],
	PU = [],
	suffix = 'Patatrack_HCALOnlyCPU',
	offset = 0.521,
	)
	upgradeWFs['PatatrackHCALOnlyCPU'].step3 = {
	'-s': 'RAW2DIGI:RawToDigi_hcalOnly,RECO:reconstruction_hcalOnly,VALIDATION:@hcalOnlyValidation,DQM:@hcalOnly+@hcal2Only',
	'--datatier': 'GEN-SIM-RECO,DQMIO',
	'--eventcontent': 'RECOSIM,DQM',
	}
	class UpgradeWorkflowPatatrack_HCALOnlyGPU(UpgradeWorkflowPatatrack):
	def setup_(self, step, stepName, stepDict, k, properties):
	if 'Reco' in step:
	stepDict[stepName][k] = merge([self.step3, stepDict[step][k]])
	elif 'HARVEST' in step:
	stepDict[stepName][k] = merge([{'-s': 'HARVESTING:@hcalOnlyValidation+@hcalOnly+@hcal2Only'}, stepDict[step][k]])
	def condition_(self, fragment, stepList, key, hasHarvest):
	return '2018' in key or '2021' in key
	upgradeWFs['PatatrackHCALOnlyGPU'] = UpgradeWorkflowPatatrack_HCALOnlyGPU(
	steps = [
	'Reco',
	'HARVEST',
	'RecoFakeHLT',
	'HARVESTFakeHLT',
	'RecoGlobal',
	'HARVESTGlobal',
	],
	PU = [],
	suffix = 'Patatrack_HCALOnlyGPU',
	offset = 0.522,
	)
	upgradeWFs['PatatrackHCALOnlyGPU'].step3 = {
	'-s': 'RAW2DIGI:RawToDigi_hcalOnly,RECO:reconstruction_hcalOnly,VALIDATION:@hcalOnlyValidation,DQM:@hcalOnly+@hcal2Only',
	'--datatier': 'GEN-SIM-RECO,DQMIO',
	'--eventcontent': 'RECOSIM,DQM',
	'--procModifiers': 'gpu'
	}
	# end of Patatrack workflows
	class UpgradeWorkflow_ProdLike(UpgradeWorkflow):
	def setup_(self, step, stepName, stepDict, k, properties):
	if 'Digi' in step and 'Trigger' not in step:
	stepDict[stepName][k] = merge([{'-s': 'DIGI,L1,DIGI2RAW,HLT:@relval2021', '--datatier':'GEN-SIM-DIGI-RAW', '--eventcontent':'RAWSIM'}, stepDict[step][k]])
	elif 'Reco' in step:
	stepDict[stepName][k] = merge([{'-s': 'RAW2DIGI,L1Reco,RECO,RECOSIM', '--datatier':'AODSIM', '--eventcontent':'AODSIM'}, stepDict[step][k]])
	elif 'MiniAOD' in step:
	# the separate miniAOD step is used here
	stepDict[stepName][k] = deepcopy(stepDict[step][k])
	if 'ALCA' in step or 'HARVEST' in step:
	# remove step
	stepDict[stepName][k] = None
	if 'Nano' in step:
	stepDict[stepName][k] = merge([{'--filein':'file:step4.root'}, stepDict[step][k]])
	def condition(self, fragment, stepList, key, hasHarvest):
	return fragment=="TTbar_14TeV" and ('2026' in key or '2021' in key)
	upgradeWFs['ProdLike'] = UpgradeWorkflow_ProdLike(
	steps = [
	'Digi',
	'Reco',
	'RecoGlobal',
	'HARVEST',
	'HARVESTGlobal',
	'MiniAOD',
	'ALCA',
	'Nano',
	],
	PU = [
	'Digi',
	'Reco',
	'RecoGlobal',
	'HARVEST',
	'HARVESTGlobal',
	'MiniAOD',
	'ALCA',
	'Nano',
	],
	suffix = '_ProdLike',
	offset = 0.21,
	)
	class UpgradeWorkflow_Neutron(UpgradeWorkflow):
	def setup_(self, step, stepName, stepDict, k, properties):
	if 'GenSim' in step:
	custNew = "SimG4Core/Application/NeutronBGforMuonsXS_cff.customise"
	else:
	custNew = "SLHCUpgradeSimulations/Configuration/customise_mixing.customise_Mix_LongLived_Neutrons"
	stepDict[stepName][k] = deepcopy(stepDict[step][k])
	if '--customise' in stepDict[stepName][k].keys():
	stepDict[stepName][k]['--customise'] += ","+custNew
	else:
	stepDict[stepName][k]['--customise'] = custNew
	def condition(self, fragment, stepList, key, hasHarvest):
	return any(fragment==nfrag for nfrag in self.neutronFrags) and any(nkey in key for nkey in self.neutronKeys)
	upgradeWFs['Neutron'] = UpgradeWorkflow_Neutron(
	steps = [
	'GenSim',
	'GenSimHLBeamSpot',
	'GenSimHLBeamSpot14',
	'Digi',
	'DigiTrigger',
	],
	PU = [
	'Digi',
	'DigiTrigger',
	],
	suffix = '_Neutron',
	offset = 0.12,
	)
	# add some extra info
	upgradeWFs['Neutron'].neutronKeys = [x for x in upgradeKeys[2026] if 'PU' not in x]
	upgradeWFs['Neutron'].neutronFrags = ['ZMM_14','MinBias_14TeV']
	class UpgradeWorkflow_heCollapse(UpgradeWorkflow):
	def setup_(self, step, stepName, stepDict, k, properties):
	stepDict[stepName][k] = merge([{'--procModifiers': 'run2_HECollapse_2018'}, stepDict[step][k]])
	def condition(self, fragment, stepList, key, hasHarvest):
	return fragment=="TTbar_13" and '2018' in key
	upgradeWFs['heCollapse'] = UpgradeWorkflow_heCollapse(
	steps = [
	'GenSim',
	'Digi',
	'Reco',
	'HARVEST',
	'ALCA',
	],
	PU = [
	'Digi',
	'Reco',
	'HARVEST',
	],
	suffix = '_heCollapse',
	offset = 0.6,
	)
	class UpgradeWorkflow_ecalDevel(UpgradeWorkflow):
	def setup_(self, step, stepName, stepDict, k, properties):
	# temporarily remove trigger & downstream steps
	mods = {'--era': stepDict[step][k]['--era']+',phase2_ecal_devel'}
	if 'Digi' in step:
	mods['-s'] = 'DIGI:pdigi_valid'
	stepDict[stepName][k] = merge([mods, stepDict[step][k]])
	def condition(self, fragment, stepList, key, hasHarvest):
	return fragment=="TTbar_14TeV" and '2026' in key
	upgradeWFs['ecalDevel'] = UpgradeWorkflow_ecalDevel(
	steps = [
	'DigiTrigger',
	'RecoGlobal',
	'HARVESTGlobal',
	],
	PU = [
	'DigiTrigger',
	'RecoGlobal',
	'HARVESTGlobal',
	],
	suffix = '_ecalDevel',
	offset = 0.61,
	)
	class UpgradeWorkflow_0T(UpgradeWorkflow):
	def setup_(self, step, stepName, stepDict, k, properties):
	myGT=stepDict[step][k]['--conditions']
	myGT+="_0T"
	stepDict[stepName][k] = merge([{'-n':'1','--magField':'0T','--conditions':myGT}, stepDict[step][k]])
	def condition(self, fragment, stepList, key, hasHarvest):
	return (fragment=="TTbar_13" or fragment=="TTbar_14TeV") and ('2017' in key or '2018' in key or '2021' in key)
	upgradeWFs['0T'] = UpgradeWorkflow_0T(
	steps = [
	'GenSim',
	'Digi',
	'Reco',
	'HARVEST',
	'ALCA',
	],
	PU = [
	'Digi',
	'Reco',
	'HARVEST',
	],
	suffix = '_0T',
	offset = 0.24,
	)
	class UpgradeWorkflow_ParkingBPH(UpgradeWorkflow):
	def setup_(self, step, stepName, stepDict, k, properties):
	if 'Reco' in step and 'Run2_2018' in stepDict[step][k]['--era']:
	stepDict[stepName][k] = merge([{'--era': 'Run2_2018,bParking'}, stepDict[step][k]])
	def condition(self, fragment, stepList, key, hasHarvest):
	return fragment=="TTbar_13" and '2018' in key
	upgradeWFs['ParkingBPH'] = UpgradeWorkflow_ParkingBPH(
	steps = [
	'Reco',
	],
	PU = [],
	suffix = '_ParkingBPH',
	offset = 0.8,
	)
	class UpgradeWorkflow_JMENano(UpgradeWorkflow):
	def setup_(self, step, stepName, stepDict, k, properties):
	if 'Nano' in step:
	stepDict[stepName][k] = merge([{'--customise': 'PhysicsTools/NanoAOD/custom_jme_cff.PrepJMECustomNanoAOD_MC'}, stepDict[step][k]])
	def condition(self, fragment, stepList, key, hasHarvest):
	return fragment=="TTbar_13" and ('2017' in key or '2018' in key)
	upgradeWFs['JMENano'] = UpgradeWorkflow_JMENano(
	steps = [
	'Nano',
	],
	PU = [],
	suffix = '_JMENano',
	offset = 0.15,
	)
	# common operations for aging workflows
	class UpgradeWorkflowAging(UpgradeWorkflow):
	def setup_(self, step, stepName, stepDict, k, properties):
	if 'Digi' in step or 'Reco' in step:
	stepDict[stepName][k] = merge([{'--customise': 'SLHCUpgradeSimulations/Configuration/aging.customise_aging_'+self.lumi}, stepDict[step][k]])
	def condition(self, fragment, stepList, key, hasHarvest):
	return fragment=="TTbar_14TeV" and '2026' in key
	# define several of them
	upgradeWFs['Aging1000'] = UpgradeWorkflowAging(
	steps = [
	'Digi',
	'DigiTrigger',
	'RecoLocal',
	'Reco',
	'RecoGlobal',
	],
	PU = [
	'Digi',
	'DigiTrigger',
	'RecoLocal',
	'Reco',
	'RecoGlobal',
	],
	suffix = 'Aging1000',
	offset = 0.101,
	)
	upgradeWFs['Aging1000'].lumi = '1000'
	upgradeWFs['Aging3000'] = deepcopy(upgradeWFs['Aging1000'])
	upgradeWFs['Aging3000'].suffix = 'Aging3000'
	upgradeWFs['Aging3000'].offset = 0.103
	upgradeWFs['Aging3000'].lumi = '3000'
	# Specifying explicitly the --filein is not nice but that was the
	# easiest way to "skip" the output of step2 (=premixing stage1) for
	# filein (as it goes to pileup_input). It works (a bit accidentally
	# though) also for "-i all" because in that case the --filein for DAS
	# input is after this one in the list of command line arguments to
	# cmsDriver, and gets then used in practice.
	digiPremixLocalPileup = {
	"--filein": "file:step1.root",
	"--pileup_input": "file:step2.root"
	}
	# for premix
	class UpgradeWorkflowPremix(UpgradeWorkflow):
	def setup_(self, step, stepName, stepDict, k, properties):
	# just copy steps
	stepDict[stepName][k] = merge([stepDict[step][k]])
	def setupPU_(self, step, stepName, stepDict, k, properties):
	# setup for stage 1
	if "GenSim" in stepName:
	stepNamePmx = stepName.replace('GenSim','Premix')
	if not stepNamePmx in stepDict: stepDict[stepNamePmx] = {}
	stepDict[stepNamePmx][k] = merge([
	{
	'-s': 'GEN,SIM,DIGI:pdigi_valid',
	'--datatier': 'PREMIX',
	'--eventcontent': 'PREMIX',
	'--procModifiers': 'premix_stage1'
	},
	stepDict[stepName][k]
	])
	# setup for stage 2
	elif "Digi" in step or "Reco" in step:
	# go back to non-PU step version
	d = merge([stepDict[self.getStepName(step)][k]])
	if d is None: return
	if "Digi" in step:
	tmpsteps = []
	for s in d["-s"].split(","):
	if s == "DIGI" or "DIGI:" in s:
	tmpsteps.extend([s, "DATAMIX"])
	else:
	tmpsteps.append(s)
	d = merge([{"-s" : ",".join(tmpsteps),
	"--datamix" : "PreMix",
	"--procModifiers": "premix_stage2"},
	d])
	# for combined stage1+stage2
	if "_PMXS1S2" in self.suffix:
	d = merge([digiPremixLocalPileup, d])
	elif "Reco" in step:
	if "--procModifiers" in d:
	d["--procModifiers"] += ",premix_stage2"
	else:
	d["--procModifiers"] = "premix_stage2"
	stepDict[stepName][k] = d
	# Increase the input file step number by one for Nano in combined stage1+stage2
	elif "Nano" in step:
	# go back to non-PU step version
	d = merge([stepDict[self.getStepName(step)][k]])
	if "--filein" in d:
	filein = d["--filein"]
	m = re.search("step(?P<ind>\d+)_", filein)
	if m:
	d["--filein"] = filein.replace(m.group(), "step%d_"%(int(m.group("ind"))+1))
	stepDict[stepName][k] = d
	# run2/3 WFs use Nano (not NanoPU) in PU WF
	stepDict[self.getStepName(step)][k] = merge([d])
	def condition(self, fragment, stepList, key, hasHarvest):
	if not 'PU' in key:
	return False
	if not any(y in key for y in ['2021', '2023', '2024', '2026']):
	return False
	if self.suffix.endswith("S1"):
	return "NuGun" in fragment
	return True
	def workflow_(self, workflows, num, fragment, stepList, key):
	fragmentTmp = fragment
	if self.suffix.endswith("S1"):
	fragmentTmp = 'PREMIXUP' + key[2:].replace("PU", "").replace("Design", "") + '_PU25'
	super(UpgradeWorkflowPremix,self).workflow_(workflows, num, fragmentTmp, stepList, key)
	# Premix stage1
	upgradeWFs['PMXS1'] = UpgradeWorkflowPremix(
	steps = [
	],
	PU = [
	'GenSim',
	'GenSimHLBeamSpot',
	'GenSimHLBeamSpot14',
	],
	suffix = '_PMXS1',
	offset = 0.97,
	)
	# Premix stage2
	upgradeWFs['PMXS2'] = UpgradeWorkflowPremix(
	steps = [],
	PU = [
	'Digi',
	'DigiTrigger',
	'RecoLocal',
	'Reco',
	'RecoGlobal',
	'Nano',
	],
	suffix = '_PMXS2',
	offset = 0.98,
	)
	# Premix combined stage1+stage2
	upgradeWFs['PMXS1S2'] = UpgradeWorkflowPremix(
	steps = [],
	PU = [
	'GenSim',
	'GenSimHLBeamSpot',
	'GenSimHLBeamSpot14',
	'Digi',
	'DigiTrigger',
	'RecoLocal',
	'Reco',
	'RecoGlobal',
	'Nano',
	],
	suffix = '_PMXS1S2',
	offset = 0.99,
	)
	# Alternative version of above w/ less PU for PR tests
	class UpgradeWorkflowAdjustPU(UpgradeWorkflowPremix):
	def setupPU_(self, step, stepName, stepDict, k, properties):
	# adjust first, so it gets copied into new Premix step
	if '--pileup' in stepDict[stepName][k]:
	stepDict[stepName][k]['--pileup'] = 'AVE_50_BX_25ns_m3p3'
	super(UpgradeWorkflowAdjustPU,self).setupPU_(step, stepName, stepDict, k, properties)
	def condition(self, fragment, stepList, key, hasHarvest):
	# restrict to phase2
	return super(UpgradeWorkflowAdjustPU,self).condition(fragment, stepList, key, hasHarvest) and '2026' in key
	upgradeWFs['PMXS1S2PR'] = UpgradeWorkflowAdjustPU(
	steps = [],
	PU = [
	'GenSim',
	'GenSimHLBeamSpot',
	'GenSimHLBeamSpot14',
	'Digi',
	'DigiTrigger',
	'RecoLocal',
	'Reco',
	'RecoGlobal',
	'Nano',
	'HARVEST',
	'HARVESTGlobal',
	],
	suffix = '_PMXS1S2PR',
	offset = 0.999,
	)
	class UpgradeWorkflowPremixProdLike(UpgradeWorkflowPremix,UpgradeWorkflow_ProdLike):
	def setup_(self, step, stepName, stepDict, k, properties):
	# copy steps, then apply specializations
	UpgradeWorkflowPremix.setup_(self, step, stepName, stepDict, k, properties)
	UpgradeWorkflow_ProdLike.setup_(self, step, stepName, stepDict, k, properties)
	def condition(self, fragment, stepList, key, hasHarvest):
	# use both conditions
	return UpgradeWorkflowPremix.condition(self, fragment, stepList, key, hasHarvest) and UpgradeWorkflow_ProdLike.condition(self, fragment, stepList, key, hasHarvest)
	# premix stage2
	upgradeWFs['PMXS2ProdLike'] = UpgradeWorkflowPremixProdLike(
	steps = [],
	PU = [
	'Digi',
	'DigiTrigger',
	'RecoLocal',
	'Reco',
	'RecoGlobal',
	'Nano',
	'HARVEST',
	'HARVESTGlobal',
	'MiniAOD',
	'ALCA',
	],
	suffix = '_PMXS2ProdLike',
	offset = 0.9821,
	)
	# premix combined stage1+stage2
	upgradeWFs['PMXS1S2ProdLike'] = UpgradeWorkflowPremixProdLike(
	steps = [],
	PU = [
	'GenSim',
	'GenSimHLBeamSpot',
	'GenSimHLBeamSpot14',
	'Digi',
	'DigiTrigger',
	'RecoLocal',
	'Reco',
	'RecoGlobal',
	'Nano',
	'HARVEST',
	'HARVESTGlobal',
	'MiniAOD',
	'ALCA',
	],
	suffix = '_PMXS1S2ProdLike',
	offset = 0.9921,
	)
	class UpgradeWorkflow_DD4hep(UpgradeWorkflow):
	def setup_(self, step, stepName, stepDict, k, properties):
	if 'Run3' in stepDict[step][k]['--era']:
	stepDict[stepName][k] = merge([{'--geometry': 'DD4hepExtended2021', '--procModifiers': 'dd4hep'}, stepDict[step][k]])
	elif 'Phase2' in stepDict[step][k]['--era']:
	dd4hepGeom="DD4hep"
	dd4hepGeom+=stepDict[step][k]['--geometry']
	stepDict[stepName][k] = merge([{'--geometry' : dd4hepGeom, '--procModifiers': 'dd4hep'}, stepDict[step][k]])
	def condition(self, fragment, stepList, key, hasHarvest):
	return ((fragment=='TTbar_14TeV' or fragment=='ZMM_14' or fragment=='SingleMuPt10') and '2021' in key) \
	or ((fragment=='TTbar_14TeV' or fragment=='ZMM_14' or fragment=='SingleMuPt10') and '2026' in key)
	upgradeWFs['DD4hep'] = UpgradeWorkflow_DD4hep(
	steps = [
	'GenSim',
	'GenSimHLBeamSpot',
	'GenSimHLBeamSpot14',
	'Digi',
	'DigiTrigger',
	'Reco',
	'RecoGlobal',
	'HARVEST',
	'HARVESTGlobal',
	'ALCA',
	],
	PU = [],
	suffix = '_DD4hep',
	offset = 0.911,
	)
	upgradeWFs['DD4hep'].allowReuse = False
	class UpgradeWorkflow_SonicTriton(UpgradeWorkflow):
	def setup_(self, step, stepName, stepDict, k, properties):
	stepDict[stepName][k] = merge([{'--procModifiers': 'allSonicTriton'}, stepDict[step][k]])
	def condition(self, fragment, stepList, key, hasHarvest):
	return (fragment=='TTbar_13' and '2021' in key) \
	or (fragment=='TTbar_14TeV' and '2026' in key)
	upgradeWFs['SonicTriton'] = UpgradeWorkflow_SonicTriton(
	steps = [
	'GenSim',
	'GenSimHLBeamSpot',
	'GenSimHLBeamSpot14',
	'Digi',
	'DigiTrigger',
	'Reco',
	'RecoGlobal',
	'HARVEST',
	'HARVESTGlobal',
	'ALCA',
	],
	PU = [
	'GenSim',
	'GenSimHLBeamSpot',
	'GenSimHLBeamSpot14',
	'Digi',
	'DigiTrigger',
	'Reco',
	'RecoGlobal',
	'HARVEST',
	'HARVESTGlobal',
	'ALCA',
	],
	suffix = '_SonicTriton',
	offset = 0.9001,

), do we have any rules for the offset on each class? I would like to bring back what @srimanob proposed a few weeks ago about the documentation #32869 and convention. Could you consider this?

[kpedro88]

@chayanit the offsets in this PR are just a concatenation of the offsets of the component workflows. I agree it would be good to have more documentation. As a start, we could add a README.md in Configuration/PyReleaseValidation, much as we previously did for Configuration/Geometry.

[chayanit]

@chayanit the offsets in this PR are just a concatenation of the offsets of the component workflows. I agree it would be good to have more documentation. As a start, we could add a README.md in Configuration/PyReleaseValidation, much as we previously did for Configuration/Geometry.

Cool thanks for your reply. Would you be able to help on this README?

[kpedro88]

@chayanit I will add it to my todo list.

[chayanit]

Thanks a lot @kpedro88!

[srimanob]

Hi @kpedro88 @chayanit
I've added the list of offset in README.md together with updating DD4Hep workflow in
#33087
Please feel free to comment on what you think should be mentioned. Thanks.

[chayanit]

+1

[cmsbuild]

Pull request #33037 was updated. @jordan-martins, @chayanit, @wajidalikhan, @kpedro88, @cmsbuild, @srimanob can you please check and sign again.

[kpedro88]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3c0d6d/13340/summary.html
COMMIT: 85ec5d4
CMSSW: CMSSW_11_3_X_2021-03-08-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/33037/13340/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-3c0d6d/23434.9921_TTbar_14TeV+2026D49PU_PMXS1S2ProdLike+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+PREMIX_PremixHLBeamSpot14PU+DigiTriggerPU+RecoGlobalPU+MiniAODPU
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-3c0d6d/34634.0_TTbar_14TeV+2026D76+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-3c0d6d/34834.999_TTbar_14TeV+2026D76PU_PMXS1S2PR+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+PREMIX_PremixHLBeamSpot14PU+DigiTriggerPU+RecoGlobalPU+HARVESTGlobalPU

Summary:

• No significant changes to the logs found
• Reco comparison results: 3 differences found in the comparisons
• DQMHistoTests: Total files compared: 38
• DQMHistoTests: Total histograms compared: 2849195
• DQMHistoTests: Total failures: 6
• DQMHistoTests: Total nulls: 1
• DQMHistoTests: Total successes: 2849166
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.004 KiB( 37 files compared)
• DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
• Checked 160 log files, 37 edm output root files, 38 DQM output files

[kpedro88]

@slava77 we actually need the matrix map for 34634.0 and 34834.999

[slava77]

@slava77 we actually need the matrix map for 34634.0 and 34834.999

what's the first IB with these?

[kpedro88]

It should be CMSSW_11_3_X_2021-03-04-1800, which includes #33007

[slava77]

It should be CMSSW_11_3_X_2021-03-04-1800, which includes #33007

looking back in my inbox; indeed, it's been a while. I guess I managed to filter these out mentally.
Thanks for the ping. I'll update today.

[slava77]

I'll update today.

cms-sw/cms-bot#1500

[srimanob]

+Upgrade

This PR is to add Prod-like Premixing workflow, and update README.

[chayanit]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[qliphy]

+1

[srimanob]

By the way, @kpedro88
Is this expected to use pdigi_valid for prod-like workflow?
I think we use it for L1T campaign, however, not in general. Or now, ppl needs this information in general MC campaign?

Sorry for late response. Thanks.

[kpedro88]

@srimanob the most recent TDR campaign uses pdigi_valid: https://twiki.cern.ch/twiki/bin/view/CMS/Phase2HLTTDRWinter20DR

[srimanob]

@kpedro88
It is in that way because when I constructed cmsDriver from testing, I forget to change it. Not sure if we need MC truth information in that campaign, no one complained because it is there.