Enable concurrent GEN in Configuration/Generator/python #34710

colizz · 2021-07-31T07:23:34Z

PR description:

Enable concurrent GEN configurations in the cfi/cff file in Configuration/Generator/python.
These concurrent GEN utilities improve the multithreading performance, and help to solve issue (for GEN modules): #25090 (comment).

As previously discussed with GEN and PdmV, the change is base on four rules documented in https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookGenMultithread#TL_DR, with details as follows.

Search over all fragments in Configuration/Generator/python ..

for all the case matches twiki rule#1 (ExtenalLHEProducer), directly modify the fragment. Note that a subset of these fragments used for ReVal (relval_generator + relval_extendedgen) workflows is already modified in a previous PR implement concurrent ExternalLHEProducer technique #33516
for all the case matches twiki rule#2 (PythiaHadronizerFilter), directly modify the fragment. Note that all fragments satisfying this rule are already modified in PR Use Pythia8ConcurrentHadronizerFilter when possible #29305
for all the case matches twiki rule#3 (PythiaGeneratorFilter), directly modify the fragment.
pick eight existing workflows running different types of GeneratorFilter (they matches twiki rule#4), to enable the use of ExternalGeneratorFilter. Eight new RelVal IDs are created for them and placed into relval_extendedgen.py.

A summary of all changed cfi/cff files are shown in the below table.

Additional change: modify the number of events to 12 for workflow 537 to fix the issue in 4-thread concurrent mode. See #34531

A backport to 10_6_X is also needed.

PR validation:

Validated on all modified/added RelVal workflows. Here is a summary

workflows 562.0,537.0,512.0,547.0,515.0,518.0,526.0,529.0,522.0,555.0,1361.17,1361.0,1361.181,1361.18,25211.18,25211.0,25211.17,514.0,523.0,551.0,1360.0,25210.0,548.0,556.0,1362.0,1362.17,1362.18,25212.0,25212.18,25212.17,1363.0,1363.17,1363.18,25213.0,25213.17,25213.18,554.0,517.0,520.0,524.0,527.0,530.0,552.0,516.0,519.0,521.0,525.0,528.0,513.0
workflows 1361.17,1361.0,1361.181,1361.18,25211.18,25211.0,25211.17,514.0,552.0,517.0,520.0,512.0,547.0,513.0,515.0,516.0,519.0,518.0,555.0,551.0,548.0,556.0,554.0,562.0,1370.0,1370.17,1370.18,25215.17,25215.18,25215.0,1362.0,1362.17,1362.18,25212.0,25212.18,25212.17,1363.0,1363.17,1363.18,25213.0,25213.17,25213.18
workflows 1310.0,10.0,505.0,546.0,1355.0,1348.0,135.7,1332.0,2017.7,2018.7,25203.0,25403.0,25403.17,25403.18,50203.0,250203.0,250203.18,250203.17,250403.0,250403.17,250403.18,500203.0,5.7,32.0,203.0,108.0,109.0,1352.0,103.0,23.0,111.0,1350.0,135.8,1301.0,1301.17,1311.0,2017.8,2018.8,10023.0,10823.0,503.0,11640.0,1.0,11.0,105.0,1340.0,158.2,1333.0,159.2,33.0,135.6,1338.0,2017.6,2018.6,25209.0,25409.0,25409.17,25409.18,10071.0,10871.0,250408.0,250408.17,250408.18,11643.0,5.6,38.0,117.0,112.0,118.0,11723.17,113.0,1303.0,1303.17,1313.0,10059.0,10859.0,11630.0,3.0,13.0,123.2,504.0,114.0,4007.0,115.0,1339.0,10026.0,10826.0,39.0,135.12,149.0,158.1,1328.0,2017.12,2018.12,159.1,28.0,116.0,123.1,4008.0,107.0,1334.0,25204.0,50204.0,250204.0,250204.17,250204.18,500204.0,34.0,204.0,133.0,1353.0,1341.0,135.11,1337.0,2017.11,2018.11,25208.0,25408.0,25408.17,25408.18,50208.0,250208.17,250208.18,250409.0,250409.17,250409.18,507.0,509.0,510.0,508.0,1324.0,25214.0,24.0,135.1,135.13,1302.17,1302.0,1302.18,1302.181,1325.0,1325.1,1325.3,1325.2,1325.4,1325.9,1325.91,2017.1,2017.13,2018.1,2018.13,139902.0,13992502.0,25202.0,25202.1,25202.2,25202.15,25402.0,25402.18,25402.17,50202.0,10024.0,10024.1,10024.2,10024.3,10024.4,10024.5,10224.0,10224.15,10424.0,10824.0,10824.1,10824.5,10824.501,10824.505,10824.511,10824.521,10824.8,10824.6,11024.15,11024.6,11024.0,11224.0,11224.6,250202.0,250202.1,250202.171,250202.118,250202.17,250202.172,250202.18,250202.181,250202.182,250202.183,250202.2,250202.3,250202.5,250202.4,250402.0,250402.1,250402.17,250402.171,250402.18,250402.181,500202.0,11634.0,11634.24,11634.1,11634.5,11634.501,11634.511,11634.505,11634.521,11634.7,11634.91,11634.911,11634.912,11834.13,11834.0,11834.24,11834.99,12434.0,12634.0,12034.0,12634.99,12834.0,13034.0,13034.99,23234.0,23234.21,23234.9,23234.103,23434.21,23434.99,23434.9921,23434.999,28234.0,31434.0,32234.0,34634.21,34634.0,34834.21,34834.9921,34834.999,35034.0,36634.0,35834.0,35434.0,36234.0,37034.0,37434.0,38234.0,37834.0,38634.0,2.0,5.1,25.0,121.0,202.0,1102.0,4000.0,4001.0,122.0,1347.0,1326.0,26.0,1327.0,27.0,506.0,1336.0,36.0,1335.0,35.0,1314.0,14.0,1354.0,561.0,158.3,135.4,1304.18,1304.181,1304.182,1329.0,1356.0,2017.4,2018.4,25200.0,25400.17,25400.0,25400.18,50200.0,10025.0,10225.0,10825.0,11025.0,250200.0,250200.1,250200.118,250200.117,250200.17,250200.18,250200.182,250400.0,250400.17,250400.18,500200.0,500200.1,159.4,11646.0,11846.0,5.4,29.0,200.0,4002.0,135.9,1330.0,2017.9,2018.9,139901.0,13992501.0,25206.0,25406.17,25406.18,25406.0,50206.0,10042.0,10842.501,10842.505,10842.0,250206.0,250206.118,250206.181,250206.182,250206.17,250206.18,250406.17,250406.18,250406.0,500206.0,159.3,11650.0,11650.501,11650.505,11650.911,30.0,4003.0,125.0,135.5,1331.0,2017.5,2018.5,25205.0,25405.0,25405.17,25405.18,50205.0,250205.0,250205.17,250205.18,250405.0,250405.17,250405.18,500205.0,31.0,119.0,205.0,1344.0,44.0,1312.0,1343.0,43.0,12.0,1345.0,45.0,560.0,106.0
workflows 570,571,572,573,574,575,576,577

Appendix: summary of all cfi/cff files in Configuration/Generator/python enabled with concurrent GEN utilities.

modified cfi/cff files for ExternalLHEProducer

Config filename	Matched ReVal ID(s)
BulkG_M1200_narrow_2L2Q_LHE_13TeV_cff	562.0
DYToLL012Jets_5FS_TuneCH3_13TeV_amcatnloFxFx_herwig7_cff	537.0
DYToll01234Jets_5f_LO_MLM_Madgraph_LHE_13TeV_cff	512.0,547.0,515.0,518.0,526.0,529.0
DYToll012Jets_5f_LO_MLM_Madgraph_LHE_13TeV_cff	522.0
DYToll012Jets_5f_NLO_FXFX_Madgraph_LHE_13TeV_cff	555.0
DarkSUSY_mH_125_mN1_60_mGammaD_10_cT_10_Madgraph_LHE_13TeV_cfi
GGHZZ4L_JHUGen_Pow_NNPDF30_LHE_13TeV_cff
GGHZZ4L_JHUGen_Pow_NNPDF30_LHE_13TeV_cfi	1361.17,1361.0,1361.181,1361.18,25211.18,25211.0,25211.17
GGToH_Pow_LHE_13TeV_cff	514.0
TTbar012Jets_5f_NLO_FXFX_Madgraph_LHE_13TeV_cff	523.0,551.0
TTbar012Jets_5f_NLO_FXFX_Madgraph_LHE_13TeV_cfi	1360.0,25210.0
TTbar_Pow_LHE_13TeV_cff	548.0,556.0
VBFHZZ4Nu_Pow_NNPDF30_LHE_13TeV_cff
VBFHZZ4Nu_Pow_NNPDF30_LHE_13TeV_cfi	1362.0,1362.17,1362.18,25212.0,25212.18,25212.17
VBFHbb_Pow_NNPDF30_LHE_13TeV_cff
VBFHbb_Pow_NNPDF30_LHE_13TeV_cfi	1363.0,1363.17,1363.18,25213.0,25213.17,25213.18
VBFToH_Pow_JHU4l_LHE_13TeV_cff	554.0
VHToH_Pow_LHE_13TeV_cff	517.0,520.0,524.0,527.0,530.0,552.0
WTolNu01234Jets_5f_LO_MLM_Madgraph_LHE_13TeV_cff	516.0,519.0,521.0,525.0,528.0
WTolNu012Jets_5f_LO_MLM_Madgraph_LHE_13TeV_cff	513.0

modified cfi/cff files for PythiaHadronizerFilter

Config filename	Matched ReVal ID(s)
DarkSUSY_mH_125_mN1_60_mGammaD_10_cT_10_Madgraph_LHE_13TeV_cfi
GGHZZ4L_JHUGen_Pow_NNPDF30_LHE_13TeV_cfi	1361.17,1361.0,1361.181,1361.18,25211.18,25211.0,25211.17
Hadronizer_MgmMatchTune4C_13TeV_madgraph_pythia8_cff
Hadronizer_MgmMatchTuneCUETP8M1_13TeV_madgraph_pythia8_cff
Hadronizer_MgmMatchTuneCUETP8M1_13TeV_madgraph_pythia8_taupinu_cff
Hadronizer_MgmMatchTuneCUETP8M1_13TeV_madgraph_pythia8_taurhonu_cff
Hadronizer_TuneCP5_13TeV_Hgg_powhegEmissionVeto_pythia8_cff	514.0
Hadronizer_TuneCP5_13TeV_Htt_powhegEmissionVeto_pythia8_cff	552.0
Hadronizer_TuneCP5_13TeV_Httpinu_powhegEmissionVeto_pythia8_cff	517.0
Hadronizer_TuneCP5_13TeV_Httrhonu_powhegEmissionVeto_pythia8_cff	520.0
Hadronizer_TuneCP5_13TeV_MLM_5f_max4j_LHE_pythia8_cff	512.0,547.0,513.0
Hadronizer_TuneCP5_13TeV_MLM_5f_max4j_LHE_pythia8_taupinu_cff	515.0,516.0
Hadronizer_TuneCP5_13TeV_MLM_5f_max4j_LHE_pythia8_taurhonu_cff	519.0,518.0
Hadronizer_TuneCP5_13TeV_aMCatNLO_FXFX_5f_max2j_max0p_LHE_pythia8_cff	555.0
Hadronizer_TuneCP5_13TeV_aMCatNLO_FXFX_5f_max2j_max1p_LHE_pythia8_cff	551.0
Hadronizer_TuneCP5_13TeV_powhegEmissionVeto2p_pythia8_cff	548.0,556.0
Hadronizer_TuneCP5_13TeV_powhegEmissionVeto_pythia8_cff	554.0
Hadronizer_TuneCUETP8M1_13TeV_Hgg_powhegEmissionVeto_pythia8_cff
Hadronizer_TuneCUETP8M1_13TeV_Htt_powhegEmissionVeto_pythia8_cff
Hadronizer_TuneCUETP8M1_13TeV_Httpinu_powhegEmissionVeto_pythia8_cff
Hadronizer_TuneCUETP8M1_13TeV_Httrhonu_powhegEmissionVeto_pythia8_cff
Hadronizer_TuneCUETP8M1_13TeV_MLM_5f_max4j_LHE_pythia8_cff
Hadronizer_TuneCUETP8M1_13TeV_MLM_5f_max4j_LHE_pythia8_taupinu_cff
Hadronizer_TuneCUETP8M1_13TeV_MLM_5f_max4j_LHE_pythia8_taurhonu_cff
Hadronizer_TuneCUETP8M1_13TeV_aMCatNLO_FXFX_5f_max1j_max1p_LHE_pythia8_cff
Hadronizer_TuneCUETP8M1_13TeV_aMCatNLO_FXFX_5f_max2j_max0p_LHE_pythia8_cff
Hadronizer_TuneCUETP8M1_13TeV_aMCatNLO_FXFX_5f_max2j_max1p_LHE_pythia8_cff
Hadronizer_TuneCUETP8M1_13TeV_generic_LHE_pythia8_cff	562.0
Hadronizer_TuneCUETP8M1_13TeV_ggHZZ4L_powhegEmissionVeto_pythia8_cff
Hadronizer_TuneCUETP8M1_13TeV_powhegEmissionVeto2p_pythia8_cff
Hadronizer_TuneCUETP8M1_13TeV_powhegEmissionVeto_2p_HToGG_M125_LHE_py8_cff	1370.0,1370.17,1370.18,25215.17,25215.18,25215.0
Hadronizer_TuneCUETP8M1_13TeV_powhegEmissionVeto_3p_HToBB_M-125_LHE_pythia8_cff
Hadronizer_TuneCUETP8M1_13TeV_powhegEmissionVeto_3p_HToZZ4nu_M-125_LHE_pythia8_cff
Hadronizer_TuneCUETP8M1_13TeV_powhegEmissionVeto_pythia8_cff
VBFHZZ4Nu_Pow_NNPDF30_LHE_13TeV_cfi	1362.0,1362.17,1362.18,25212.0,25212.18,25212.17
VBFHbb_Pow_NNPDF30_LHE_13TeV_cfi	1363.0,1363.17,1363.18,25213.0,25213.17,25213.18

modified cfi/cff files for PythiaGeneratorFilter

Config filename	Matched ReVal ID(s)
ADDMonoJet_13TeV_d3MD3_TuneCUETP8M1_cfi	1310.0
ADDMonoJet_13TeV_d3MD3_cfi
ADDMonoJet_8TeV_d3MD3_TuneCUETP8M1_cfi	10.0
DYToLL_M-50_13TeV_pythia8_cff	505.0,546.0
DYToLL_M-50_14TeV_pythia8_cff
DisplacedSUSY_stopToBottom_M_300_1000mm_TuneCUETP8M1_13TeV_pythia8_cff	1355.0
DisplacedSUSY_stopToBottom_M_800_500mm_TuneCP5_13TeV_pythia8_cff
DisplacedSUSY_stopToBottom_M_800_500mm_TuneCP5_14TeV_pythia8_cff
EtaBToJpsiJpsi_14TeV_TuneCP5_pythia8_cfi
EtaBToJpsiJpsi_forSTEAM_TuneCUEP8M1_13TeV_cfi	1348.0
GGToHtautau_13TeV_pythia8_cff
GGToHtautau_13TeV_pythia8_taupinu_cff
GGToHtautau_13TeV_pythia8_taurhonu_cff
H125GGgluonfusion_13TeV_TuneCP5_cfi
H125GGgluonfusion_13TeV_TuneCUETP8M1_cfi	135.7,1332.0,2017.7,2018.7,25203.0,25403.0,25403.17,25403.18,50203.0,250203.0,250203.18,250203.17,250403.0,250403.17,250403.18,500203.0
H125GGgluonfusion_14TeV_TuneCP5_cfi
H125GGgluonfusion_14TeV_TuneCUETP8M1_cfi
H130GGgluonfusion_14TeV_TuneCUETP8M1_cfi
H130GGgluonfusion_8TeV_TuneCUETP8M1_cfi	5.7,32.0,203.0
H165WW2L_8TeV_TuneCUETP8M1_cfi	108.0
H200ZZ4L_13TeV_TuneCUETP8M1_cfi
H200ZZ4L_8TeV_TuneCUETP8M1_cfi	109.0
HSCPstop_M_200_TuneCUETP8M1_13TeV_pythia8_cff	1352.0
InclusiveppMuX_8TeV_TuneCUETP8M1_cfi	103.0
JpsiMM_13TeV_TuneCUETP8M1_cfi
JpsiMM_14TeV_TuneCUETP8M1_cfi
JpsiMM_8TeV_TuneCUETP8M1_cfi	23.0
JpsiMM_Pt_20_inf_8TeV_TuneCUETP8M1_cfi	111.0
JpsiMuMu_Pt-8_forSTEAM_13TeV_TuneCUETP8M1_cfi	1350.0
MinBias_13TeV_pythia8_TuneCUETP8M1_cfi	135.8,1301.0,1301.17,1311.0,2017.8,2018.8,10023.0,10823.0
MinBias_13TeV_pythia8_cff	503.0
MinBias_14TeV_pythia8_TuneCP5_cfi	11640.0
MinBias_14TeV_pythia8_TuneCUETP8M1_cfi
MinBias_8TeV_pythia8_TuneCUETP8M1_cff	1.0,11.0,105.0
MinBias_TuneCUETP8M1_13TeV_pythia8_cff
PYTHIA8_PhiToMuMu_TuneCUETP8M1_13TeV_cff	1340.0
PhotonJet_Pt_10_13TeV_TuneCUETP8M1_cfi	158.2,1333.0
PhotonJet_Pt_10_14TeV_TuneCP5_cfi	159.2
PhotonJet_Pt_10_14TeV_TuneCUETP8M1_cfi
PhotonJet_Pt_10_8TeV_TuneCUETP8M1_cfi	33.0
Psi2SToJPsiPiPi_14TeV_TuneCP5_pythia8_cfi
Pythia8_PhotonJetpt15_20_10TeV_cff
QCDForPF_13TeV_TuneCUETP8M1_cfi	135.6,1338.0,2017.6,2018.6,25209.0,25409.0,25409.17,25409.18,10071.0,10871.0,250408.0,250408.17,250408.18
QCDForPF_14TeV_TuneCP5_cfi	11643.0
QCDForPF_14TeV_TuneCUETP8M1_cfi
QCDForPF_8TeV_TuneCUETP8M1_cfi	5.6,38.0
QCD_Pt-15To7000_TuneCUETP8M1_Flat_14TeV-pythia8_cff
QCD_Pt-20toInf_MuEnrichedPt15_TuneCUETP8M1_13TeV_pythia8_cff
QCD_Pt-20toInf_MuEnrichedPt15_TuneCUETP8M1_14TeV_pythia8_cff
QCD_Pt15To7000_Flat_14TeV_TuneCP5_cff
QCD_Pt20toInf_MuEnrichedPt15_14TeV_TuneCP5_cff
QCD_Pt_120_170_13TeV_TuneCUETP8M1_cfi
QCD_Pt_120_170_8TeV_TuneCUETP8M1_cfi	117.0
QCD_Pt_15_20_8TeV_TuneCUETP8M1_cfi	112.0
QCD_Pt_170_230_8TeV_TuneCUETP8M1_cfi	118.0
QCD_Pt_1800_2400_13TeV_TuneCP5_cfi
QCD_Pt_1800_2400_14TeV_TuneCP5_cfi	11723.17
QCD_Pt_20_30_8TeV_TuneCUETP8M1_cfi	113.0
QCD_Pt_3000_3500_13TeV_TuneCUETP8M1_cfi	1303.0,1303.17,1313.0,10059.0,10859.0
QCD_Pt_3000_3500_14TeV_TuneCUETP8M1_cfi	11630.0
QCD_Pt_3000_3500_8TeV_TuneCUETP8M1_cfi	3.0,13.0,123.2
QCD_Pt_30_13TeV_pythia8_cff	504.0
QCD_Pt_30_50_13TeV_TuneCUETP8M1_cfi
QCD_Pt_30_50_8TeV_TuneCUETP8M1_cfi	114.0
QCD_Pt_30_80_BCtoE_8TeV_TuneCUETP8M1_cfi	4007.0
QCD_Pt_50_80_8TeV_TuneCUETP8M1_cfi	115.0
QCD_Pt_600_800_13TeV_TuneCUETP8M1_cfi	1339.0,10026.0,10826.0
QCD_Pt_600_800_14TeV_TuneCUETP8M1_cfi
QCD_Pt_600_800_8TeV_TuneCUETP8M1_cfi	39.0
QCD_Pt_80_120_13TeV_TuneCUETP8M1_cfi	135.12,149.0,158.1,1328.0,2017.12,2018.12
QCD_Pt_80_120_14TeV_TuneCP5_cfi	159.1
QCD_Pt_80_120_14TeV_TuneCUETP8M1_cfi
QCD_Pt_80_120_8TeV_TuneCUETP8M1_cfi	28.0,116.0,123.1
QCD_Pt_80_170_BCtoE_8TeV_TuneCUETP8M1_cfi	4008.0
QQH120Inv_8TeV_TuneCUETP8M1_cfi	107.0
QQH1352T_13TeV_TuneCUETP8M1_cfi	1334.0,25204.0,50204.0,250204.0,250204.17,250204.18,500204.0
QQH1352T_8TeV_TuneCUETP8M1_cfi	34.0,204.0
QQToHToTauTau_mh125_TuneCP5_14TeV_pythia8_cfi
RS750_quarks_and_leptons_8TeV_TuneCUETP8M1_cff	133.0
RSGravitonToGammaGamma_kMpl01_M_3000_TuneCUETP8M1_13TeV_pythia8_cfi	1353.0
RSKKGluon_m3000GeV_13TeV_TuneCUETP8M1_cff	1341.0
RSKKGluon_m3000GeV_13TeV_cff
SMS-T1tttt_mGl-1500_mLSP-100_13TeV-pythia8_cfi	135.11,1337.0,2017.11,2018.11,25208.0,25408.0,25408.17,25408.18,50208.0,250208.17,250208.18,250409.0,250409.17,250409.18
SMS-T1tttt_mGl-1500_mLSP-100_TuneCP5_14TeV_pythia8_cfi
SoftQCDDiffractive_13TeV_pythia8_cff	507.0
SoftQCDelastic_13TeV_pythia8_cff	509.0
SoftQCDinelastic_13TeV_pythia8_cff	510.0
SoftQCDnonDiffractive_13TeV_pythia8_cff	508.0
TTbarLepton_13TeV_TuneCUETP8M1_cfi	1324.0,25214.0
TTbarLepton_8TeV_TuneCUETP8M1_cfi	24.0
TTbarToDilepton_mt172p5_TuneCP5_14TeV_pythia8_cfi
TTbar_13TeV_TuneCUETP8M1_cfi	135.1,135.13,1302.17,1302.0,1302.18,1302.181,1325.0,1325.1,1325.3,1325.2,1325.4,1325.9,1325.91,2017.1,2017.13,2018.1,2018.13,139902.0,13992502.0,25202.0,25202.1,25202.2,25202.15,25402.0,25402.18,25402.17,50202.0,10024.0,10024.1,10024.2,10024.3,10024.4,10024.5,10224.0,10224.15,10424.0,10824.0,10824.1,10824.5,10824.501,10824.505,10824.511,10824.521,10824.8,10824.6,11024.15,11024.6,11024.0,11224.0,11224.6,250202.0,250202.1,250202.171,250202.118,250202.17,250202.172,250202.18,250202.181,250202.182,250202.183,250202.2,250202.3,250202.5,250202.4,250402.0,250402.1,250402.17,250402.171,250402.18,250402.181,500202.0
TTbar_14TeV_TuneCP5_cfi	11634.0,11634.24,11634.1,11634.5,11634.501,11634.511,11634.505,11634.521,11634.7,11634.91,11634.911,11634.912,11834.13,11834.0,11834.24,11834.99,12434.0,12634.0,12034.0,12634.99,12834.0,13034.0,13034.99,23234.0,23234.21,23234.9,23234.103,23434.21,23434.99,23434.9921,23434.999,28234.0,31434.0,32234.0,34634.21,34634.0,34834.21,34834.9921,34834.999,35034.0,36634.0,35834.0,35434.0,36234.0,37034.0,37434.0,38234.0,37834.0,38634.0
TTbar_14TeV_TuneCUETP8M1_cfi
TTbar_8TeV_TuneCUETP8M1_cfi	2.0,5.1,25.0,121.0,202.0,1102.0,4000.0,4001.0
TauToMuMuMu_14TeV_TuneCP5_cfi
UpsMM_8TeV_TuneCUETP8M1_cfi	122.0
Upsilon1SToMuMu_forSTEAM_13TeV_TuneCUETP8M1_cfi	1347.0
Upsilon1SToMuMu_forSTEAM_14TeV_TuneCP5_cfi
VBFHZZ4Nu_TuneCP5_14TeV_pythia8_cfi
WE_13TeV_TuneCUETP8M1_cfi	1326.0
WE_14TeV_TuneCUETP8M1_cfi
WE_8TeV_TuneCUETP8M1_cfi	26.0
WM_13TeV_TuneCUETP8M1_cfi	1327.0
WM_8TeV_TuneCUETP8M1_cfi	27.0
WToLNu_13TeV_pythia8_cff	506.0
WToLNu_14TeV_TuneCP5_pythia8_cfi
WToMuNu_14TeV_TuneCP5_pythia8_cfi
Wjet_Pt_3000_3500_13TeV_TuneCUETP8M1_cfi	1336.0
Wjet_Pt_3000_3500_14TeV_TuneCUETP8M1_cfi
Wjet_Pt_3000_3500_8TeV_TuneCUETP8M1_cfi	36.0
Wjet_Pt_80_120_13TeV_TuneCUETP8M1_cfi	1335.0
Wjet_Pt_80_120_14TeV_TuneCUETP8M1_cfi
Wjet_Pt_80_120_8TeV_TuneCUETP8M1_cfi	35.0
WpM_13TeV_TuneCUETP8M1_cfi	1314.0
WpM_8TeV_TuneCUETP8M1_cfi	14.0
WprimeToENu_M-2000_TuneCUETP8M1_13TeV-pythia8_cff	1354.0
WprimeToLNu_M2000_14TeV_TuneCP5_pythia8_cfi
WprimeTolNu_M3000_13TeV_pythia8_cff	561.0
ZEEMM_13TeV_TuneCUETP8M1_cfi	158.3
ZEE_13TeV_TuneCUETP8M1_cfi	135.4,1304.18,1304.181,1304.182,1329.0,1356.0,2017.4,2018.4,25200.0,25400.17,25400.0,25400.18,50200.0,10025.0,10225.0,10825.0,11025.0,250200.0,250200.1,250200.118,250200.117,250200.17,250200.18,250200.182,250400.0,250400.17,250400.18,500200.0,500200.1
ZEE_14TeV_TuneCP5_cfi	159.4,11646.0,11846.0
ZEE_14TeV_TuneCUETP8M1_cfi
ZEE_8TeV_TuneCUETP8M1_cfi	5.4,29.0,200.0,4002.0
ZMM_13TeV_TuneCUETP8M1_cfi	135.9,1330.0,2017.9,2018.9,139901.0,13992501.0,25206.0,25406.17,25406.18,25406.0,50206.0,10042.0,10842.501,10842.505,10842.0,250206.0,250206.118,250206.181,250206.182,250206.17,250206.18,250406.17,250406.18,250406.0,500206.0
ZMM_14TeV_TuneCP5_cfi	159.3,11650.0,11650.501,11650.505,11650.911
ZMM_14TeV_TuneCUETP8M1_cfi
ZMM_8TeV_TuneCUETP8M1_cfi	30.0,4003.0
ZPrime5000JJ_8TeV_TuneCUETP8M1_cfi	125.0
ZTT_All_hadronic_13TeV_TuneCUETP8M1_cfi	135.5,1331.0,2017.5,2018.5,25205.0,25405.0,25405.17,25405.18,50205.0,250205.0,250205.17,250205.18,250405.0,250405.17,250405.18,500205.0
ZTT_All_hadronic_14TeV_TuneCP5_cfi
ZTT_All_hadronic_14TeV_TuneCUETP8M1_cfi
ZTT_All_hadronic_8TeV_TuneCUETP8M1_cfi	31.0,119.0,205.0
ZmumuJets_Pt_20_300_GEN_13TeV_TuneCUETP8M1_cfg
ZpEE_2250_13TeV_Tauola_TuneCUETP8M1_cfi
ZpEE_2250_13TeV_TuneCUETP8M1_cfi	1344.0
ZpEE_2250_8TeV_TuneCUETP8M1_cfi	44.0
ZpMM_13TeV_TuneCUETP8M1_cfi	1312.0
ZpMM_2250_13TeV_Tauola_TuneCUETP8M1_cfi
ZpMM_2250_13TeV_TuneCUETP8M1_cfi	1343.0
ZpMM_2250_8TeV_TuneCUETP8M1_cfi	43.0
ZpMM_8TeV_TuneCUETP8M1_cfi	12.0
ZpTT_1500_13TeV_TuneCUETP8M1_cfi	1345.0
ZpTT_1500_14TeV_TuneCP5_cfi
ZpTT_1500_8TeV_TuneCUETP8M1_cfi	45.0
ZpToEE_m6000_TuneCP5_14TeV_pythia8_cfi
ZpToMM_m6000_TuneCP5_14TeV_pythia8_cfi
ZprimeToll_M3000_13TeV_pythia8_cff	560.0
bJpsiX_8TeV_TuneCUETP8M1_cfi	106.0

added cfi/cff files for ExternalGeneratorFilter

Config filename	Matched ReVal ID(s)
BuToKstarJPsiToMuMu_forSTEAM_13TeV_ExtGen_cfi	570
BsToMuMu_forSTEAM_13TeV_ExtGen_cfi	571
ZTT_Tauola_OneLepton_OtherHadrons_8TeV_TuneCUETP8M1_ExtGen_cfi	572
sherpa_ttbar_2j_MENLOPS_13TeV_MASTER_ExtGen_cff	573
Hydjet_Quenched_B12_5020GeV_ExtGen_cfi	574
AMPT_PPb_5020GeV_MinimumBias_ExtGen_cfi	575
ReggeGribovPartonMC_EposLHC_4080_4080GeV_pPb_ExtGen_cfi	576
Pyquen_ZeemumuJets_pt10_2760GeV_ExtGen_cfi	577

step 1 of the concurrent GEN updating plan: enable concurrent ExternalLHEProducer for all cfi & cff in Configuration/Generator/python/

step 2 of the concurrent GEN update plan: replace Pythia8GeneratorFilter with Pythia8ConcurrentGeneratorFilter if the module does not use ExternalDecays, for all cfi & cff in Configuration/Generator/python

cmsbuild · 2021-07-31T07:30:35Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34710/24345

This PR adds an extra 212KB to repository

cmsbuild · 2021-07-31T07:30:56Z

A new Pull Request was created by @colizz (Congqiao Li) for master.

It involves the following packages:

Configuration/Generator (generators)

@SiewYan, @mkirsano, @Saptaparna, @cmsbuild, @GurpreetSinghChahal, @agrohsje, @alberto-sanchez can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @fabiocos this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy, @perrotta you are the release manager for this.

cms-bot commands are listed here

colizz · 2021-07-31T07:31:22Z

@jordan-martins FYI. The new ReVal IDs are in (4), i.e.
570,571,572,573,574,575,576,577

Please let me know if you have any comments :) I'll also backport to 10_6_X in #34527 when we agree on the change.

colizz · 2021-07-31T13:22:37Z

(forgot to push the changed file in Configuration/PyReleaseValidation/python. Now modified)

cmsbuild · 2021-07-31T13:26:33Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34710/24350

This PR adds an extra 60KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/relval_standard.py modified in PR(s): Changing Commissioning 2020 to Commissioning 2021 with new run # and dataset. #34643
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): Disable vertex smearing for workflows using CloseByParticleGun #34681, Changing Commissioning 2020 to Commissioning 2021 with new run # and dataset. #34643

cmsbuild · 2021-07-31T13:26:56Z

Pull request #34710 was updated. @SiewYan, @mkirsano, @jordan-martins, @chayanit, @bbilin, @wajidalikhan, @Saptaparna, @cmsbuild, @GurpreetSinghChahal, @srimanob, @agrohsje, @kpedro88, @kskovpen, @alberto-sanchez can you please check and sign again.

srimanob · 2021-07-31T13:35:01Z

Please test

cmsbuild · 2021-07-31T16:32:58Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3fcfb0/17388/summary.html
COMMIT: 932a8f2
CMSSW: CMSSW_12_1_X_2021-07-30-0900/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/34710/17388/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

The workflows 140.53 have different files in step1_dasquery.log than the ones found in the baseline. You may want to check and retrigger the tests if necessary. You can check it in the "files" directory in the results of the comparisons

Summary:

No significant changes to the logs found
Reco comparison results: 1301 differences found in the comparisons
DQMHistoTests: Total files compared: 39
DQMHistoTests: Total histograms compared: 2998564
DQMHistoTests: Total failures: 3678
DQMHistoTests: Total nulls: 19
DQMHistoTests: Total successes: 2994845
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 45.703 KiB( 38 files compared)
DQMHistoSizes: changed ( 140.53 ): 44.531 KiB Hcal/DigiRunHarvesting
DQMHistoSizes: changed ( 140.53 ): 1.172 KiB RPC/DCSInfo
Checked 165 log files, 37 edm output root files, 39 DQM output files
TriggerResults: no differences found

srimanob · 2021-07-31T16:57:08Z

Hi @colizz @cms-sw/generators-l2
What is the point to have many GEN only workflows across relvals_*?

srimanob · 2021-08-10T13:15:22Z

+Upgrade

This PR is to enable concurrent GEN. It includes new gen-only workflows as cases described in the PR description. Test of newly workflows and thread-enable runs fine.

Backport from 2e7e1c0 in cms-sw#34710

perrotta · 2021-08-13T07:40:34Z

@cms-sw/pdmv-l2 could you please have a look, and sign if you think so?

jordan-martins · 2021-08-13T09:31:20Z

+1

cmsbuild · 2021-08-13T09:31:40Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

qliphy · 2021-08-13T10:46:26Z

@cms-sw/pdmv-l2 @cms-sw/generators-l2 @rappoccio
As discussed at last ORP, would it be possible if @colizz can give a summary about this big PR (and its backport #34527) at one of next PPD meetings, say Aug 19th or 26th?
In the meanwhile, we can first merge this PR and test in master IB, if there is no objection?

agrohsje · 2021-08-13T12:46:55Z

hi @qliphy , indeed. 19th of august might be no pdd but then 26th.

colizz · 2021-08-13T15:55:27Z

Thanks @qliphy and @agrohsje , 26th August sounds good to me. (both times work fine)

qliphy · 2021-08-13T16:15:06Z

Thanks @qliphy and @agrohsje , 26th August sounds good to me. (both times work fine)

Thanks @colizz Perfect！ PPD @rappoccio should be able to add you to the agenda.

perrotta · 2021-08-17T21:25:26Z

+1

Merging this in the master.
Decision about a possible backport will be taken after the discussion in the alca meeting; having this merged and tested in the master IBs in the meanwhile can support such a decision.

jordan-martins · 2021-08-24T08:59:49Z

Hi @colizz, just to x-check. Are we confirmed for a presentation at this PPD (26th) to resume all of these nice efforts? Thanks!
FYI @agrohsje

qliphy · 2021-08-24T09:07:21Z

I saw it is on the agenda of this Thursday's meeting: https://indico.cern.ch/event/1067353/#10-concurrent-gen-processing

Backport of cms-sw#34710

Backport of cms-sw#34710. Note that the requested events for workflow 537 has modified prior than this PR in cms-sw#34861

colizz added 2 commits July 30, 2021 22:15

Enable concurrent ExternalLHEProducer

c4e2c85

step 1 of the concurrent GEN updating plan: enable concurrent ExternalLHEProducer for all cfi & cff in Configuration/Generator/python/

Use Pythia8ConcurrentGeneratorFilter for pure Pythia8 routines

2c7b50e

step 2 of the concurrent GEN update plan: replace Pythia8GeneratorFilter with Pythia8ConcurrentGeneratorFilter if the module does not use ExternalDecays, for all cfi & cff in Configuration/Generator/python

cmsbuild added this to the CMSSW_12_1_X milestone Jul 31, 2021

cmsbuild added code-checks-pending generators-pending orp-pending pending-signatures tests-pending labels Jul 31, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 31, 2021

colizz force-pushed the dev-121X-enableConcurrentGen branch from 4aa1ba3 to 932a8f2 Compare July 31, 2021 13:20

cmsbuild added code-checks-pending pdmv-pending upgrade-pending and removed code-checks-approved labels Jul 31, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 31, 2021

cmsbuild added tests-started and removed tests-pending labels Jul 31, 2021

cmsbuild added tests-approved and removed tests-started labels Jul 31, 2021

cmsbuild added upgrade-approved and removed upgrade-pending labels Aug 10, 2021

agrohsje mentioned this pull request Aug 10, 2021

Modules prohibiting concurrent lumis in RelVal workflows #25090

Closed

colizz added a commit to colizz/cmssw that referenced this pull request Aug 13, 2021

Change event number to 12 for workflow 537

b001a8c

Backport from 2e7e1c0 in cms-sw#34710

colizz mentioned this pull request Aug 13, 2021

[12_0_X] Backport the fix on the workflow 537 and the mergeLHE.py script #34861

Merged

cmsbuild added fully-signed pdmv-approved and removed pending-signatures pdmv-pending labels Aug 13, 2021

cmsbuild added orp-approved and removed orp-pending labels Aug 17, 2021

cmsbuild merged commit ea0b9d0 into cms-sw:master Aug 17, 2021

This was referenced Aug 18, 2021

Fix radix sort test #34929

Merged

More Eigen GPU work, with fix for CUDA 11.4 cms-externals/eigen-git-mirror#7

Merged

qliphy mentioned this pull request Aug 19, 2021

IB Failure from Worflow 537 (DYToLL012Jets_5FS_TuneCH3_13TeV_amcatnloFxFx_herwig7) #34531

Closed

smuzaffar mentioned this pull request Aug 19, 2021

IB error in workflow 4007.0: invalid PDG id: 9010221 #34586

Closed

colizz added a commit to colizz/cmssw that referenced this pull request Aug 30, 2021

Enable concurrent GEN in Configuration/Generator/python (12_0_X)

e8e5ca6

Backport of cms-sw#34710

colizz added a commit to colizz/cmssw that referenced this pull request Aug 30, 2021

Enable concurrent GEN in Configuration/Generator/python (12_0_X)

d5688d6

Backport of cms-sw#34710. Note that the requested events for workflow 537 has modified prior than this PR in cms-sw#34861

colizz mentioned this pull request Aug 30, 2021

[12_0_X] Backport concurrent GEN utilities #35075

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable concurrent GEN in Configuration/Generator/python #34710

Enable concurrent GEN in Configuration/Generator/python #34710

colizz commented Jul 31, 2021 •

edited

cmsbuild commented Jul 31, 2021

cmsbuild commented Jul 31, 2021

colizz commented Jul 31, 2021 •

edited

colizz commented Jul 31, 2021

cmsbuild commented Jul 31, 2021

cmsbuild commented Jul 31, 2021

srimanob commented Jul 31, 2021

cmsbuild commented Jul 31, 2021

srimanob commented Jul 31, 2021

srimanob commented Aug 10, 2021

perrotta commented Aug 13, 2021

jordan-martins commented Aug 13, 2021

cmsbuild commented Aug 13, 2021

qliphy commented Aug 13, 2021

agrohsje commented Aug 13, 2021

colizz commented Aug 13, 2021 •

edited

qliphy commented Aug 13, 2021

perrotta commented Aug 17, 2021

jordan-martins commented Aug 24, 2021

qliphy commented Aug 24, 2021

Enable concurrent GEN in Configuration/Generator/python #34710

Enable concurrent GEN in Configuration/Generator/python #34710

Conversation

colizz commented Jul 31, 2021 • edited

PR description:

PR validation:

cmsbuild commented Jul 31, 2021

cmsbuild commented Jul 31, 2021

colizz commented Jul 31, 2021 • edited

colizz commented Jul 31, 2021

cmsbuild commented Jul 31, 2021

cmsbuild commented Jul 31, 2021

srimanob commented Jul 31, 2021

cmsbuild commented Jul 31, 2021

Comparison Summary

srimanob commented Jul 31, 2021

srimanob commented Aug 10, 2021

perrotta commented Aug 13, 2021

jordan-martins commented Aug 13, 2021

cmsbuild commented Aug 13, 2021

qliphy commented Aug 13, 2021

agrohsje commented Aug 13, 2021

colizz commented Aug 13, 2021 • edited

qliphy commented Aug 13, 2021

perrotta commented Aug 17, 2021

jordan-martins commented Aug 24, 2021

qliphy commented Aug 24, 2021

colizz commented Jul 31, 2021 •

edited

colizz commented Jul 31, 2021 •

edited

colizz commented Aug 13, 2021 •

edited