[12_0_X] Backport concurrent GEN utilities

[colizz]

PR description:

The concurrent GEN utilities are also be backported to 12_0_X, to benefit some samples produced by POG in the Run 3 condition. (similar to the 10_6_X backport #34527)

It includes two PRs backported from the master branch:

• Enable concurrent GEN in Configuration/Generator/python #34710
• Fix the random segfault of RelVal workflow 573 #35072

PR validation:

No local test done. The PR should be well validated by triggering 'please test' (running on all workflows)

[cmsbuild]

A new Pull Request was created by @colizz (Congqiao Li) for CMSSW_12_0_X.

It involves the following packages:

• Configuration/Generator (generators)
• Configuration/PyReleaseValidation (pdmv, upgrade)

@SiewYan, @mkirsano, @jordan-martins, @chayanit, @bbilin, @wajidalikhan, @Saptaparna, @cmsbuild, @AdrianoDee, @srimanob, @agrohsje, @kpedro88, @kskovpen, @alberto-sanchez, @GurpreetSinghChahal can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @makortel, @fabiocos, @slomeo this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[AdrianoDee]

please test

[AdrianoDee]

test parameters:

• workflows = 573

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9383af/18142/summary.html
COMMIT: e0dff19
CMSSW: CMSSW_12_0_X_2021-08-30-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/35075/18142/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 39
• DQMHistoTests: Total histograms compared: 2998564
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2998542
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
• Checked 165 log files, 37 edm output root files, 39 DQM output files
• TriggerResults: no differences found

[qliphy]

test parameters:

• enable_test = threading
• workflow_threading = 570,571,572,573,574,575,576,577,562.0,537.0,1361.17,1361.0,1310.0,560.0
• workflows = 570,571,572,573,574,575,576,577,562.0,537.0,1361.17,1361.0,1310.0,560.0

[qliphy]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9383af/18151/summary.html
COMMIT: e0dff19
CMSSW: CMSSW_12_0_X_2021-08-30-2300/slc7_amd64_gcc900
Additional Tests: THREADING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/35075/18151/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-9383af/1310.0_ADDMonoJet_d3MD3_13+ADDMonoJet_d3MD3_13+DIGIUP15+RECOUP15+HARVESTUP15
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-9383af/1361.0_GluGluHToZZTo4L_M125_Pow_py8_Evt_13+GluGluHToZZTo4L_M125_Pow_py8_Evt_13+DIGIUP15+RECOUP15+HARVESTUP15
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-9383af/1361.17_GluGluHToZZTo4L_M125_Pow_py8_Evt_13UP17+GluGluHToZZTo4L_M125_Pow_py8_Evt_13UP17+DIGIUP17+RECOUP17+HARVESTUP17+NANOUP17
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-9383af/537.0_DYToLL012Jets_5FS_TuneCH3_13TeV_amcatnloFxFx_herwig7+DYToLL012Jets_5FS_TuneCH3_13TeV_amcatnloFxFx_herwig7+HARVESTGEN
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-9383af/560.0_ZprimeToll_M3000_13TeV_pythia8+ZprimeToll_M3000_13TeV_pythia8+HARVESTGEN
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-9383af/562.0_BulkG_ZZ_2L2Q_M1200_narrow_13TeV_pythia8+BulkG_M1200_narrow_2L2Q_LHE_13TeV+Hadronizer_TuneCUETP8M1_Mad_pythia8+HARVESTGEN2
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-9383af/570.0_BuToKstarJPsiToMuMu_forSTEAM_13TeV_ExtGen+BuToKstarJPsiToMuMu_forSTEAM_13TeV_ExtGen+HARVESTGEN
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-9383af/571.0_BsToMuMu_forSTEAM_13TeV_ExtGen+BsToMuMu_forSTEAM_13TeV_ExtGen+HARVESTGEN
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-9383af/572.0_ZTTFS_ExtGen+ZTTFS_ExtGen+HARVESTGEN
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-9383af/573.0_sherpa_ttbar_2j_MENLOPS_13TeV_MASTER_ExtGen+sherpa_ttbar_2j_MENLOPS_13TeV_MASTER_ExtGen+HARVESTGEN
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-9383af/574.0_HydjetQ_B12_5020GeV_2018_ExtGen+HydjetQ_B12_5020GeV_2018_ExtGen+HARVESTGEN
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-9383af/575.0_AMPT_PPb_5020GeV_MinimumBias_ExtGen+AMPT_PPb_5020GeV_MinimumBias_ExtGen+HARVESTGEN
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-9383af/576.0_EPOS_PPb_8160GeV_MinimumBias_ExtGen+EPOS_PPb_8160GeV_MinimumBias_ExtGen+HARVESTGEN
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-9383af/577.0_Pyquen_ZeemumuJets_pt10_2760GeV_ExtGen+Pyquen_ZeemumuJets_pt10_2760GeV_ExtGen+HARVESTGEN

Summary:

• No significant changes to the logs found
• Reco comparison results: 2 differences found in the comparisons
• DQMHistoTests: Total files compared: 39
• DQMHistoTests: Total histograms compared: 2998564
• DQMHistoTests: Total failures: 6
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2998536
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
• Checked 165 log files, 37 edm output root files, 39 DQM output files
• TriggerResults: no differences found

[agrohsje]

+generators

[jordan-martins]

+1

[qliphy]

ping @cms-sw/upgrade-l2 Do you have any comment?

[srimanob]

+Upgrade

This PR is a backport of GEN concurrent utilities.

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_0_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_1_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[qliphy]

+1