ConcurrentHadronizerFilter (106X)

[colizz]

PR description:

This PR is a backport of #28913 by @Dr15Jones and #28879 by @wddgit.
The former introduces a thread-safe method ConcurrentHadronizerFilter which runs the hadronizer concurrently. The latter implements the base method cloneEngine as the support of ConcurrentHadronizerFilter.
With the use of ConcurrentHadronizerFilter in GEN step and MadGraph multithreading support (#30030) in LHE step, the overall CPU efficiency and runtime is significantly improved, which may benefit future UL production. More detailed explanation and validation can be found in these slides.

PR validation:

This PR is validated on multiple datasets, by replacing Pythia8HadronizerFilter with Pythia8ConcurrentHadronizerFilter in the GEN step configuration.
Below is a quick test on SMP-RunIISummer19UL17wmLHEGEN-00001.

cmsrel CMSSW_10_6_X_2020-05-25-2300
cd CMSSW_10_6_X_2020-05-25-2300/src/
cmsenv
git cms-merge-topic colizz:dev-106X-ConcurrentHadronizerFilter
curl -s --insecure https://cms-pdmv.cern.ch/mcm/public/restapi/requests/get_fragment/SMP-RunIISummer19UL17wmLHEGEN-00001 --retry 2 --create-dirs -o Configuration/GenProduction/python/SMP-RunIISummer19UL17wmLHEGEN-00001-fragment.py
scram b -j 8
cd ../..
cmsDriver.py Configuration/GenProduction/python/SMP-RunIISummer19UL17wmLHEGEN-00001-fragment.py --fileout file:SMP-RunIISummer19UL17wmLHEGEN-00001.root --mc --eventcontent RAWSIM,LHE --datatier GEN,LHE --conditions 106X_mc2017_realistic_v6 --beamspot Realistic25ns13TeVEarly2017Collision --step LHE,GEN --geometry DB:Extended --era Run2_2017 --python_filename SMP-RunIISummer19UL17wmLHEGEN-00001_1_cfg.py --no_exec --customise Configuration/DataProcessing/Utils.addMonitoring --customise_commands process.source.numberEventsInLuminosityBlock="cms.untracked.uint32(2222)" -n 400 --nThreads 4

sed -i "s/Pythia8HadronizerFilter/Pythia8ConcurrentHadronizerFilter/g" SMP-RunIISummer19UL17wmLHEGEN-00001_1_cfg.py
cmsRun -e -j SMP-RunIISummer19UL17wmLHEGEN-00001_rt.xml SMP-RunIISummer19UL17wmLHEGEN-00001_1_cfg.py

[cmsbuild]

A new Pull Request was created by @colizz (Congqiao Li) for CMSSW_10_6_X.

It involves the following packages:

FWCore/Utilities
GeneratorInterface/Core
GeneratorInterface/ExternalDecays
GeneratorInterface/Pythia8Interface
IOMC/RandomEngine

@SiewYan, @smuzaffar, @Dr15Jones, @mkirsano, @makortel, @cmsbuild, @GurpreetSinghChahal, @agrohsje, @alberto-sanchez, @qliphy can you please review it and eventually sign? Thanks.
@mkirsano, @makortel, @felicepantaleo, @wddgit, @agrohsje, @alberto-sanchez, @fabiocos this is something you requested to watch as well.
@silviodonato, @dpiparo you are the release manager for this.

cms-bot commands are listed here

[colizz]

please test

[qliphy]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/6700/console Started: 2020/05/30 04:07

[cmsbuild]

+1
Tested at: 26392cb
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c63a05/6700/summary.html
CMSSW: CMSSW_10_6_X_2020-05-29-2300
SCRAM_ARCH: slc7_amd64_gcc700

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c63a05/6700/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 33
• DQMHistoTests: Total histograms compared: 3212324
• DQMHistoTests: Total failures: 33
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3211957
• DQMHistoTests: Total skipped: 334
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 32 files compared)
• Checked 137 log files, 14 edm output root files, 33 DQM output files

[qliphy]

local test at lxplus looks good to me.

[qliphy]

+1

[makortel]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_10_6_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_11_2_X is complete. This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo (and backports should be raised in the release meeting by the corresponding L2)

[silviodonato]

+1