Created GlobalEvFOutputModule

[Dr15Jones]

PR description:

• The GlobalEvFOutputModule creates the same file format as EvFOutputModule but allows concurrent processing of the different cmsRun streams. This allows substantially better thread scaling.
• To make it easier to reason about the change, some code in EventFilter/Utilities had its const correctness improved.

PR validation:

Code compiles and a locally run test program runs fine.

No checks were made on the contents of the created file.

resolves cms-sw/framework-team#190

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34621/24205

• This PR adds an extra 32KB to repository

[cmsbuild]

A new Pull Request was created by @Dr15Jones (Chris Jones) for master.

It involves the following packages:

• EventFilter/Utilities (daq, reconstruction)

@perrotta, @emeschi, @cmsbuild, @slava77, @jpata, @smorovic can you please review it and eventually sign? Thanks.
@Martin-Grunewald this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy, @perrotta you are the release manager for this.

cms-bot commands are listed here

[Dr15Jones]

please test

[Dr15Jones]

Here is the comparison results for event throughput as a function of number of threads (where number of concurrent streams == number of threads). The job was a combination of the data creation from

https://github.com/cms-sw/cmssw/blob/master/EventFilter/Utilities/test/startBU.py

with the path filtering and file writing from

https://github.com/cms-sw/cmssw/blob/master/EventFilter/Utilities/test/startFU.py

Different OutputModules were substituted for each run of the test.

• The use of AsciiOutputModule shows the highest possible throughput scaling of the configuration as the module allows all streams to run independently and the module itself does nearly no work each event.
• The job with the regular EvFOutputModule shows the least amount of scaling which is expected for a one:: module.
• Two different implementation of GlobalEvFOutputModule are shown
  • the with locks variant used a blocking mutex at the point where the module needs to write to the output file
  • the external work variant uses the ExternalWork ability to run object serialization and compression in parallel and then use a edm::SerialTaskQueue to guarantee that only one stream at a time can write to the file. This variant requires more infrastructure but is non blocking and will scale better than the locking variant when writing is slow.

[cmsbuild]

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-32a7f8/17202/summary.html
COMMIT: 010333b
CMSSW: CMSSW_12_0_X_2021-07-26-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/34621/17202/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

• 11601.011601.0_SingleElectronPt10+2021+SingleElectronPt10_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA/step2_SingleElectronPt10+2021+SingleElectronPt10_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA.log
• 11602.011602.0_SingleElectronPt35+2021+SingleElectronPt35_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA/step2_SingleElectronPt35+2021+SingleElectronPt35_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA.log
• 11603.011603.0_SingleElectronPt1000+2021+SingleElectronPt1000_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA/step2_SingleElectronPt1000+2021+SingleElectronPt1000_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA.log

Expand to see more relval errors ...

• 11634.501
• 11604.0
• 11605.0
• 11606.0
• 11607.0
• 11608.0
• 11609.0
• 11630.0
• 11634.0
• 11634.1
• 11634.5
• 11634.24
• 11634.505
• 11650.505
• 11634.511
• 11634.521
• 11634.7
• 11634.91
• 11640.0
• 11643.0
• 11646.0
• 11650.0
• 11650.501
• 11723.17
• 11725.0
• 11834.0
• 11834.13
• 11834.24
• 11846.0
• 11925.0
• 12034.0
• 12434.0
• 12634.0
• 12834.0
• 13034.0
• 23434.21
• 23434.99
• 23434.9921
• 34834.9921
• 34834.21

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 3 differences found in the comparisons
• DQMHistoTests: Total files compared: 39
• DQMHistoTests: Total histograms compared: 2998564
• DQMHistoTests: Total failures: 6
• DQMHistoTests: Total nulls: 1
• DQMHistoTests: Total successes: 2998535
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.004 KiB( 38 files compared)
• DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
• Checked 165 log files, 37 edm output root files, 39 DQM output files
• TriggerResults: no differences found

[Dr15Jones]

This code change can not be the cause of the PR test problems as the touched classes are not even USED in any RelVals.

[smorovic]

+1

[slava77]

+reconstruction

for #34621 010333b

• code changes in EventFilter/Utilities are in files not related to reco
  • as we discussed with @jpata , it's worth to reconsider if this package still needs to be in reco category; it is in only via the TcdsRawToDigi
• jenkins tests pass and comparisons with the baseline show no (relevant) differences

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (but tests are reportedly failing). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy, @perrotta (and backports should be raised in the release meeting by the corresponding L2)

[qliphy]

merge