Revive of reverted #37852 with a fix on the issue #38044

[quark2]

PR description:

This PR revives PR #37852 which is reverted because of issue #38044. The issue has been fixed, replacing part of calculations with a massive loop by lighter codes.

Also, this PR contains a backport of #37995 which fixes issue #37906.

Since the issue happened on CMSSW_12_3_X, a PR to this branch is provided first.

PR validation:

Test are done and one can check again by runTheMatrix workflows

@jshlee @watson-ij

[cmsbuild]

A new Pull Request was created by @quark2 for CMSSW_12_3_X.

It involves the following packages:

• DQM/GEM (dqm)
• Validation/Configuration (dqm, simulation)

@civanch, @emanueleusai, @mdhildreth, @cmsbuild, @jfernan2, @ahmad3213, @pmandrik, @micsucmed, @rvenditti can you please review it and eventually sign? Thanks.
@apsallid, @ebrondol, @rovere, @watson-ij, @lecriste this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[perrotta]

please @quark2 submit the fix to the master first.
Once merged there, and fully tested, we can backport the fix alone to 12_4_X, and the whole PR to 12_3_X

[quark2]

Hi @perrotta,

Okay, I had no idea about the correct order to make PRs. I'll make the PR soon.

[emanueleusai]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-369f5b/25167/summary.html
COMMIT: bdf158a
CMSSW: CMSSW_12_3_X_2022-06-01-1200/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38100/25167/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 178362 lines to the logs
• Reco comparison results: 21 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3624629
• DQMHistoTests: Total failures: 37
• DQMHistoTests: Total nulls: 1
• DQMHistoTests: Total successes: 3624569
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 65153.672 KiB( 48 files compared)
• DQMHistoSizes: changed ( 10024.0,... ): 147.714 KiB GEM/Digis
• DQMHistoSizes: changed ( 10024.0,... ): -18.914 KiB GEM/RecHits
• DQMHistoSizes: changed ( 11634.0,... ): 2170.807 KiB GEM/Digis
• DQMHistoSizes: changed ( 11634.0,... ): 11.172 KiB GEM/RecHits
• DQMHistoSizes: changed ( 11634.0,... ): 0.770 KiB GEM/EventInfo
• DQMHistoSizes: changed ( 23234.0,... ): 4332.684 KiB GEM/Digis
• DQMHistoSizes: changed ( 23234.0,... ): -5.781 KiB GEM/RecHits
• DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
• DQMHistoSizes: changed ( 35034.0,... ): 6048.131 KiB GEM/Digis
• DQMHistoSizes: changed ( 35034.0,... ): -13.969 KiB GEM/RecHits
• Checked 205 log files, 45 edm output root files, 49 DQM output files
• TriggerResults: no differences found

[emanueleusai]

+1

[civanch]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_3_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_5_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[qliphy]

@quark2 Please update in this backport PR similarly as done in dad56f5

[cmsbuild]

Pull request #38100 was updated. @civanch, @emanueleusai, @mdhildreth, @cmsbuild, @jfernan2, @ahmad3213, @pmandrik, @micsucmed, @rvenditti can you please check and sign again.

[quark2]

Hi @qliphy,

Yep, sure. And also I added the idea of Andrea, which wasn't equipped yet.

[qliphy]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-369f5b/25345/summary.html
COMMIT: eb11212
CMSSW: CMSSW_12_3_X_2022-06-07-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/38100/25345/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 126433 lines to the logs
• Reco comparison results: 19 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3626219
• DQMHistoTests: Total failures: 22
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3626175
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 65153.668 KiB( 48 files compared)
• DQMHistoSizes: changed ( 10024.0,... ): 147.714 KiB GEM/Digis
• DQMHistoSizes: changed ( 10024.0,... ): -18.914 KiB GEM/RecHits
• DQMHistoSizes: changed ( 11634.0,... ): 2170.807 KiB GEM/Digis
• DQMHistoSizes: changed ( 11634.0,... ): 11.172 KiB GEM/RecHits
• DQMHistoSizes: changed ( 11634.0,... ): 0.770 KiB GEM/EventInfo
• DQMHistoSizes: changed ( 23234.0,... ): 4332.684 KiB GEM/Digis
• DQMHistoSizes: changed ( 23234.0,... ): -5.781 KiB GEM/RecHits
• DQMHistoSizes: changed ( 35034.0,... ): 6048.131 KiB GEM/Digis
• DQMHistoSizes: changed ( 35034.0,... ): -13.969 KiB GEM/RecHits
• Checked 205 log files, 45 edm output root files, 49 DQM output files
• TriggerResults: no differences found

[civanch]

+1

[emanueleusai]

testing at P5

[qliphy]

testing at P5

@emanueleusai Any news on the P5 test? Thanks.

[emanueleusai]

tests completed successfully. Processing time is similar to what we observed before.

[emanueleusai]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_3_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_5_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[qliphy]

+1