Bug fixes to Run2_2018_pp_on_AA in EI and DQM

[mandrenguyen]

103X backport of #24958

Crashes in a replay using the Run2_2018_pp_on_AA era were identified in:
https://hypernews.cern.ch/HyperNews/CMS/get/tier0-Ops/2031/1/1/1/1.html

These crashes are not spotted in the relevant MC workflow (158), as they only occur in data validation.
They were reproduced using the following driver command, which executes the real data version.
cmsDriver.py reco --runUnscheduled --conditions auto:phase1_2018_realistic_hi -s RAW2DIGI,L1Reco,RECO,EI,VALIDATION:@standardValidation,DQM:@common+@standardDQM+@ExtraHLT --datatier AOD,DQMIO -n 2 --era Run2_2018_pp_on_AA --eventcontent AOD,DQM

This can be tested on the DIGI output from wf 158

[fabiocos]

please test workflow 144.55

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/31198/console Started: 2018/10/22 21:25

[cmsbuild]

A new Pull Request was created by @mandrenguyen (Matthew Nguyen) for CMSSW_10_3_X.

It involves the following packages:

CommonTools/ParticleFlow
DQMOffline/Configuration
DQMOffline/RecoB

@perrotta, @andrius-k, @kmaeshima, @schneiml, @monttj, @cmsbuild, @jfernan2, @slava77, @gpetruc, @arizzi can you please review it and eventually sign? Thanks.
@rappoccio, @swertz, @imarches, @ahinzmann, @smoortga, @acaudron, @HeinerTholen, @jdolen, @mverzett, @JyothsnaKomaragiri, @gkasieczka, @threus, @cbernet, @ferencek, @pvmulder, @clelange, @rociovilar this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

• Backported from Bug fixes to Run2_2018_pp_on_AA in EI and DQM #24958

[perrotta]

backport of #24958

[andrius-k]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/31204/console Started: 2018/10/22 22:39

[cmsbuild]

+1
Tested at: 8ef2c07
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24960/31204/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24960/31204/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 10166 differences found in the comparisons
• DQMHistoTests: Total files compared: 32
• DQMHistoTests: Total histograms compared: 3163099
• DQMHistoTests: Total failures: 22266
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3140636
• DQMHistoTests: Total skipped: 197
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: -0.122 KiB( 31 files compared)
• DQMHistoSizes: changed ( 250202.181 ): -0.122 KiB SiStrip/MechanicalView
• Checked 134 log files, 14 edm output root files, 32 DQM output files

[slava77]

@cmsbuild please test

there are some unrelated differences

[cmsbuild]

Pull request #24960 was updated. @perrotta, @andrius-k, @kmaeshima, @schneiml, @monttj, @cmsbuild, @jfernan2, @slava77, @gpetruc, @arizzi can you please check and sign again.

[slava77]

@cmsbuild please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/31252/console Started: 2018/10/24 17:06

[cmsbuild]

+1
Tested at: ace2705
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24960/31252/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24960/31252/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 32
• DQMHistoTests: Total histograms compared: 3163099
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 3
• DQMHistoTests: Total successes: 3162898
• DQMHistoTests: Total skipped: 197
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.136 KiB( 31 files compared)
• DQMHistoSizes: changed ( 250202.181,... ): 0.004 KiB MessageLogger/Errors
• DQMHistoSizes: changed ( 250202.181,... ): 0.004 KiB MessageLogger/Warnings
• Checked 134 log files, 14 edm output root files, 32 DQM output files

[slava77]

+1

for #24960 ace2705

• the backport from Bug fixes to Run2_2018_pp_on_AA in EI and DQM #24958 74d35af is OK (changes are the same, ignoring empty lines/whitespace differences)
• jenkins tests pass
• local test in 10_3_0 confirms that the 2018 pp_on_AA workflow without PAT works OK (and is broken without this PR)

[fabiocos]

@jfernan2 @andrius-k could you please check and sign it?

[fabiocos]

+1

[fabiocos]

merge

already signed by DQM, latest changes do not affect their part

[andrius-k]

+1

Signing for the sake of completeness.