Fix DQM Matrix relval wfs

[kskovpen]

PR description:

Addresses #36188 and #36190 from the latest IBs relevant to 136.XXX wfs

PR validation:

Tested on 136.761, 136.829, 136.886, 136.8861, 136.8862, 136.829.

if this PR is a backport please specify the original PR and why you need to backport that PR:

Not a backport.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36194/26783

• This PR adds an extra 56KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): consistent implementation of nano step for Run 3 #36167

[cmsbuild]

A new Pull Request was created by @kskovpen for master.

It involves the following packages:

• Configuration/PyReleaseValidation (pdmv, upgrade)

@jordan-martins, @bbilin, @wajidalikhan, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen can you please review it and eventually sign? Thanks.
@makortel, @kpedro88, @Martin-Grunewald, @missirol, @fabiocos, @slomeo this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[kskovpen]

please test

[perrotta]

test parameters:

• workflow = 136.829, 136.886, 136.8861, 136.8862, 136.761, 136.762, 136.783, 136.819, 136.82

[perrotta]

please test

[cmsbuild]

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-63c8d3/20648/summary.html
COMMIT: f9d732c
CMSSW: CMSSW_12_2_X_2021-11-21-0000/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36194/20648/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

• 136.761

----- Begin Fatal Exception 21-Nov-2021 14:26:20 CET-----------------------
An exception of category 'HCALDQM' occurred while
   [0] Processing  Event run: 277069 lumi: 91 event: 54448390 stream: 0
   [1] Running path 'dqmoffline_1_step'
   [2] Calling method for module TPTask/'tpTask'
Exception Message:
TPTask::Collection HcalTrigPrimDigiCollection isn't available: valHcalTriggerPrimitiveDigis 
----- End Fatal Exception -------------------------------------------------

• 136.783

----- Begin Fatal Exception 21-Nov-2021 14:26:39 CET-----------------------
An exception of category 'HCALDQM' occurred while
   [0] Processing  Event run: 297557 lumi: 74 event: 109139542 stream: 0
   [1] Running path 'dqmoffline_1_step'
   [2] Calling method for module TPTask/'tpTask'
Exception Message:
TPTask::Collection HcalTrigPrimDigiCollection isn't available: valHcalTriggerPrimitiveDigis 
----- End Fatal Exception -------------------------------------------------

• 136.762

----- Begin Fatal Exception 21-Nov-2021 14:26:40 CET-----------------------
An exception of category 'HCALDQM' occurred while
   [0] Processing  Event run: 277069 lumi: 116 event: 104263967 stream: 0
   [1] Running path 'dqmoffline_1_step'
   [2] Calling method for module TPTask/'tpTask'
Exception Message:
TPTask::Collection HcalTrigPrimDigiCollection isn't available: valHcalTriggerPrimitiveDigis 
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

• 136.82
• 136.819

[kskovpen]

These wfs fail because of the hcal trigger DQM step. @jfernan2, do you know what is the difference between the available hcal and hcal2 DQM options?

[kskovpen]

assign dqm

[cmsbuild]

New categories assigned: dqm

@jfernan2,@ahmad3213,@rvenditti,@emanueleusai,@pbo0,@pmandrik you have been requested to review this Pull request/Issue and eventually sign? Thanks

[kskovpen]

please test

[jfernan2]

These wfs fail because of the hcal trigger DQM step. @jfernan2, do you know what is the difference between the available hcal and hcal2 DQM options?

The difference is, hcal:

cmssw/DQM/HcalTasks/python/OfflineSourceSequence_pp.py

Line 24 in e7263ec

hcalOfflineSourceSequence = cms.Sequence(

vs

hcal2:

cmssw/DQMOffline/Hcal/python/HcalDQMOfflineSequence_cff.py

Line 16 in 6d2f660

HcalDQMOfflineSequence = cms.Sequence(NoiseRatesDQMOffline*RecHitsDQMOffline*AllCaloTowersDQMOffline)

The crash is due to the fact that @L1TMon should go with @hcal hand by hand. The solution you adopted with hcalOnly is perfectly fine. Thanks

[kskovpen]

These wfs fail because of the hcal trigger DQM step. @jfernan2, do you know what is the difference between the available hcal and hcal2 DQM options?

The difference is, hcal:

cmssw/DQM/HcalTasks/python/OfflineSourceSequence_pp.py

Line 24 in e7263ec

hcalOfflineSourceSequence = cms.Sequence(

vs
hcal2:

cmssw/DQMOffline/Hcal/python/HcalDQMOfflineSequence_cff.py

Line 16 in 6d2f660

HcalDQMOfflineSequence = cms.Sequence(NoiseRatesDQMOffline*RecHitsDQMOffline*AllCaloTowersDQMOffline)

The crash is due to the fact that @L1TMon should go with @hcal hand by hand. The solution you adopted with hcalOnly is perfectly fine. Thanks

Thanks for clarifying @jfernan2 !

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-63c8d3/20651/summary.html
COMMIT: 4027112
CMSSW: CMSSW_12_2_X_2021-11-21-0000/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36194/20651/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-63c8d3/136.82_RunMET2017E+RunMET2017E+HLTDR2_2017+RECODR2_2017reHLT_skimMET_Prompt+HARVEST2017_skimMET
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-63c8d3/136.8861_RunEGamma2018Dml1+RunEGamma2018Dml1+HLTDR2_2018ml+RECODR2_2018reHLT_skimEGamma_Prompt_L1TEgDQM+HARVEST2018_L1TEgDQM_Prompt
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-63c8d3/136.8862_RunEGamma2018Dml2+RunEGamma2018Dml2+HLTDR2_2018ml+RECODR2_2018reHLT_skimEGamma_Prompt_L1TEgDQM+HARVEST2018_L1TEgDQM_Prompt

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 42
• DQMHistoTests: Total histograms compared: 3247021
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3246999
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 41 files compared)
• Checked 177 log files, 37 edm output root files, 42 DQM output files
• TriggerResults: no differences found

[qliphy]

type bugfix

[kskovpen]

+1

[jfernan2]

+1

[perrotta]

Thank you @kskovpen for having provided the fix.

It is not clear to me why a @cms-sw/upgrade-l2 signature is needed here... In any case @AdrianoDee, @srimanob if you have any comment please post them, otherwise we'll try to merge anyhow in time for the 1100 IBs, so that the RelVal errors can be cleaned in the IBs.

[srimanob]

+Upgrade

Bugfix on DQM & Harvesting sequence of relvals.

[perrotta]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.