Add 2021 MinimumBias express and prompt reco wf to limited

[tvami]

PR description:

Noticed that --limited doesn't test the 2021 pilot beam express and prompt reco, so this PR add that two wf to it

if this PR is a backport please specify the original PR and why you need to backport that PR:

N/A

[tvami]

@cmsbuild , please test

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36942/28282

• This PR adds an extra 16KB to repository

[cmsbuild]

A new Pull Request was created by @tvami (Tamas Vami) for master.

It involves the following packages:

• Configuration/PyReleaseValidation (pdmv, upgrade)

@jordan-martins, @bbilin, @wajidalikhan, @AdrianoDee, @srimanob, @kskovpen can you please review it and eventually sign? Thanks.
@makortel, @kpedro88, @Martin-Grunewald, @missirol, @fabiocos, @slomeo this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12d18e/22368/summary.html
COMMIT: 66f9043
CMSSW: CMSSW_12_3_X_2022-02-11-1400/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/36942/22368/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @mmusich Improved Payload Inspector plots for DropBoxMetaData #36934
• @jpata Updated MLPF producer with ONNX #36841
• @maxgalli BeamSpot + Rho variables in nanoAODs #36037

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12d18e/22368/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12d18e/22368/git-merge-result

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-12d18e/138.4_PromptCollisions+RunMinimumBias2021+ALCARECOPROMPTR3+HARVESTDPROMPTR3
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-12d18e/138.5_ExpressCollisions+RunMinimumBias2021+TIER0EXPRUN3+ALCARECOEXPR3+HARVESTDEXPR3

Summary:

• No significant changes to the logs found
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 16 differences found in the comparisons
• DQMHistoTests: Total files compared: 46
• DQMHistoTests: Total histograms compared: 3764395
• DQMHistoTests: Total failures: 8
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3764365
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 7.296 KiB( 45 files compared)
• DQMHistoSizes: changed ( 11634.0,... ): 0.912 KiB Physics/NanoAODDQM
• Checked 193 log files, 42 edm output root files, 46 DQM output files
• TriggerResults: no differences found

[tvami]

We have both 138.4 and 138.5 in the output, as expected:

• https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12d18e/22368/runTheMatrix-results/138.4_PromptCollisions+RunMinimumBias2021+ALCARECOPROMPTR3+HARVESTDPROMPTR3/
• https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12d18e/22368/runTheMatrix-results/138.5_ExpressCollisions+RunMinimumBias2021+TIER0EXPRUN3+ALCARECOEXPR3+HARVESTDEXPR3/

[srimanob]

+Upgrade

Maybe we should review a bit on the list of the short matrix, i.e. do we still need reco/express of 2011.

[francescobrivio]

@tvami maybe we should also add one wf for offline Run3 data, i.e. one from 139.001/139.002/139.003/139.004. We would have spostted the last failure with the L1CaloGeometry tag.

[tvami]

Ok I removed the 2011 data based express and added the 139.001

[tvami]

@cmsbuild , please test

[cmsbuild]

Pull request #36942 was updated. @jordan-martins, @bbilin, @wajidalikhan, @AdrianoDee, @srimanob, @kskovpen can you please check and sign again.

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12d18e/22412/summary.html
COMMIT: bd21dc8
CMSSW: CMSSW_12_3_X_2022-02-13-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36942/22412/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @perrotta Remove dead branches and clean up GlobalRecHit validation #36949
• @mariadalfonso NANOAOD: simplify configuration #36845
• @bsunanda Run3-alca218A Clean some of the AlCaReco codes of HCAL #36950
• @mmusich more esConsumes in RecoPixelVertexing #36830
• @Dr15Jones Added LZ4 as a compression option in RootOutputModule #36943
• @nurfikri89 [NanoAOD] PileupJetId speed up #36848

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12d18e/22412/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12d18e/22412/git-merge-result

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-12d18e/138.4_PromptCollisions+RunMinimumBias2021+ALCARECOPROMPTR3+HARVESTDPROMPTR3
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-12d18e/138.5_ExpressCollisions+RunMinimumBias2021+TIER0EXPRUN3+ALCARECOEXPR3+HARVESTDEXPR3
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-12d18e/139.001_RunMinimumBias2021+RunMinimumBias2021+HLTDR3_2021+RECODR3_MinBiasOffline+HARVESTD2021MB

Summary:

• No significant changes to the logs found
• Reco comparison results: 2 differences found in the comparisons
• DQMHistoTests: Total files compared: 46
• DQMHistoTests: Total histograms compared: 3764435
• DQMHistoTests: Total failures: 7
• DQMHistoTests: Total nulls: 1
• DQMHistoTests: Total successes: 3764405
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.004 KiB( 45 files compared)
• DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
• Checked 193 log files, 42 edm output root files, 46 DQM output files
• TriggerResults: no differences found

[tvami]

I'm not sure I understand how this came back with a +1
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12d18e/22412/runTheMatrix-results/139.001_RunMinimumBias2021+RunMinimumBias2021+HLTDR3_2021+RECODR3_MinBiasOffline+HARVESTD2021MB/step2_RunMinimumBias2021+RunMinimumBias2021+HLTDR3_2021+RECODR3_MinBiasOffline+HARVESTD2021MB.log
the HLT GT is used here. So this means we'd be good for collisions? Isnt this controversial to the issue #36806 ?

[tvami]

Maybe @missirol could comment about this?

[missirol]

.. The RelVal failing in the IBs is 139.004, not 139.001.

[tvami]

The RelVal failing in the IBs is 139.004, not 139.001.

Yes, and both of them use the same thing in the HLT step, namely HLTDR3_2021, meaning
https://github.com/cms-sw/cmssw/blob/master/Configuration/PyReleaseValidation/python/relval_steps.py#L1908

steps['HLTDR3_2021']=merge( [ {'-s':'L1REPACK:Full,HLT:@%s'%hltKey2021,},{'--conditions':'auto:run3_hlt'},{'--era':'Run3'},steps['HLTD'] ] )

[missirol]

My understanding is the following:

• 139.001 and 139.004 run on different (real-data) events.

• In 139.001, no event executes the problematic module, i.e. hltRechitInRegionsECAL (for all events of that wf, the relevant trigger paths return false before that module is used). This is also the case for the HLT-Validation tests in the IBs, which are currently passing for Data, despite the removal of the L1CaloGeometry record in the Data GT, because hltRechitInRegionsECAL never gets to run in those (Data) tests.

• In 139.004, there is at least one event where one path (i.e. DST_Run3_PFScoutingPixelTracking_v16) wants to run the problematic producer, because all its preceeding filters passed (in this specific case, the L1T seed of the Run3-PFScouting path). At that point, the module throws an exception due to the missing record in the GT.

• In MC, the problem does not occur (not in PR tests, nor in HLT-Validation tests in IB) because the Run-3 MC GT still provides the relevant record
  https://cms-conddb.cern.ch/cmsDbBrowser/list/Prod/gts/123X_mcRun3_2021_realistic_v6
  https://cms-conddb.cern.ch/cmsDbBrowser/list/Prod/tags/L1CaloGeometry_STARTUP_v3

[srimanob]

+Upgrade

Re-sign.

[bbilin]

+pdmv

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[perrotta]

+1