Add customisation to make sure that upgrade WFs will cross LS in the short matrix

[srimanob]

PR description:

This purpose of this PR is to add customisation on GEN-SIM step which will force all upgrade WFs to cross LS in the short matrix. It should not affect relvals submitted by PdmV. Crossing LS in the short matrix will help us to identify rare issues that can happen during crossing LS as we spotted lately in the past.

PR validation:

Config from the following command runs fine.

runTheMatrix.py -w upgrade -l 34640.0 --wm init

if this PR is a backport please specify the original PR and why you need to backport that PR:

No backport is needed.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-32957/21195

• This PR adds an extra 44KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): Adding GEN fragment (GluGluTo2Jets_ExHume) specific for PPS wf testing #32765, Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983, Patatrack integration - Pixel workflows (12/N) #31854

[cmsbuild]

A new Pull Request was created by @srimanob (Phat Srimanobhas) for master.

It involves the following packages:

Configuration/PyReleaseValidation

@jordan-martins, @chayanit, @wajidalikhan, @kpedro88, @cmsbuild, @srimanob can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @fabiocos, @slomeo this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[srimanob]

Please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5a291b/12999/summary.html
COMMIT: bc8ebbd
CMSSW: CMSSW_11_3_X_2021-02-20-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/32957/12999/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 401 lines to the logs
• Reco comparison results: 314 differences found in the comparisons
• DQMHistoTests: Total files compared: 37
• DQMHistoTests: Total histograms compared: 2750983
• DQMHistoTests: Total failures: 3103
• DQMHistoTests: Total nulls: 178
• DQMHistoTests: Total successes: 2747680
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 4.65 KiB( 36 files compared)
• DQMHistoSizes: changed ( 10024.0,... ): 0.473 KiB Hcal/DigiRunHarvesting
• DQMHistoSizes: changed ( 23234.0,... ): 0.445 KiB Hcal/DigiRunHarvesting
• DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
• Checked 156 log files, 37 edm output root files, 37 DQM output files

[srimanob]

+Upgrade

[kpedro88]

@srimanob a few questions:

1. are all the comparison changes and logfile additions expected/understood?
2. do we really want to apply this to all workflows in the upgrade matrix? I understand the point that this won't affect RelVals because the machinery overwrites that setting anyway, but I wonder if this will have any negative consequences for people who use runTheMatrix to generate their own configs and private samples. Maybe a special workflow tested in IBs would be less disruptive.

[davidlange6]

Regarding point 2 that @kpedro88 raises - this change would also mean that the gem-sim is not so useful as a basis for any sort of performance studies.

[srimanob]

hold

To modify the PR, to run on a workflow only, just to make sure things run properly.

[cmsbuild]

Pull request has been put on hold by @srimanob
They need to issue an unhold command to remove the hold state or L1 can unhold it for all

[silviodonato]

kind reminder @srimanob

[silviodonato]

@srimanob please push on "Ready for review" when the PR will be ready

[srimanob]

Close ... to make it in a new way.