Scripts to Test MC 22-23 Production (UL-like) in IBs

[AdrianoDee]

This PR proposes the addition of a set of scripts to be used as IB tests to show the desired behavior of the next 2022-2023 MC campaign and to allow experts to have this run in all the IBs to follow issues and eventually find solutions. The schema for the campaign should be the following:

2022 campaign

• GEN-SIM-RAW-DIGI steps run in 14_0_X;
• HLT step for 2022 run in 12_4_X;
• all the subsequent steps run in 14_0_X;
• ideal pre EE issue conditions. The EE issue to be mimicked via a veto.

2023 campaign

• GEN-SIM-RAW-DIGI steps run in 14_0_X;
• HLT step for 2023 run in 13_0_X;
• all the subsequent steps run in 14_0_X;
• ideal pre BPix hole conditions. The BPix issue to be absorbed in the SF.

This setup poses some issues:

• forward compatibility: CMSSW does not formally support it so 12_4_X or 13_0_X being able to process 14_0_X samples is not granted.
• conditions: we need to be able to have exactly the same conditions for [140X_*preEE* and 124X_*preEE*] and [140X_*preBPix* and 130X_*preBPix*]. Possibly having the SimBeamSpotObjectsRcd in them, to be able to use the BS smearing from GT mechanism at GEN-SIM step.

To make it less cumbersome (hopefully), I've split each step in a different script and have the different setups running the same bash script with different inputs. As they are, none of the tests technically fails. This is intentional not to have some tests that will by definition break any IB. But any suggestion there is welcome. Each script runs the HLT step in a specific target release and all the steps in the current release.

Issue (?): as they are these tests take O(100s) each (so x4).

We would need a back port to 14_0_X to have it tested there.

FYI: @cms-sw/alca-l2, @cms-sw/core-l2

[cmsbuild]

cms-bot internal usage

[AdrianoDee]

assign core
(I would ask for core signature on this)

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44578/39736

• This PR adds an extra 20KB to repository

[cmsbuild]

New categories assigned: core

@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

A new Pull Request was created by @AdrianoDee for master.

It involves the following packages:

• Configuration/PyReleaseValidation (upgrade, pdmv)

@AdrianoDee, @Dr15Jones, @smuzaffar, @sunilUIET, @makortel, @miquork, @cmsbuild, @subirsarkar, @srimanob can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @missirol, @makortel, @fabiocos, @slomeo this is something you requested to watch as well.
@antoniovilela, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[AdrianoDee]

please test

[mmusich]

a couple of typos (also missing newlines since there was recently a campaign to get rid of them).

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44578/39738

• This PR adds an extra 16KB to repository

[cmsbuild]

Pull request #44578 was updated. @Dr15Jones, @AdrianoDee, @makortel, @smuzaffar, @cmsbuild, @srimanob, @subirsarkar, @sunilUIET, @miquork can you please check and sign again.

[AdrianoDee]

please test

[makortel]

• went for the symbolic GT for all the steps but the HLT. This poses some doubts on how compatible are the 2023/2022 GTs between master/140X and the target HLT release. But at the moment I have not solution for this. And most probably some action would be needed by AlCa.

Given the discussion and outcome from last Friday XC, it seems to me these tests would not really be needed in master (as the only use case would be in 14_0_X). So perhaps we keep them in the master e.g. until the setup starts working, and then remove them (i.e. leave the setup to be tested only in 14_0_X)? (or am I jumping to that conclusion too quickly?)

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-10f471/39158/summary.html
COMMIT: 94e6d61
CMSSW: CMSSW_14_1_X_2024-04-29-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44578/39158/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

There are some workflows for which there are errors in the baseline:
24834.78 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

• You potentially removed 75 lines from the logs
• Reco comparison results: 48 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3326929
• DQMHistoTests: Total failures: 3
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3326906
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 202 log files, 165 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[srimanob]

• went for the symbolic GT for all the steps but the HLT. This poses some doubts on how compatible are the 2023/2022 GTs between master/140X and the target HLT release. But at the moment I have not solution for this. And most probably some action would be needed by AlCa.

Given the discussion and outcome from last Friday XC, it seems to me these tests would not really be needed in master (as the only use case would be in 14_0_X). So perhaps we keep them in the master e.g. until the setup starts working, and then remove them (i.e. leave the setup to be tested only in 14_0_X)? (or am I jumping to that conclusion too quickly?)

I would say we keep the test as we can't predict what will happen :)
Is there a way to keep, but disable the test to run? This is, for example, the test starts to fell in future. Thx.

[srimanob]

+Upgrade

[AdrianoDee]

We could simply keep the machinery, the scripts for future but remove the tests for master (namely commenting the BuildFile) since now we know they work as expected. Then in the back-port to 14_0_X I'd turn them back on.

[makortel]

+core

(although I slightly disagree with keeping around commented-out tests, as they could pretty much as easily be resurrected from the history; but that is beyond this PR)

[AdrianoDee]

+pdmv

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @rappoccio, @sextonkennedy, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)

[AdrianoDee]

(although I slightly disagree with keeping around commented-out tests, as they could pretty much as easily be resurrected from the history; but that is beyond this PR)

A note: for the moment I havent' commented anything. We can still go and remove them once they are no more useful.

[rappoccio]

+1

• My 0.02CHF: Let's keep the tests around because people may want private tests with 14_1 and we may also need to adjust the strategy if we find any errors with 14_0 reconstruction down the line.