Fix runTheMatrix interactive and add tests.

[rovere]

PR description:

Recent addition of tests to runTheMatrix caused a crash while running it interactively. This PR fixes the crash.
An additional test has also been added to periodically check the validity of the --interactive option of runTheMatrix.

PR validation:

runTheMatrix still works.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40596/33850

• This PR adds an extra 20KB to repository

[cmsbuild]

A new Pull Request was created by @rovere (Marco Rovere) for master.

It involves the following packages:

• Configuration/PyReleaseValidation (pdmv, upgrade)

@bbilin, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen, @sunilUIET can you please review it and eventually sign? Thanks.
@makortel, @kpedro88, @fabiocos, @Martin-Grunewald, @missirol, @trtomei, @beaucero, @slomeo this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

[rovere]

@cmsbuild please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-520e8e/30128/summary.html
COMMIT: c221e1c
CMSSW: CMSSW_13_0_X_2023-01-23-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40596/30128/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 6 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3555495
• DQMHistoTests: Total failures: 3
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3555470
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 211 log files, 162 edm output root files, 49 DQM output files
• TriggerResults: no differences found

[srimanob]

@cmsbuild please test

Re-trigger the test after a week.

[srimanob]

Hi @rovere
I see that the unit test work as expected in https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-520e8e/30128/unitTests/src/Configuration/PyReleaseValidation/test/test-runTheMatrix-interactive/testing.log
Could you please confirm?

By the way, is the test considered to be OK in terms of resource? It seems to be a long printout, but maybe it is OK.

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-520e8e/30283/summary.html
COMMIT: c221e1c
CMSSW: CMSSW_13_0_X_2023-01-31-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/40596/30283/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 3 lines to the logs
• Reco comparison results: 5 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3555495
• DQMHistoTests: Total failures: 4
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3555469
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 211 log files, 162 edm output root files, 49 DQM output files
• TriggerResults: no differences found

[rovere]

Ciao @srimanob thanks for looking into this.
Part of the printouts comes directly from runTheMatrix and, unless redirected otherwise, they'll be part of the output. They are sent to stdout, not stderr, so one cannot really disentangle them.
Apart from that, everything looks ok from my side.

[srimanob]

+Upgrade

[srimanob]

Kindly ping @cms-sw/pdmv-l2

[srimanob]

@cmsbuild please test

Refresh the test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-520e8e/30563/summary.html
COMMIT: c221e1c
CMSSW: CMSSW_13_0_X_2023-02-09-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40596/30563/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 2 lines from the logs
• Reco comparison results: 10 differences found in the comparisons
• DQMHistoTests: Total files compared: 49
• DQMHistoTests: Total histograms compared: 3555852
• DQMHistoTests: Total failures: 9
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3555821
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
• Checked 213 log files, 164 edm output root files, 49 DQM output files
• TriggerResults: no differences found

[kskovpen]

+pdmv

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

[perrotta]

+1