simple option to rerun failed workflows from a matrix report

[vlimant]

PR description:

with this one can do runTheMatrix.py --from-failed runall-report-step123-.log to run the workflows that failed previously

PR validation:

not much to validate, but turns out to be useful while fixing incrementally failures in the matrix.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39850/32732

• This PR adds an extra 16KB to repository

[cmsbuild]

A new Pull Request was created by @vlimant (vlimant) for master.

It involves the following packages:

• Configuration/PyReleaseValidation (pdmv, upgrade)

@bbilin, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen, @sunilUIET can you please review it and eventually sign? Thanks.
@makortel, @kpedro88, @fabiocos, @Martin-Grunewald, @missirol, @trtomei, @beaucero, @slomeo this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

[vlimant]

please test
nothing really expected to change as the option is not used anywhere just yet

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-722d0c/28512/summary.html
COMMIT: f608af1
CMSSW: CMSSW_12_6_X_2022-10-25-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39850/28512/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

There are some workflows for which there are errors in the baseline:
20834.911 step 3
4.6 step 3
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

• No significant changes to the logs found
• Reco comparison results: 11 differences found in the comparisons
• Reco comparison had 6 failed jobs
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3384029
• DQMHistoTests: Total failures: 96
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3383911
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 201 log files, 48 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[vlimant]

would @cms-sw/pdmv-l2 or @cms-sw/upgrade-l2 have feedback on this ?

[kskovpen]

+pdmv

nice addition!

[srimanob]

+Upgrade

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

[rappoccio]

+1

• Utility addition.

[rappoccio]

please merge