Add DD4hep Run 3 workflows 11624.911 and 11642.91 to PR/IB tests

[cvuosalo]

As we are in the process of completing the DD4hep migration for Run 3, it would be helpful to have DD4hep workflows running in the PR/IB tests to make sure that new PRs don't break DD4hep-related functionality. The two added workflows are:

11624.911 2021_DD4hep+TTbar_13TeV_TuneCUETP8M1_GenSim+Digi+Reco+HARVEST+ALCA
11642.911 2021_DD4hep+ZMM_13TeV_TuneCUETP8M1_GenSim+Digi+Reco+HARVEST+ALCA

At the present time, the physics results of these workflows may not be entirely correct, but they do run to completion.

These workflows test GenSim, so they do not use pre-generated samples. Changes were required to allow these workflows to perform GenSim even while the other workflows were running with recycled samples.

PR validation:

The PR tests for this PR will show if the workflows have been added correctly.

No backport is desired.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-32096/19740

• This PR adds an extra 12KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/PyReleaseValidation/python/relval_2017.py modified in PR(s): Patatrack integration - Pixel workflows (12/N) #31854, Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983

[cmsbuild]

A new Pull Request was created by @cvuosalo (Carl Vuosalo) for master.

It involves the following packages:

Configuration/PyReleaseValidation

@chayanit, @cmsbuild, @wajidalikhan, @kpedro88, @jordan-martins can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @fabiocos, @slomeo this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[cvuosalo]

@cmsbuild please test

[cmsbuild]

The tests are being triggered in jenkins.

• CMSSW_11_2_X_2020-11-10-1100/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/10630/console Started: 2020/11/11 01:44

[cmsbuild]

+1
Tested at: 08856dd
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e4bf45/10630/summary.html
CMSSW: CMSSW_11_2_X_2020-11-10-1100
SCRAM_ARCH: slc7_amd64_gcc820

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e4bf45/10630/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 35
• DQMHistoTests: Total histograms compared: 2529296
• DQMHistoTests: Total failures: 3
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2529271
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 34 files compared)
• Checked 148 log files, 22 edm output root files, 35 DQM output files

[cvuosalo]

@kpedro88 I added the workflow numbers as you instructed, but I don't see any indication in the PR test results that the new tests were run.

[kpedro88]

@cvuosalo I misread your question and my instructions only added the workflows to IBs. If you want them to run by default in PRs (rather than on demand), you should also add them to:

cmssw/Configuration/PyReleaseValidation/scripts/runTheMatrix.py

Line 58 in 7c9d085

'limited' : [5.1, #FastSim ttbar

[cmsbuild]

+1
Tested at: e6aa365
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e4bf45/10814/summary.html
CMSSW: CMSSW_11_2_X_2020-11-17-1100
SCRAM_ARCH: slc7_amd64_gcc820

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e4bf45/10814/summary.html

@slava77 comparisons for the following workflows were not done due to missing matrix map:

• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-e4bf45/11624.911_TTbar_13+2021_DD4hep+TTbar_13TeV_TuneCUETP8M1_GenSim+Digi+Reco+HARVEST+ALCA
• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-e4bf45/11642.911_ZMM_13+2021_DD4hep+ZMM_13TeV_TuneCUETP8M1_GenSim+Digi+Reco+HARVEST+ALCA

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 2 differences found in the comparisons
• DQMHistoTests: Total files compared: 35
• DQMHistoTests: Total histograms compared: 2529296
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2529273
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 34 files compared)
• Checked 148 log files, 22 edm output root files, 35 DQM output files

[silviodonato]

Thanks. @cms-sw/upgrade-l2 @cms-sw/pdmv-l2 do you have comments? I tend to agree with Fabio about having only one additional workflow in the limited matrix (ie. keep only ttbar). Of course we can add more workflows in the extended matrix.

[kpedro88]

+upgrade

[chayanit]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[silviodonato]

+1

[silviodonato]

@cvuosalo the new workflows are crashing. See #32181