Add workflows for 2023 and 2024. Replace 13TeV 2021 by 14TeV. #27433

zhenhu · 2019-07-03T10:17:42Z

PR description:

Add new workflows for run3 2023 and 2024 scenarios.
Replace the 13TeV 2021 workflows by 14TeV.
This PR should be merged with the AlCa PR at the same time: https://github.com/cms-sw/cmssw/compare/CMSSW_11_0_X...christopheralanwest:alca-gt-2023-2024?expand=1

PR validation:

Local validation done

cmsbuild · 2019-07-03T10:18:12Z

The code-checks are being triggered in jenkins.

cmsbuild · 2019-07-03T10:25:09Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-27433/10736

This PR adds an extra 48KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): Add bx skim and corresponding RelVal #26476, Move Data ReReco ZeroBias RelvVals to use @rerecoZeroBias DQM sequence #27415, Adding Run 330098 (2019 MWGR3) in data relvals #27341, Closeby Workflows with TICL Variations in runTheMatrix #27392
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Closeby Workflows with TICL Variations in runTheMatrix #27392, Switch HLT menus for 110X #27331

cmsbuild · 2019-07-03T10:25:33Z

A new Pull Request was created by @zhenhu (Zhen Hu) for master.

It involves the following packages:

Configuration/PyReleaseValidation

@pgunnell, @zhenhu, @prebello, @cmsbuild, @kpedro88 can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

kpedro88 · 2019-07-03T12:58:41Z

@zhenhu what is 2024? I was under the impression Run3 would end after 2023...

makortel · 2019-07-03T13:02:07Z

Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py

+
+upgradeProperties[2017]['2024PU'] = deepcopy(upgradeProperties[2017]['2024'])
+upgradeProperties[2017]['2024PU']['ScenToRun'] = ['GenSimFull','DigiFullPU','RecoFullPU','HARVESTFullPU','NanoFull']
+
 upgradeProperties[2023] = {


Will the Run 4 workflows be renamed at some point? Now there would be workflows with "2023" in their names for Run 3 and Run 4.

Yes, this will be done as part of #26938. Maybe it's better to hold this until I can implement all the cleanup there.

@kpedro88 @makortel Nominally Run 3 will end after 2023 but there are studies underway to investigate detector performance if Run 3 is extended by one year. The "2023" and "2024" scenarios correspond to simulation of the CMS detector after integrated luminosities of 450 fb^-1 and 550 fb^-1 respectively. See slide 18 of the PPD update at CMS Week

From the AlCa perspective, we would like to introduce 11_0_X queues for the 450 fb^-1 and 550 fb^-1 scenarios so that we can integrate new developments for these queues while closing the 10_6_X mcRun3_2023 and mcRun3_2024 queues to anything other than bug fixes. At the same time, we would like to rename all of the Run 3 and later GTs to standardize the naming. The year would be left off the name of the "mcRun4" scenario until a need to distinguish between the various years arises. This is the content of the autoCond update that @zhenhu referenced at the introduction to this PR.

@kpedro88, would the autoCond update by itself conflict with your goals in #26938?

Currently the autoCond key used in the Phase 2 workflows is auto:phase2_realistic. If this key is preserved (even if the value changes), it should not interfere with the upcoming Phase 2 cleanup.

@kpedro88 I think we need to agree on a global strategy before integrating bits and pieces that contradict each other

I would prefer to submit the phase 2 cleanup before this, as it has been delayed for several weeks now by other conflicting developments. Today is a holiday in the US, but I might be able to prepare it tomorrow.

mmusich · 2019-07-04T10:15:19Z

Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py

+    '2023' : {
+        'Geom' : 'DB:Extended',
+        'GT' : 'auto:phase1_2023_realistic',
+        'HLTmenu': '@relval2017',


Hi @zhenhu I have a question. Is there a particular reason to have a fake menu in these relvals? I see that recently a new tag @relval2021 pointing to a real menu has been made available in #27331

@zhenhu @mmusich indeed the idea is to use the new menu for Run3 as proposed by @Martin-Grunewald in #27331 for already existing workflows

cmsbuild · 2019-07-04T14:29:59Z

The tests are being triggered in jenkins.
Tested with other pull request(s) #27441
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/1327/console Started: 2019/07/04 16:31

cmsbuild · 2019-07-04T17:19:54Z

+1
Tested at: 50fc673
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-bdfb5d/1327/summary.html

cmsbuild · 2019-07-04T17:19:59Z

Comparison job queued.

cmsbuild · 2019-07-04T19:52:20Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-bdfb5d/1327/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 849 differences found in the comparisons
DQMHistoTests: Total files compared: 33
DQMHistoTests: Total histograms compared: 3160399
DQMHistoTests: Total failures: 962
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3159103
DQMHistoTests: Total skipped: 334
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 32 files compared)
Checked 137 log files, 14 edm output root files, 33 DQM output files

cmsbuild · 2019-07-05T13:19:28Z

The tests are being triggered in jenkins.
Tested with other pull request(s) #27441
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/1339/console Started: 2019/07/05 15:20

cmsbuild · 2019-07-28T13:29:21Z

Pull request #27433 was updated. @pgunnell, @zhenhu, @prebello, @cmsbuild, @kpedro88 can you please check and sign again.

cmsbuild · 2019-07-28T15:11:51Z

+1
Tested at: c5cae83
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-60de47/1677/summary.html

cmsbuild · 2019-07-28T15:11:54Z

Comparison job queued.

cmsbuild · 2019-07-28T17:12:10Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-60de47/1677/summary.html

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/build/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-60de47/11634.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2021_GenSimFull+DigiFull_2021+RecoFull_2021+HARVESTFull_2021+ALCAFull_2021
/build/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-60de47/11834.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2021PU_GenSimFull+DigiFullPU_2021PU+RecoFullPU_2021PU+HARVESTFullPU_2021PU+NanoFull_2021PU
/build/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-60de47/12434.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023_GenSimFull+DigiFull_2023+RecoFull_2023+HARVESTFull_2023+ALCAFull_2023
/build/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-60de47/12634.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023PU_GenSimFull+DigiFullPU_2023PU+RecoFullPU_2023PU+HARVESTFullPU_2023PU+NanoFull_2023PU
/build/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-60de47/12834.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2024_GenSimFull+DigiFull_2024+RecoFull_2024+HARVESTFull_2024+ALCAFull_2024
/build/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-60de47/13034.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2024PU_GenSimFull+DigiFullPU_2024PU+RecoFullPU_2024PU+HARVESTFullPU_2024PU+NanoFull_2024PU

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 31
DQMHistoTests: Total histograms compared: 2405061
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2404755
DQMHistoTests: Total skipped: 305
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 30 files compared)
Checked 128 log files, 14 edm output root files, 31 DQM output files

zhenhu · 2019-07-30T10:03:51Z

+1

kpedro88 · 2019-07-30T15:45:24Z

unhold

kpedro88 · 2019-07-30T15:45:39Z

+upgrade

cmsbuild · 2019-07-30T15:45:52Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

kpedro88 · 2019-07-30T21:45:17Z

+1

slava77 · 2019-08-03T14:25:40Z

@zhenhu
please edit the PR title to reflect that 2021 13TeV was replaced by 14TeV.
Thank you.

cmsbuild added this to the CMSSW_11_0_X milestone Jul 3, 2019

cmsbuild added code-checks-pending comparison-pending orp-pending pdmv-pending pending-signatures tests-pending upgrade-pending labels Jul 3, 2019

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 3, 2019

makortel reviewed Jul 3, 2019

View reviewed changes

mmusich reviewed Jul 4, 2019

View reviewed changes

christopheralanwest mentioned this pull request Jul 4, 2019

Add 11_0_X global tags for 2021 and later MC scenarios #27441

Merged

cmsbuild added requires-external tests-started and removed tests-pending labels Jul 4, 2019

cmsbuild added tests-approved and removed tests-started labels Jul 4, 2019

cmsbuild added comparison-available and removed comparison-pending labels Jul 4, 2019

cmsbuild removed the tests-approved label Jul 5, 2019

cmsbuild added tests-started and removed tests-pending labels Jul 28, 2019

cmsbuild added tests-approved and removed tests-started labels Jul 28, 2019

cmsbuild added comparison-available and removed comparison-pending labels Jul 28, 2019

cmsbuild added pdmv-approved and removed pdmv-pending labels Jul 30, 2019

cmsbuild added fully-signed upgrade-approved and removed hold pending-signatures upgrade-pending labels Jul 30, 2019

cmsbuild added orp-approved and removed orp-pending labels Jul 30, 2019

cmsbuild merged commit 3047f5d into cms-sw:master Jul 30, 2019

cmsbuild mentioned this pull request Jul 31, 2019

DD: Replace PseudoTrap shape by TruncTubs #27632

Merged

christopheralanwest mentioned this pull request Aug 2, 2019

enabling UL regression for 2018 (and likely Run3) #27644

Merged

kpedro88 mentioned this pull request Aug 2, 2019

fix short matrix for 2021 #27678

Merged

zhenhu changed the title ~~add workflows for 2023 and 2024 scenarios~~ Add workflows for 2023 and 2024. Replace 13TeV 2021 by 14TeV. Aug 4, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add workflows for 2023 and 2024. Replace 13TeV 2021 by 14TeV. #27433

Add workflows for 2023 and 2024. Replace 13TeV 2021 by 14TeV. #27433

zhenhu commented Jul 3, 2019 •

edited

cmsbuild commented Jul 3, 2019

cmsbuild commented Jul 3, 2019

cmsbuild commented Jul 3, 2019

kpedro88 commented Jul 3, 2019

makortel Jul 3, 2019

kpedro88 Jul 3, 2019

christopheralanwest Jul 3, 2019

kpedro88 Jul 3, 2019

fabiocos Jul 4, 2019

kpedro88 Jul 4, 2019

mmusich Jul 4, 2019 •

edited

fabiocos Jul 4, 2019

cmsbuild commented Jul 4, 2019 •

edited

cmsbuild commented Jul 4, 2019

cmsbuild commented Jul 4, 2019

cmsbuild commented Jul 4, 2019

cmsbuild commented Jul 5, 2019 •

edited

cmsbuild commented Jul 28, 2019

cmsbuild commented Jul 28, 2019

cmsbuild commented Jul 28, 2019

cmsbuild commented Jul 28, 2019

zhenhu commented Jul 30, 2019

kpedro88 commented Jul 30, 2019

kpedro88 commented Jul 30, 2019

cmsbuild commented Jul 30, 2019

kpedro88 commented Jul 30, 2019

slava77 commented Aug 3, 2019

Add workflows for 2023 and 2024. Replace 13TeV 2021 by 14TeV. #27433

Add workflows for 2023 and 2024. Replace 13TeV 2021 by 14TeV. #27433

Conversation

zhenhu commented Jul 3, 2019 • edited

PR description:

PR validation:

cmsbuild commented Jul 3, 2019

cmsbuild commented Jul 3, 2019

cmsbuild commented Jul 3, 2019

kpedro88 commented Jul 3, 2019

makortel Jul 3, 2019

Choose a reason for hiding this comment

kpedro88 Jul 3, 2019

Choose a reason for hiding this comment

christopheralanwest Jul 3, 2019

Choose a reason for hiding this comment

kpedro88 Jul 3, 2019

Choose a reason for hiding this comment

fabiocos Jul 4, 2019

Choose a reason for hiding this comment

kpedro88 Jul 4, 2019

Choose a reason for hiding this comment

mmusich Jul 4, 2019 • edited

Choose a reason for hiding this comment

fabiocos Jul 4, 2019

Choose a reason for hiding this comment

cmsbuild commented Jul 4, 2019 • edited

cmsbuild commented Jul 4, 2019

cmsbuild commented Jul 4, 2019

cmsbuild commented Jul 4, 2019

cmsbuild commented Jul 5, 2019 • edited

cmsbuild commented Jul 28, 2019

cmsbuild commented Jul 28, 2019

cmsbuild commented Jul 28, 2019

cmsbuild commented Jul 28, 2019

zhenhu commented Jul 30, 2019

kpedro88 commented Jul 30, 2019

kpedro88 commented Jul 30, 2019

cmsbuild commented Jul 30, 2019

kpedro88 commented Jul 30, 2019

slava77 commented Aug 3, 2019

zhenhu commented Jul 3, 2019 •

edited

mmusich Jul 4, 2019 •

edited

cmsbuild commented Jul 4, 2019 •

edited

cmsbuild commented Jul 5, 2019 •

edited