integrate 0T workflows for Cosmics and Collisions #30527

mmusich · 2020-07-03T13:24:03Z

PR description:

PR #30435 highlighted the lack of appropriate testing workflows to check the reconstruction of 0T data (see in particular #30435 (comment) ). As we (Pixel / Tracker) support 0T conditions for both CRUZET (and eventually 0T collisions data, as it might be foreseeable in the scenario of Run-3 startup [*] in which there is a pilot run in 2021) it is important to not introduce bugs in the 0T data reconstruction, which if untested for long time might hit production workflows.
I introduce in this PR, simulation workflows for comics (wf 7.24) and collisions (w/o PU , wf 11634.24 and w/ PU - std mixing, wf 11834.24) in the 2021 scenario.
To achieve this, a new class UpgradeWorkflow_0T is introduced in Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py and also several new symbolic global tags (derived from the existing auto:phase* keys) supporting the 0T case (via appending of the 0T version of the RunInfo payload).
I provide below a comparison of the parmateres for the 0T and 3.8T RunInfo payloads:

0T payload	3.8T payload
`RunInfo_0T_v1_mc`	`RunInfo_v1_mc`

PR validation:

This branch has been tested by running:

runTheMatrix.py -l 7.24,11634.24,11834.24 -t 4 -j 8 --ibeos

if this PR is a backport please specify the original PR and why you need to backport that PR:

[*] link

cmsbuild · 2020-07-03T13:24:29Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-07-03T13:28:56Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30527/16740

This PR adds an extra 64KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/AlCa/python/autoCond.py modified in PR(s): Update global tags to use V5 JEC for 2017 data/MC and Run 3 MC and add V7b JR for 2018 data/MC #30478
- File Configuration/PyReleaseValidation/python/relval_standard.py modified in PR(s): Add workflow 158.181 as 158.0 with concurrent lumis enabled #30492
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): Add workflow 158.181 as 158.0 with concurrent lumis enabled #30492

cmsbuild · 2020-07-03T13:29:17Z

A new Pull Request was created by @mmusich (Marco Musich) for master.

It involves the following packages:

Configuration/AlCa
Configuration/PyReleaseValidation

@pgunnell, @chayanit, @wajidalikhan, @christopheralanwest, @tocheng, @cmsbuild, @kpedro88, @tlampen, @pohsun can you please review it and eventually sign? Thanks.
@makortel, @tocheng, @Martin-Grunewald, @mmusich, @fabiocos, @slomeo this is something you requested to watch as well.
@silviodonato, @dpiparo you are the release manager for this.

cms-bot commands are listed here

mmusich · 2020-07-03T13:29:31Z

test parameters
workflow = 7.24,11634.24,11834.24

mmusich · 2020-07-03T13:29:42Z

@cmsbuild, please test

cmsbuild · 2020-07-03T13:29:46Z

The tests are being triggered in jenkins.
Test Parameters:

MATRIX_EXTRAS = 7.24,11634.24,11834.24
CMSSW_11_2_X_2020-07-02-2300/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/7659/console Started: 2020/07/03 15:30

christopheralanwest · 2020-07-03T16:25:26Z

Configuration/AlCa/python/autoCond.py

+##
+## Append for 0T conditions
+##
+import six
+from Configuration.StandardSequences.CondDBESSource_cff import GlobalTag as essource
+connectionString = essource.connect.value()
+ConditionsFor0T =  ','.join( ['RunInfo_0T_v1_mc', "RunInfoRcd", connectionString, "", "2020-07-01 12:00:00.000"] )
+GlobalTags0T = {}
+for key,val in six.iteritems(autoCond):
+    if "phase" in key:
+        GlobalTags0T[key+"_0T"] = (autoCond[key], ConditionsFor0T)
+
+autoCond.update(GlobalTags0T)
+


Is there a reason that you have included this snippet here rather than in a separate file with a name such as autoCondSpecial.py or autoCondModifiers.py, analogous to the treatment of autoCondHLT and autoCondPhase2 below? Once the additional modifiers to convert the Run-2 HEfail and cosmic GTs to symbolic GTs, they would also be contained in the same file.

no particular reason, can be moved elsewhere if desired. I didn't know about additional use cases and didn't want to clutter the folder with one more 10 lines file.

@christopheralanwest I followed your suggestion and went ahead to created Configuration/AlCa/python/autoCondModifiers.py. Is that what you had in mind?

Thanks, that's exactly what I had in mind.

cmsbuild · 2020-07-03T17:24:36Z

+1
Tested at: f2f3842
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9b1d1b/7659/summary.html
CMSSW: CMSSW_11_2_X_2020-07-02-2300
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-07-03T17:24:39Z

Comparison job queued.

cmsbuild · 2020-07-03T19:19:04Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9b1d1b/7659/summary.html

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-9b1d1b/11634.24_TTbar_14TeV+TTbar_14TeV_TuneCP5_2021_GenSimFull_0T+DigiFull_0T_2021+RecoFull_0T_2021+HARVESTFull_0T_2021+ALCAFull_0T_2021
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-9b1d1b/11834.24_TTbar_14TeV+TTbar_14TeV_TuneCP5_2021PU_GenSimFull_0T+DigiFullPU_0T_2021PU+RecoFullPU_0T_2021PU+HARVESTFullPU_0T_2021PU+NanoFull_2021PU
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-9b1d1b/7.24_Cosmics_UP21_0T+Cosmics_UP21_0T+DIGICOS_UP21_0T+RECOCOS_UP21_0T+ALCACOS_UP21_0T+HARVESTCOS_UP21_0T

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2784120
DQMHistoTests: Total failures: 5
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2784064
DQMHistoTests: Total skipped: 50
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.004 KiB( 36 files compared)
DQMHistoSizes: changed ( 10224.0 ): -0.004 KiB MessageLogger/Warnings
Checked 154 log files, 17 edm output root files, 37 DQM output files

cmsbuild · 2020-07-06T12:28:54Z

Pull request #30527 was updated. @pgunnell, @chayanit, @wajidalikhan, @christopheralanwest, @tocheng, @cmsbuild, @kpedro88, @tlampen, @pohsun can you please check and sign again.

mmusich · 2020-07-06T12:29:59Z

@cmsbuild please test

cmsbuild · 2020-07-06T12:30:23Z

The tests are being triggered in jenkins.
Test Parameters:

MATRIX_EXTRAS = 7.24,11634.24,11834.24
CMSSW_11_2_X_2020-07-06-1100/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/7707/console Started: 2020/07/06 14:34

cmsbuild · 2020-07-06T14:04:16Z

+1
Tested at: 5dfb6bc
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9b1d1b/7707/summary.html
CMSSW: CMSSW_11_2_X_2020-07-06-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-07-06T14:04:19Z

Comparison job queued.

cmsbuild · 2020-07-06T15:50:32Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9b1d1b/7707/summary.html

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-9b1d1b/11634.24_TTbar_14TeV+TTbar_14TeV_TuneCP5_2021_GenSimFull_0T+DigiFull_0T_2021+RecoFull_0T_2021+HARVESTFull_0T_2021+ALCAFull_0T_2021
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-9b1d1b/11834.24_TTbar_14TeV+TTbar_14TeV_TuneCP5_2021PU_GenSimFull_0T+DigiFullPU_0T_2021PU+RecoFullPU_0T_2021PU+HARVESTFullPU_0T_2021PU+NanoFull_2021PU
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-9b1d1b/7.24_Cosmics_UP21_0T+Cosmics_UP21_0T+DIGICOS_UP21_0T+RECOCOS_UP21_0T+ALCACOS_UP21_0T+HARVESTCOS_UP21_0T

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2787364
DQMHistoTests: Total failures: 5
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2787308
DQMHistoTests: Total skipped: 50
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.004 KiB( 36 files compared)
DQMHistoSizes: changed ( 10224.0 ): -0.004 KiB MessageLogger/Warnings
Checked 154 log files, 17 edm output root files, 37 DQM output files

christopheralanwest · 2020-07-06T16:03:20Z

+1

kpedro88 · 2020-07-06T22:37:09Z

+upgrade

chayanit · 2020-07-07T04:16:56Z

+1

cmsbuild · 2020-07-07T04:17:22Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo (and backports should be raised in the release meeting by the corresponding L2)

silviodonato · 2020-07-07T07:24:15Z

+1

integrate 0T workflows for Cosmics and Collisions

f2f3842

cmsbuild added this to the CMSSW_11_2_X milestone Jul 3, 2020

cmsbuild added alca-pending code-checks-pending comparison-pending orp-pending pdmv-pending pending-signatures tests-pending upgrade-pending labels Jul 3, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 3, 2020

cmsbuild added tests-started and removed tests-pending labels Jul 3, 2020

christopheralanwest reviewed Jul 3, 2020

View reviewed changes

cmsbuild added tests-approved and removed tests-started labels Jul 3, 2020

cmsbuild added comparison-available and removed comparison-pending labels Jul 3, 2020

cmsbuild mentioned this pull request Jul 3, 2020

Use --customise instead of --customise_unsc in TestOldDigi workflow #30536

Merged

move 0T Symbolic GlobalTags to dedicated file

461dd47

cmsbuild added tests-started and removed tests-pending labels Jul 6, 2020

cmsbuild added tests-approved and removed tests-started labels Jul 6, 2020

cmsbuild added comparison-available and removed comparison-pending labels Jul 6, 2020

cmsbuild added alca-approved and removed alca-pending labels Jul 6, 2020

cmsbuild added upgrade-approved and removed upgrade-pending labels Jul 6, 2020

cmsbuild added fully-signed pdmv-approved and removed pdmv-pending pending-signatures labels Jul 7, 2020

cmsbuild mentioned this pull request Jul 7, 2020

Fix L1T bugs and add workflows #30568

Merged

cmsbuild added orp-approved and removed orp-pending labels Jul 7, 2020

cmsbuild merged commit 831cb2d into cms-sw:master Jul 7, 2020

mmusich mentioned this pull request Dec 14, 2021

Workflow 11834.24 timeouts on cs8 #36492

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

integrate 0T workflows for Cosmics and Collisions #30527

integrate 0T workflows for Cosmics and Collisions #30527

mmusich commented Jul 3, 2020

cmsbuild commented Jul 3, 2020

cmsbuild commented Jul 3, 2020

cmsbuild commented Jul 3, 2020

mmusich commented Jul 3, 2020

mmusich commented Jul 3, 2020

cmsbuild commented Jul 3, 2020 •

edited

christopheralanwest Jul 3, 2020

mmusich Jul 3, 2020 •

edited

mmusich Jul 6, 2020

christopheralanwest Jul 6, 2020

cmsbuild commented Jul 3, 2020

cmsbuild commented Jul 3, 2020

cmsbuild commented Jul 3, 2020

cmsbuild commented Jul 6, 2020

mmusich commented Jul 6, 2020

cmsbuild commented Jul 6, 2020 •

edited

cmsbuild commented Jul 6, 2020

cmsbuild commented Jul 6, 2020

cmsbuild commented Jul 6, 2020

christopheralanwest commented Jul 6, 2020

kpedro88 commented Jul 6, 2020

chayanit commented Jul 7, 2020

cmsbuild commented Jul 7, 2020

silviodonato commented Jul 7, 2020

integrate 0T workflows for Cosmics and Collisions #30527

integrate 0T workflows for Cosmics and Collisions #30527

Conversation

mmusich commented Jul 3, 2020

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented Jul 3, 2020

cmsbuild commented Jul 3, 2020

cmsbuild commented Jul 3, 2020

mmusich commented Jul 3, 2020

mmusich commented Jul 3, 2020

cmsbuild commented Jul 3, 2020 • edited

christopheralanwest Jul 3, 2020

Choose a reason for hiding this comment

mmusich Jul 3, 2020 • edited

Choose a reason for hiding this comment

mmusich Jul 6, 2020

Choose a reason for hiding this comment

christopheralanwest Jul 6, 2020

Choose a reason for hiding this comment

cmsbuild commented Jul 3, 2020

cmsbuild commented Jul 3, 2020

cmsbuild commented Jul 3, 2020

cmsbuild commented Jul 6, 2020

mmusich commented Jul 6, 2020

cmsbuild commented Jul 6, 2020 • edited

cmsbuild commented Jul 6, 2020

cmsbuild commented Jul 6, 2020

cmsbuild commented Jul 6, 2020

christopheralanwest commented Jul 6, 2020

kpedro88 commented Jul 6, 2020

chayanit commented Jul 7, 2020

cmsbuild commented Jul 7, 2020

silviodonato commented Jul 7, 2020

cmsbuild commented Jul 3, 2020 •

edited

mmusich Jul 3, 2020 •

edited

cmsbuild commented Jul 6, 2020 •

edited