Add GEM alignment in all Run3 GTs and update BeamSpot in Run3 MC GTs #38692

francescobrivio · 2022-07-11T21:35:03Z

PR description:

In this PR we add:

the new GEM Alignment and AlignmentErrors tags to all Run3 GTs (CMSTalk post)
- In order to be consistent with these new tags (and with the GEM geometry already in use in HLT/Express/Prompt), the data and data_relval GTs have also been updated with the latest GEMRECO_Geometry tag (from v2 to v3)
the new BeamSpot for Run 3 collisions at 13.6 TeV in all Run3 MC GTs
- Compatible with the VtxSmearing parameters Realistic25ns13p6TeVEarly2022Collision introduced in Add VtxSmearing scenario for Run 3 early data at 13.6 TeV #38616
- The new reco BeamSpot was derived from the RelVal produced in JIRA-PDMVRELVALS-156
new DroboxMetaData tag for the Express and Prompt frozen GTs and in the data and data_relval GTs (CMSTalk post)

The new GEM tags are:
GEMAlignment_hlt_v2
GEMAlignment_express_v2
GEMAlignment_prompt_v2
GEMAlignment_v2
GEMAlignment_MC_v1
GEMAlignment_ideal_MC_v1

GEMAlignmentErrorExtended_6x6_hlt_v2
GEMAlignmentErrorExtended_6x6_express_v2
GEMAlignmentErrorExtended_6x6_prompt_v2
GEMAlignmentErrorExtended_6x6_v2
GEMAlignmentErrorExtended_6x6_MC_v1
GEMAlignmentErrorExtended_6x6_ideal_MC_v1

The GEMRECO geometry tag is:
GEMRECO_Geometry_v3_hlt

The new BeamSpot tags are:
BeamSpotObjects_Realistic25ns_13p6TeVCollisions_Early2022_v1_mc
BeamSpotOnlineObjects_Realistic25ns_13p6TeVCollisions_Early2022_v1_mc

The new DMD tag is:
DropBoxMetadata_v8_express

GT Diffs:

Frozen HLT https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/124X_dataRun3_HLT_frozen_v4/124X_dataRun3_HLT_frozen_v6
HLT relval https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/124X_dataRun3_HLT_relval_v6/124X_dataRun3_HLT_relval_v8
Frozen Express https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/124X_dataRun3_Express_frozen_v2/124X_dataRun3_Express_frozen_v4
Frozen Prompt https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/124X_dataRun3_Prompt_frozen_v2/124X_dataRun3_Prompt_frozen_v4
Data Run3 https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/124X_dataRun3_v5/124X_dataRun3_v9
Data Run3 relval https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/124X_dataRun3_relval_v5/124X_dataRun3_relval_v8
MC Run3 design https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/124X_mcRun3_2022_design_v6/124X_mcRun3_2022_design_v7
MC Run3 realistic https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/124X_mcRun3_2022_realistic_v8/124X_mcRun3_2022_realistic_v9
MC Run3 cosmics realistic https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/124X_mcRun3_2022cosmics_realistic_deco_v9/124X_mcRun3_2022cosmics_realistic_deco_v10
MC Run3 cosmics design https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/124X_mcRun3_2022cosmics_design_deco_v6/124X_mcRun3_2022cosmics_design_deco_v7
MC Run3 realistic HI https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/124X_mcRun3_2022_realistic_HI_v8/124X_mcRun3_2022_realistic_HI_v9
MC Run3 2023 realistic https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/124X_mcRun3_2023_realistic_v9/124X_mcRun3_2023_realistic_v10
MC Run3 2024 realistic https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/124X_mcRun3_2024_realistic_v9/124X_mcRun3_2024_realistic_v10
Open HLT https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/124X_dataRun3_HLT_v3/124X_dataRun3_HLT_v4
Open Express https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/124X_dataRun3_Express_v3/124X_dataRun3_Express_v4
Open Prompt https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/124X_dataRun3_Prompt_v3/124X_dataRun3_Prompt_v4

PR validation:

Tested with
runTheMatrix.py -l 139.001,138.4,138.5,12034.0,11634.0,7.23,159.0,12434.0,12834.0 --ibeos -j 16

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Not a backport, but a 12_4_X backport will be provided soon

francescobrivio · 2022-07-11T21:37:25Z

test parameters:

pull_request = Run3 GEMGeometryBuilder Run3 modifier #38666
workflows = 12034.0,11634.0,7.23,159.0,12434.0,12834.0

cmsbuild · 2022-07-11T21:42:11Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-38692/30990

This PR adds an extra 12KB to repository

cmsbuild · 2022-07-11T21:42:33Z

A new Pull Request was created by @francescobrivio for master.

It involves the following packages:

Configuration/AlCa (alca)

@malbouis, @yuanchao, @cmsbuild, @francescobrivio, @ChrisMisan, @tvami can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @missirol, @mmusich, @fabiocos, @tocheng this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy, @rappoccio you are the release manager for this.

cms-bot commands are listed here

francescobrivio · 2022-07-11T21:49:55Z

@cmsbuild please test

cmsbuild · 2022-07-12T00:03:51Z

-1

Failed Tests: UnitTests RelVals RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8fc0cf/26150/summary.html
COMMIT: fb3b459
CMSSW: CMSSW_12_5_X_2022-07-11-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38692/26150/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestDQMOnlineClient-beam_dqm_sourceclient had ERRORS
---> test TestDQMOnlineClient-mutracking_dqm_sourceclient had ERRORS
---> test TestDQMOnlineClient-pixel_dqm_sourceclient had ERRORS

RelVals

39496.0

----- Begin Fatal Exception 12-Jul-2022 00:41:22 CEST-----------------------
An exception of category 'NoRecord' occurred while
   [0] Calling method for module GEMDigiProducer/'simMuonGEMDigis'
   [1] Calling method for EventSetup module GEMGeometryESModule/'gemGeometry'
   [2] While getting dependent Record from Record MuonGeometryRecord
Exception Message:
No "GEMAlignmentRcd" record found in the EventSetup.

 Please add an ESSource or ESProducer that delivers such a record.
----- End Fatal Exception -------------------------------------------------

39500.0

----- Begin Fatal Exception 12-Jul-2022 00:41:24 CEST-----------------------
An exception of category 'NoRecord' occurred while
   [0] Calling method for module GEMDigiProducer/'simMuonGEMDigis'
   [1] Calling method for EventSetup module GEMGeometryESModule/'gemGeometry'
   [2] While getting dependent Record from Record MuonGeometryRecord
Exception Message:
No "GEMAlignmentRcd" record found in the EventSetup.

 Please add an ESSource or ESProducer that delivers such a record.
----- End Fatal Exception -------------------------------------------------

23234.0

----- Begin Fatal Exception 12-Jul-2022 00:41:50 CEST-----------------------
An exception of category 'NoRecord' occurred while
   [0] Calling method for module GEMDigiProducer/'simMuonGEMDigis'
   [1] Calling method for EventSetup module GEMGeometryESModule/'gemGeometry'
   [2] While getting dependent Record from Record MuonGeometryRecord
Exception Message:
No "GEMAlignmentRcd" record found in the EventSetup.

 Please add an ESSource or ESProducer that delivers such a record.
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

RelVals-INPUT

23234.023234.0_TTbar_14TeV+2026D49+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14INPUT+DigiTrigger+RecoGlobal+HARVESTGlobal/step2_TTbar_14TeV+2026D49+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14INPUT+DigiTrigger+RecoGlobal+HARVESTGlobal.log
23234.2123234.21_TTbar_14TeV+2026D49_ProdLike+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14INPUT+DigiTrigger+RecoGlobal+MiniAOD/step2_TTbar_14TeV+2026D49_ProdLike+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14INPUT+DigiTrigger+RecoGlobal+MiniAOD.log
23434.2123434.21_TTbar_14TeV+2026D49PU_ProdLike+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14INPUT+DigiTriggerPU+RecoGlobalPU+MiniAODPU/step2_TTbar_14TeV+2026D49PU_ProdLike+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14INPUT+DigiTriggerPU+RecoGlobalPU+MiniAODPU.log

Expand to see more relval errors ...

francescobrivio · 2022-07-12T12:57:08Z

@cmsbuild please test

cmsbuild · 2022-07-12T13:01:12Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-38692/30999

This PR adds an extra 12KB to repository

cmsbuild · 2022-07-13T07:15:30Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8fc0cf/26177/summary.html
COMMIT: 95d0e5f
CMSSW: CMSSW_12_5_X_2022-07-12-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38692/26177/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestDQMOnlineClient-gem_dqm_sourceclient had ERRORS

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-8fc0cf/12034.0_TTbar_14TeV+2021Design+TTbar_14TeV_TuneCP5_GenSim+Digi+RecoNano+HARVESTNano
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-8fc0cf/7.23_Cosmics_UP21+Cosmics_UP21+DIGICOS_UP21+RECOCOS_UP21+ALCACOS_UP21+HARVESTCOS_UP21

Summary:

No significant changes to the logs found
Reco comparison results: 23132 differences found in the comparisons
DQMHistoTests: Total files compared: 54
DQMHistoTests: Total histograms compared: 4116336
DQMHistoTests: Total failures: 65543
DQMHistoTests: Total nulls: 7
DQMHistoTests: Total successes: 4050764
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 2.254 KiB( 53 files compared)
DQMHistoSizes: changed ( 11834.0 ): 2.250 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 227 log files, 45 edm output root files, 54 DQM output files
TriggerResults: found differences in 8 / 53 workflows

francescobrivio · 2022-07-14T10:11:45Z

Since the GEM fix for DQM might take a while (see #38666 (comment)) I want to test this PR without that PR.

francescobrivio · 2022-07-14T10:11:53Z

test parameters:
*

francescobrivio · 2022-07-14T10:12:25Z

test parameters:

workflows = 12034.0,11634.0,7.23,159.0,12434.0,12834.0

francescobrivio · 2022-07-14T10:12:51Z

@cmsbuild please test

cmsbuild · 2022-07-14T15:59:36Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8fc0cf/26239/summary.html
COMMIT: 95d0e5f
CMSSW: CMSSW_12_5_X_2022-07-13-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38692/26239/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-8fc0cf/12034.0_TTbar_14TeV+2021Design+TTbar_14TeV_TuneCP5_GenSim+Digi+RecoNano+HARVESTNano
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-8fc0cf/7.23_Cosmics_UP21+Cosmics_UP21+DIGICOS_UP21+RECOCOS_UP21+ALCACOS_UP21+HARVESTCOS_UP21

Summary:

No significant changes to the logs found
Reco comparison results: 23133 differences found in the comparisons
DQMHistoTests: Total files compared: 54
DQMHistoTests: Total histograms compared: 4116592
DQMHistoTests: Total failures: 65531
DQMHistoTests: Total nulls: 7
DQMHistoTests: Total successes: 4051032
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 2.246 KiB( 53 files compared)
DQMHistoSizes: changed ( 11834.0 ): 2.250 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 227 log files, 45 edm output root files, 54 DQM output files
TriggerResults: found differences in 8 / 53 workflows

tvami · 2022-07-15T13:08:58Z

+alca

tests pass, changes are widespread, but in line with the conditions changes in the GT

cmsbuild · 2022-07-15T13:09:17Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2022-07-15T16:43:13Z

+1

Differences in the run3 workflows had been evaluated as being compatible with the GT changed

mmusich · 2022-07-15T16:48:54Z

@perrotta @cms-sw/alca-l2 merging this PR as is has been a mistake.
The reconstruction level beamspot is now totally out of synch with the vertex smearing used for the Run3 workflows, completely destroying tracking performance.
Please either revert or fix ASAP.

francescobrivio · 2022-07-15T16:54:31Z

Please either revert or fix

How do we update the Run 3 workflow? (sorry if the question is naive....)
I'll be able to work on this only tomorrow btw

mmusich · 2022-07-15T17:01:54Z

@francescobrivio here

cmssw/Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py

Line 2058 in a1ba0ee

'BeamSpot': 'Run3RoundOptics25ns13TeVLowSigmaZ',

and elsewhere, where relevant.

Ad GEM alignment in all Run3 GTs and update BeamSpot in Run3 MC GTs

fb3b459

cmsbuild added this to the CMSSW_12_5_X milestone Jul 11, 2022

cmsbuild added alca-pending code-checks-pending orp-pending pending-signatures tests-pending labels Jul 11, 2022

cmsbuild added the requires-external label Jul 11, 2022

francescobrivio mentioned this pull request Jul 11, 2022

[12_4_X] Add GEM alignment in all Run3 GTs #38693

Merged

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 11, 2022

francescobrivio mentioned this pull request Jul 11, 2022

Run3 GEMGeometryBuilder Run3 modifier #38666

Merged

cmsbuild added tests-started and removed tests-pending labels Jul 11, 2022

cmsbuild added tests-rejected and removed tests-started labels Jul 12, 2022

update snapshot of Run3 data GTs

fef6639

cmsbuild added code-checks-pending tests-pending and removed tests-rejected code-checks-approved labels Jul 12, 2022

cmsbuild added tests-started and removed tests-pending labels Jul 12, 2022

cmsbuild added the tests-started label Jul 12, 2022

cmsbuild added tests-rejected and removed tests-started labels Jul 13, 2022

cmsbuild removed the requires-external label Jul 14, 2022

cmsbuild added tests-started and removed tests-rejected labels Jul 14, 2022

cmsbuild added tests-approved and removed tests-started labels Jul 14, 2022

cmsbuild added alca-approved fully-signed and removed alca-pending pending-signatures labels Jul 15, 2022

cmsbuild added orp-approved and removed orp-pending labels Jul 15, 2022

cmsbuild merged commit a1ba0ee into cms-sw:master Jul 15, 2022

mmusich mentioned this pull request Jul 15, 2022

Update BeamSpot in Run3 MC GTs + swap Run3RoundOptics25ns13TeVLowSigmaZ with Realistic25ns13p6TeVEarly2022Collision vtx smearing #38760

Merged

perrotta mentioned this pull request Jul 15, 2022

Revert "Add GEM alignment in all Run3 GTs and update BeamSpot in Run3 MC GTs" #38762

Merged

francescobrivio deleted the alca-GEMAlignment branch July 19, 2022 13:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GEM alignment in all Run3 GTs and update BeamSpot in Run3 MC GTs #38692

Add GEM alignment in all Run3 GTs and update BeamSpot in Run3 MC GTs #38692

francescobrivio commented Jul 11, 2022 •

edited

francescobrivio commented Jul 11, 2022

cmsbuild commented Jul 11, 2022

cmsbuild commented Jul 11, 2022

francescobrivio commented Jul 11, 2022

cmsbuild commented Jul 12, 2022

francescobrivio commented Jul 12, 2022

cmsbuild commented Jul 12, 2022

cmsbuild commented Jul 13, 2022

francescobrivio commented Jul 14, 2022

francescobrivio commented Jul 14, 2022

francescobrivio commented Jul 14, 2022

francescobrivio commented Jul 14, 2022

cmsbuild commented Jul 14, 2022

tvami commented Jul 15, 2022

cmsbuild commented Jul 15, 2022

perrotta commented Jul 15, 2022

mmusich commented Jul 15, 2022

francescobrivio commented Jul 15, 2022

mmusich commented Jul 15, 2022

Add GEM alignment in all Run3 GTs and update BeamSpot in Run3 MC GTs #38692

Add GEM alignment in all Run3 GTs and update BeamSpot in Run3 MC GTs #38692

Conversation

francescobrivio commented Jul 11, 2022 • edited

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

francescobrivio commented Jul 11, 2022

cmsbuild commented Jul 11, 2022

cmsbuild commented Jul 11, 2022

francescobrivio commented Jul 11, 2022

cmsbuild commented Jul 12, 2022

Unit Tests

RelVals

RelVals-INPUT

francescobrivio commented Jul 12, 2022

cmsbuild commented Jul 12, 2022

cmsbuild commented Jul 13, 2022

Unit Tests

Comparison Summary

francescobrivio commented Jul 14, 2022

francescobrivio commented Jul 14, 2022

francescobrivio commented Jul 14, 2022

francescobrivio commented Jul 14, 2022

cmsbuild commented Jul 14, 2022

Comparison Summary

tvami commented Jul 15, 2022

cmsbuild commented Jul 15, 2022

perrotta commented Jul 15, 2022

mmusich commented Jul 15, 2022

francescobrivio commented Jul 15, 2022

mmusich commented Jul 15, 2022

francescobrivio commented Jul 11, 2022 •

edited