[12.6.X] Adjust collections of Tracker clusters to be saved for Phase-2 ALCARECO #40394

mmusich · 2022-12-22T09:18:44Z

PR description:

This is a follow-up of PR #39858.
After the backport of that PR (#39898) was integrated, it was possible to test the production of RelVal samples (see e.g. on DAS: /RelValMinBias_14TeV/CMSSW_12_5_2-TkAlMinBias-125X_mcRun4_realistic_v3_2026D88PU-v1/ALCARECO).
When trying to run over it, it became clear that is not possible to refit tracks, due to the missing event products (the Phase-2 Outer Tracker clusters) that are necessary for CPE re-computation, leading to this error:

Begin processing the 1st record. Run 1, Event 301, LumiSection 4 on stream 6 at 13-Dec-2022 12:50:01.219 CET
----- Begin Fatal Exception 13-Dec-2022 12:50:57 CET-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 4 event: 306 stream: 2
   [1] Running path 'p2'
   [2] Calling method for module TrackRefitter/'FinalTrackRefitter'
Exception Message:
RefCore: A request to resolve a reference to a product of type 'edmNew::DetSetVector<Phase2TrackerCluster1D>' with ProductID '4:449'
can not be satisfied because the product cannot be found.
Probably the branch containing the product is not stored in the input file.
   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "SkipEvent = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

This PR adds the minimal amount of changes in the CommonTools/RecoAlgos package in order to be able to retain the Phase-2 OT clusters after running the AlignmentTrackSelector module that relies on it:

cmssw/Alignment/CommonAlignmentProducer/plugins/AlignmentTrackSelectorModule.cc

Lines 13 to 17 in 52c0dac

    
           // the following include is necessary to clone all track branches 
        
           // including recoTrackExtras and TrackingRecHitsOwned (in future also "owned clusters"?). 
        
           // if you remove it the code will compile, but the cloned 
        
           // tracks have only the recoTracks branch! 
        
           #include "CommonTools/RecoAlgos/interface/TrackSelector.h"

This is done in commit 35ef2e2.
Secondly the outputCommands of all the Tracker Alignment ALCARECO producers is modified in order to retain Phase2TrackerCluster1DedmNewDetSetVector in the output - in case the production is run for Phase-2 - while removing SiStripClusteredmNewDetSetVector (since there is no SiStrip detector in phase-2). I profit of that to re-adjust the commands for event contents to cope with no SCAL in Run3 that were leftover from #38415 and #38590.
This is done in commit 08f72fa.
Additionally this contains commit 474efd4 which fixes a problem with the vector hits workflow spotted in master and commit e78b930 which makes the ClusterStorer::rekey methods non-const as per recommendations in #40363.

PR validation:

Run two typical workflows for Run-3 and Phase-2 with

runTheMatrix.py -l 11634.0,20834.0 -t 4 -j 8 --ibeos

And tested the event content of TkAlMinBias indeed changes as expected.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

This is a combined backport of #40323 and #40363 (partially backported). It is backported in this cycle in order to use it for the production of ALCARECOs within the context of the L1T production happening in that cycle, see https://its.cern.ch/jira/browse/PDMVRELVALS-178 and McM.

…to store the Phase-2 clusters

…T clusters

cmsbuild · 2022-12-22T09:19:06Z

A new Pull Request was created by @mmusich (Marco Musich) for CMSSW_12_6_X.

It involves the following packages:

Alignment/CommonAlignmentProducer (alca)
CommonTools/RecoAlgos (reconstruction)
DataFormats/TrackerRecHit2D (reconstruction)

@malbouis, @yuanchao, @ChrisMisan, @clacaputo, @cmsbuild, @saumyaphor4252, @tvami, @mandrenguyen, @francescobrivio can you please review it and eventually sign? Thanks.
@pakhotin, @rappoccio, @jdolen, @tlampen, @VourMa, @adewit, @ahinzmann, @abbiendi, @JanFSchulte, @rovere, @VinInn, @missirol, @gkasieczka, @tocheng, @hatakeyamak, @gpetruc, @mmusich, @mtosi, @jhgoh this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

Backported from Adjust collections of Tracker clusters to be saved for Phase-2 ALCARECO #40323

mmusich · 2022-12-22T09:19:36Z

@cmsbuild, please test

cmsbuild · 2022-12-22T13:26:53Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9cc942/29752/summary.html
COMMIT: e78b930
CMSSW: CMSSW_12_6_X_2022-12-21-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/40394/29752/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3458651
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3458623
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 206 log files, 158 edm output root files, 48 DQM output files
TriggerResults: no differences found

mandrenguyen · 2022-12-22T13:59:30Z

Just checking, you don't also want this commit?
fe0f886

mmusich · 2022-12-22T14:01:48Z

Just checking, you don't also want this commit?

I thought to leave it out, as it's only needed for future development (which will happen in master)

mmusich · 2022-12-22T14:02:34Z

backport of #40323

mmusich · 2022-12-22T14:02:53Z

backport of #40363 (partial)

mandrenguyen · 2022-12-22T14:28:59Z

+reconstruction

tvami · 2022-12-22T14:37:02Z

+alca

cmsbuild · 2022-12-22T14:37:22Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_6_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_13_0_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2022-12-22T15:56:22Z

test parameters:

workflow = 20834.9, 20818.9
relvals_opt= -w upgrade

perrotta · 2022-12-22T15:56:55Z

please test

mmusich · 2022-12-22T16:28:24Z

@cmsbuild please abort

mmusich · 2022-12-22T16:29:25Z

@perrotta

wf 20818.9 was not backported, see #40394 (comment)
please change the test parameters

mmusich · 2022-12-22T17:44:09Z

test parameters:

workflow = 20834.9
relvals_opt= -w upgrade

mmusich · 2022-12-22T17:44:17Z

please test

cmsbuild · 2022-12-22T22:17:19Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9cc942/29760/summary.html
COMMIT: e78b930
CMSSW: CMSSW_12_6_X_2022-12-22-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/40394/29760/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3554525
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3554503
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 211 log files, 162 edm output root files, 49 DQM output files
TriggerResults: no differences found

perrotta · 2022-12-23T05:37:44Z

+1

mmusich added 4 commits December 21, 2022 10:25

Introduce necessary modifications to AlignmentTrackSelector in order …

35ef2e2

…to store the Phase-2 clusters

change output commands of Tracker ALCARECOs to take care of phase-2 O…

08f72fa

…T clusters

ClusterStorer: deal with the VectorHits case separately

474efd4

make rekey non-const as per PR review

e78b930

cmsbuild added this to the CMSSW_12_6_X milestone Dec 22, 2022

cmsbuild added alca-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Dec 22, 2022

cmsbuild added tests-started and removed tests-pending labels Dec 22, 2022

cmsbuild added tests-approved and removed tests-started labels Dec 22, 2022

cmsbuild added the backport-ok label Dec 22, 2022

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Dec 22, 2022

cmsbuild added alca-approved fully-signed and removed alca-pending pending-signatures labels Dec 22, 2022

cmsbuild added tests-started and removed tests-approved labels Dec 22, 2022

cmsbuild added tests-pending and removed tests-started labels Dec 22, 2022

cmsbuild added tests-started and removed tests-pending labels Dec 22, 2022

cmsbuild added tests-approved and removed tests-started labels Dec 22, 2022

cmsbuild added orp-approved and removed orp-pending labels Dec 23, 2022

cmsbuild merged commit 0e938af into cms-sw:CMSSW_12_6_X Dec 23, 2022

mmusich deleted the AlignmentTrackSelectorPhase2_12_6_X branch December 23, 2022 07:56

mmusich mentioned this pull request Jan 18, 2023

[12.6.X] fix ClusterStorer::ClusterHitRecord<T>::rekey for Phase2TrackerRecHit1D #40558

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[12.6.X] Adjust collections of Tracker clusters to be saved for Phase-2 ALCARECO #40394

[12.6.X] Adjust collections of Tracker clusters to be saved for Phase-2 ALCARECO #40394

mmusich commented Dec 22, 2022

cmsbuild commented Dec 22, 2022 •

edited

mmusich commented Dec 22, 2022

cmsbuild commented Dec 22, 2022

mandrenguyen commented Dec 22, 2022

mmusich commented Dec 22, 2022

mmusich commented Dec 22, 2022

mmusich commented Dec 22, 2022

mandrenguyen commented Dec 22, 2022

tvami commented Dec 22, 2022

cmsbuild commented Dec 22, 2022

perrotta commented Dec 22, 2022

perrotta commented Dec 22, 2022

mmusich commented Dec 22, 2022

mmusich commented Dec 22, 2022

mmusich commented Dec 22, 2022

mmusich commented Dec 22, 2022

cmsbuild commented Dec 22, 2022

perrotta commented Dec 23, 2022

	// the following include is necessary to clone all track branches
	// including recoTrackExtras and TrackingRecHitsOwned (in future also "owned clusters"?).
	// if you remove it the code will compile, but the cloned
	// tracks have only the recoTracks branch!
	#include "CommonTools/RecoAlgos/interface/TrackSelector.h"

[12.6.X] Adjust collections of Tracker clusters to be saved for Phase-2 ALCARECO #40394

[12.6.X] Adjust collections of Tracker clusters to be saved for Phase-2 ALCARECO #40394

Conversation

mmusich commented Dec 22, 2022

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

cmsbuild commented Dec 22, 2022 • edited

mmusich commented Dec 22, 2022

cmsbuild commented Dec 22, 2022

Comparison Summary

mandrenguyen commented Dec 22, 2022

mmusich commented Dec 22, 2022

mmusich commented Dec 22, 2022

mmusich commented Dec 22, 2022

mandrenguyen commented Dec 22, 2022

tvami commented Dec 22, 2022

cmsbuild commented Dec 22, 2022

perrotta commented Dec 22, 2022

perrotta commented Dec 22, 2022

mmusich commented Dec 22, 2022

mmusich commented Dec 22, 2022

mmusich commented Dec 22, 2022

mmusich commented Dec 22, 2022

cmsbuild commented Dec 22, 2022

Comparison Summary

perrotta commented Dec 23, 2022

cmsbuild commented Dec 22, 2022 •

edited