Phase2 workflow revamp

[kpedro88]

This PR reorganizes the Phase2 workflows as discussed in the Upgrade Studies meeting, in preparation for HGCal "cheated" reco.

The new naming scheme is described in Configuration/Geometry/README as requested by @ianna. There is also a new Configuration/Geometry/test with a script to generate 2023 geometries automatically. (For the Phase2 Eras, the only difference is caused by the calorimeter content, so I named them Phase2C1 and Phase2C2.)

The upgrade workflow matrix numbers have been reorganized. This is reflected in all the necessary configuration files.

Other items of note:

• run2_common is now chained into the Phase2 Eras
• hbheUpgradeReco and hfUpgradeReco are no longer used as tags; the reco modules themselves are now replaced using toReplaceWith, so the usual hbhereco and hfreco tags for the RecHit collections will continue to work in all downstream reco
• phase2 validation and DQM sequences are now defined in the autoValidation and autoDQM configs (@davidlange6, let me know if this is what you had in mind)
• upgrade workflow components are reorganized slightly (@boudoul, let me know if you have any qualms - I just wanted to have one dictionary for everything)

All the workflows should still run. I demoted the HGCal workflows back to local reco until the cheated reco is fully ready (@lgray is working on the necessary downstream reco changes).

Don't be intimidated by the apparent size of this PR; ~98% of the changes are in Python...

[cmsbuild]

A new Pull Request was created by @kpedro88 (Kevin Pedro) for CMSSW_8_1_X.

It involves the following packages:

Calibration/IsolatedParticles
CondTools/Geometry
Configuration/Geometry
Configuration/PyReleaseValidation
Configuration/StandardSequences
DQMOffline/Configuration
Geometry/CMSCommonData
RecoJets/JetProducers
RecoLocalCalo/Configuration
RecoLocalCalo/HGCalRecProducers
RecoLocalTracker/SiPhase2Clusterizer
RecoMET/Configuration
SLHCUpgradeSimulations/Configuration
SLHCUpgradeSimulations/Geometry
SimCalorimetry/HcalSimProducers
SimGeneral/MixingModule
SimMuon/GEMDigitizer
Validation/Configuration
Validation/HGCalValidation

@ghellwig, @civanch, @Dr15Jones, @vanbesien, @cvuosalo, @ianna, @mdhildreth, @fabozzi, @cmsbuild, @srimanob, @cerminar, @franzoni, @slava77, @ggovi, @mmusich, @dmitrijus, @hengne, @davidlange6 can you please review it and eventually sign? Thanks.
@ghellwig, @TaiSakuma, @yslai, @rappoccio, @Martin-Grunewald, @bsunanda, @pfs, @threus, @mmarionncern, @sethzenz, @makortel, @jhgoh, @lgray, @jdolen, @rociovilar, @vandreev11, @wmtan, @GiacomoSguazzoni, @tocheng, @VinInn, @cseez, @nhanvtran, @rovere, @schoef, @dgulhan, @apfeiffer1, @ahinzmann, @gpetruc, @mariadalfonso this is something you requested to watch as well.
@slava77, @smuzaffar you are the release manager for this.

cms-bot commands are list here #13028

[kpedro88]

@cmsbuild please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/14424/console

[cmsbuild]

+1
Tested at: 5452c56
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-15393/14424/summary.html

[kpedro88]

@slava77 yes, that's correct. Hopefully the customization function will be going away soon.

[slava77]

apparently this syntax doesn't work. cmsDriver output with this auto ends up having only the first element baseValidation, while everything else is lost.

[slava77]

I looked at configs in 10624 (baseline) vs 10424 (this PR) and see

• changes related to run2_common < -- expected
• loss of dqm and validation sequences <-- I'm guessing, not expected.

[kpedro88]

@slava77 I will try to figure out the problem with DQM/validation this afternoon

[cmsbuild]

Pull request #15393 was updated. @ghellwig, @civanch, @Dr15Jones, @vanbesien, @cvuosalo, @ianna, @mdhildreth, @fabozzi, @cmsbuild, @srimanob, @cerminar, @franzoni, @slava77, @ggovi, @mmusich, @dmitrijus, @hengne, @davidlange6 can you please check and sign again.

[kpedro88]

@cmsbuild please test

@slava77 figured out the problem, just needed to compose the sequences differently

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/14466/console

[cmsbuild]

+1
Tested at: 7236319
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-15393/14466/summary.html

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-15393/14466/summary.html

[slava77]

+1

for #15393 7236319

• changes are in line with the description
• jenkins tests pass and show no changes in pre-phase2 workflows
• local tests of equivalent 2023 workflows show changes apparently driven by the shift to run2_common (this triggers changes as early as in SIM. So, the geant random sequences are not the same and comparisons with baseline show a lot of changes). Config inspection suggests the changes are as intended.

[davidlange6]

Hi all - as already discussed in the ORP, please make any further comments on this PR today. Thanks