Clean/rename FromMultiCl HGCal egamma collections #34439

SohamBhattacharya · 2021-07-10T16:20:33Z

This PR is made in collaboration with @hatakeyamak , and is related to the cleaning discussed in [1].
Tagging @beaudett and @archiron , who are looking at the e-gamma Validation/DQM.
The PR will clean the *FromMultiCl HGCal egamma sequences as they've have switched to TICL based reconstruction, and there's no need to run the sim-cluster based sequences for HGCal electrons and photons. So for example:

particleFlowClusterHGCalFromMultiCl will no longer exist as particleFlowClusterHGCal is TICL based.
The sim-cluster based particleFlowClusterHGCal is still kept (now called particleFlowClusterHGCalFromSimCl, and packed into _phase2_hgcal_simPFTask [2]), if it's required by other modules.
At RECO: ecalDrivenGsfElectronsFromMultiCl/photonsFromMultiCl --> ecalDrivenGsfElectronsHGC/photonsHGC.
The new collections now contain only HGCal electrons and photons due to [3, 4] (earlier they used to contain the barrel too).
The plot shows the eta distributions of ecalDrivenGsfElectronsHGC (blue) and ecalDrivenGsfElectronsFromMultiCl (red).
At MINIAOD: slimmedElectronsFromMultiCl/slimmedPhotonsFromMultiCl --> slimmedElectronsHGC/slimmedPhotonsHGC. These collections should remain the same, as it's just a renaming.
The plot shows the eta distributions of slimmedElectronsHGC (blue) and slimmedElectronsFromMultiCl (red).

[1] cms-sw/cmssw#32475 (comment)
[2] https://github.com/SohamBhattacharya/cmssw/blob/clean_FromMultiCl_12_0_0_pre3/RecoParticleFlow/Configuration/python/RecoParticleFlow_cff.py#L76
[3] https://github.com/SohamBhattacharya/cmssw/blob/clean_FromMultiCl_12_0_0_pre3/RecoEgamma/EgammaElectronProducers/python/ecalDrivenGsfElectronCoresHGC_cff.py#L6
[4] https://github.com/SohamBhattacharya/cmssw/blob/clean_FromMultiCl_12_0_0_pre3/RecoEgamma/EgammaPhotonProducers/python/photonCore_cfi.py#L20

cmsbuild · 2021-07-10T16:26:51Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34439/23866

This PR adds an extra 220KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File .gitignore modified in PR(s): WIP: New DQM GUI based on files #30594
- File PhysicsTools/PatAlgos/python/slimming/miniAOD_tools.py modified in PR(s): Unify vertex association between CHS and PUPPI #33885
- File RecoEgamma/EgammaPhotonProducers/python/photons_cfi.py modified in PR(s): Calotower -> HCAL rechit for EGM ID variables #33666

cmsbuild · 2021-07-10T16:27:15Z

A new Pull Request was created by @SohamBhattacharya for master.

It involves the following packages:

.gitignore (core)
DQMOffline/EGamma (dqm)
PhysicsTools/PatAlgos (reconstruction)
RecoEcal/Configuration (reconstruction)
RecoEcal/EgammaClusterProducers (reconstruction)
RecoEgamma/Configuration (reconstruction)
RecoEgamma/EgammaElectronProducers (reconstruction)
RecoEgamma/EgammaPhotonProducers (reconstruction)
RecoEgamma/EgammaTools (reconstruction)
RecoLocalCalo/Configuration (reconstruction)
RecoParticleFlow/Configuration (reconstruction)
RecoParticleFlow/PFClusterProducer (reconstruction)
RecoParticleFlow/PFSimProducer (reconstruction)
RecoParticleFlow/PFTracking (reconstruction)
RecoTauTag/RecoTau (reconstruction)
TrackingTools/Configuration (reconstruction)
TrackingTools/GsfTracking (reconstruction)
Validation/HGCalValidation (dqm)
Validation/RecoEgamma (dqm)

@perrotta, @smuzaffar, @andrius-k, @Dr15Jones, @kmaeshima, @ErnestaP, @ahmad3213, @cmsbuild, @makortel, @jfernan2, @slava77, @jpata, @rvenditti can you please review it and eventually sign? Thanks.
@emilbols, @gouskos, @felicepantaleo, @jainshilpi, @hatakeyamak, @rappoccio, @argiro, @bsunanda, @pfs, @thomreis, @mbluj, @ahinzmann, @varuns23, @seemasharmafnal, @mmarionncern, @sethzenz, @jdolen, @makortel, @JanFSchulte, @lgray, @apsallid, @simonepigazzini, @rociovilar, @vandreev11, @Sam-Harper, @cbernet, @GiacomoSguazzoni, @rovere, @VinInn, @cseez, @bellan, @nhanvtran, @gkasieczka, @schoef, @ebrondol, @mtosi, @dgulhan, @clelange, @jdamgov, @rchatter, @swozniewski, @JyothsnaKomaragiri, @sobhatta, @lecriste, @afiqaize, @gpetruc, @mariadalfonso, @mmusich, @andrzejnovak, @ram1123 this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy, @perrotta you are the release manager for this.

cms-bot commands are listed here

perrotta · 2021-07-12T08:03:27Z

please test

cmsbuild · 2021-07-12T11:56:16Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0e3b41/16694/summary.html
COMMIT: cc9bb11
CMSSW: CMSSW_12_0_X_2021-07-11-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/34439/16694/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 2194 differences found in the comparisons
DQMHistoTests: Total files compared: 38
DQMHistoTests: Total histograms compared: 2787742
DQMHistoTests: Total failures: 4054
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2783687
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 37 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 160 log files, 37 edm output root files, 38 DQM output files
TriggerResults: no differences found

jfernan2 · 2021-07-12T12:09:40Z

@SohamBhattacharya I understand the seen changes are in synch with the PR description, but if you can confirm please:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_0_X_2021-07-11-2300+0e3b41/44003/dqm-histo-comparison-summary.html
Thanks!

makortel · 2021-07-12T13:08:06Z

.gitignore

@@ -5,3 +5,5 @@ __init__.py
 #*#
 *~
 *.pb
+*.root
+*replace.py


Please remove these changes (also from the commit history). Addition of *.root could be discussed, but it would be better done as a separate PR.

@makortel These changes were unintended -- I will remove them. Sorry for the noise.

SohamBhattacharya · 2021-07-13T16:41:21Z

@SohamBhattacharya I understand the seen changes are in synch with the PR description, but if you can confirm please:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_0_X_2021-07-11-2300+0e3b41/44003/dqm-histo-comparison-summary.html
Thanks!

@jfernan2 I went through the change log, and the changes are related to the *FromMultiCl, particleFlowClusterHGCal, and particleFlowSuperClusterHGCal collections, which is expected.

slava77 · 2021-08-09T18:24:23Z

+reconstruction

for #34439 a361668

updates since the last reco signoff were in DQMOffline/EGamma

cmsbuild · 2021-08-09T20:39:57Z

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0e3b41/17649/summary.html
COMMIT: a361668
CMSSW: CMSSW_12_1_X_2021-08-09-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/34439/17649/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

11602.011602.0_SingleElectronPt35+2021+SingleElectronPt35_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA/step2_SingleElectronPt35+2021+SingleElectronPt35_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA.log
11601.011601.0_SingleElectronPt10+2021+SingleElectronPt10_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA/step2_SingleElectronPt10+2021+SingleElectronPt10_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA.log
11603.011603.0_SingleElectronPt1000+2021+SingleElectronPt1000_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA/step2_SingleElectronPt1000+2021+SingleElectronPt1000_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA.log

Expand to see more relval errors ...

Comparison Summary

Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 2365 differences found in the comparisons
DQMHistoTests: Total files compared: 39
DQMHistoTests: Total histograms compared: 2997858
DQMHistoTests: Total failures: 2317
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2995519
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -16.048 KiB( 38 files compared)
DQMHistoSizes: changed ( 23234.0,... ): -2704.699 KiB Egamma/stdPhotonAnalyzerHGCalFromMultiCl
DQMHistoSizes: changed ( 23234.0,... ): 2701.219 KiB Egamma/stdPhotonAnalyzerHGCal
DQMHistoSizes: changed ( 23234.0,... ): -0.531 KiB Egamma/Electrons
Checked 165 log files, 37 edm output root files, 39 DQM output files
TriggerResults: no differences found

srimanob · 2021-08-10T01:02:53Z

@cmsbuild please test

Give it another try on relvals input test

srimanob · 2021-08-10T01:03:52Z

+Upgrade

As the PR title, this PR is to clean and rename HGCal EGamma collections. Change in collection name in DQM is expected.

cmsbuild · 2021-08-10T06:16:14Z

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0e3b41/17656/summary.html
COMMIT: a361668
CMSSW: CMSSW_12_1_X_2021-08-09-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/34439/17656/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

11601.011601.0_SingleElectronPt10+2021+SingleElectronPt10_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA/step2_SingleElectronPt10+2021+SingleElectronPt10_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA.log
11602.011602.0_SingleElectronPt35+2021+SingleElectronPt35_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA/step2_SingleElectronPt35+2021+SingleElectronPt35_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA.log
11604.011604.0_SingleGammaPt10+2021+SingleGammaPt10_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA/step2_SingleGammaPt10+2021+SingleGammaPt10_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA.log

Expand to see more relval errors ...

Comparison Summary

Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 2373 differences found in the comparisons
DQMHistoTests: Total files compared: 39
DQMHistoTests: Total histograms compared: 2997858
DQMHistoTests: Total failures: 2328
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2995507
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -16.052 KiB( 38 files compared)
DQMHistoSizes: changed ( 23234.0,... ): -2704.699 KiB Egamma/stdPhotonAnalyzerHGCalFromMultiCl
DQMHistoSizes: changed ( 23234.0,... ): 2701.219 KiB Egamma/stdPhotonAnalyzerHGCal
DQMHistoSizes: changed ( 23234.0,... ): -0.531 KiB Egamma/Electrons
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 165 log files, 37 edm output root files, 39 DQM output files
TriggerResults: no differences found

jfernan2 · 2021-08-10T07:46:50Z

+1

cmsbuild · 2021-08-10T07:47:17Z

This pull request is fully signed and it will be integrated in one of the next master IBs (but tests are reportedly failing). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2021-08-10T17:12:14Z

+1

All comments received during the review have been addressed, results are in agreement with the expectations

perrotta · 2021-08-10T17:20:16Z

merge

…nCore_cfi.py to suppress the LogError mentioned in comment cms-sw#34439 (comment)

SohamBhattacharya added 9 commits July 8, 2021 13:31

First commit.

d98bec8

Minor changes.

89bf773

Minor changes.

353316e

Minor changes.

36d3c0b

Fixed DQM.

36252d9

Fixed DQM.

6c9e7d5

Fixed RecoTauTag

8704205

Minor changes.

c0fd69f

Egamma HGC to contain HGCal electrons/photons only (exclude the barrel).

cc9bb11

cmsbuild added this to the CMSSW_12_0_X milestone Jul 10, 2021

cmsbuild added code-checks-pending core-pending dqm-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Jul 10, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 10, 2021

cmsbuild added tests-started and removed tests-pending labels Jul 12, 2021

cmsbuild added tests-approved and removed tests-started labels Jul 12, 2021

makortel reviewed Jul 12, 2021

View reviewed changes

cmsbuild added the tests-started label Aug 9, 2021

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Aug 9, 2021

cmsbuild added tests-rejected and removed tests-started labels Aug 9, 2021

cmsbuild added tests-started and removed tests-rejected labels Aug 10, 2021

cmsbuild added upgrade-approved and removed upgrade-pending labels Aug 10, 2021

cmsbuild added tests-rejected and removed tests-started labels Aug 10, 2021

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Aug 10, 2021

cmsbuild added orp-approved and removed orp-pending labels Aug 10, 2021

cmsbuild merged commit 8f85f65 into cms-sw:master Aug 10, 2021

SissonJ pushed a commit to SissonJ/cmssw that referenced this pull request Aug 23, 2021

Added hgcalOnly flag in RecoEgamma/EgammaPhotonProducers/python/photo…

a4a2d04

…nCore_cfi.py to suppress the LogError mentioned in comment cms-sw#34439 (comment)

SissonJ pushed a commit to SissonJ/cmssw that referenced this pull request Aug 23, 2021

Implemented changes related to comment: cms-sw#34439 (comment)

a50d83e

SissonJ pushed a commit to SissonJ/cmssw that referenced this pull request Aug 23, 2021

Implemented review comment: cms-sw#34439 (comment)

04f0cf5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clean/rename FromMultiCl HGCal egamma collections #34439

Clean/rename FromMultiCl HGCal egamma collections #34439

SohamBhattacharya commented Jul 10, 2021

cmsbuild commented Jul 10, 2021

cmsbuild commented Jul 10, 2021

perrotta commented Jul 12, 2021

cmsbuild commented Jul 12, 2021

jfernan2 commented Jul 12, 2021

makortel Jul 12, 2021 •

edited

SohamBhattacharya Jul 13, 2021

SohamBhattacharya commented Jul 13, 2021

slava77 commented Aug 9, 2021

cmsbuild commented Aug 9, 2021

srimanob commented Aug 10, 2021

srimanob commented Aug 10, 2021

cmsbuild commented Aug 10, 2021

jfernan2 commented Aug 10, 2021

cmsbuild commented Aug 10, 2021

perrotta commented Aug 10, 2021

perrotta commented Aug 10, 2021

Clean/rename FromMultiCl HGCal egamma collections #34439

Clean/rename FromMultiCl HGCal egamma collections #34439

Conversation

SohamBhattacharya commented Jul 10, 2021

cmsbuild commented Jul 10, 2021

cmsbuild commented Jul 10, 2021

perrotta commented Jul 12, 2021

cmsbuild commented Jul 12, 2021

Comparison Summary

jfernan2 commented Jul 12, 2021

makortel Jul 12, 2021 • edited

Choose a reason for hiding this comment

SohamBhattacharya Jul 13, 2021

Choose a reason for hiding this comment

SohamBhattacharya commented Jul 13, 2021

slava77 commented Aug 9, 2021

cmsbuild commented Aug 9, 2021

RelVals-INPUT

Comparison Summary

srimanob commented Aug 10, 2021

srimanob commented Aug 10, 2021

cmsbuild commented Aug 10, 2021

RelVals-INPUT

Comparison Summary

jfernan2 commented Aug 10, 2021

cmsbuild commented Aug 10, 2021

perrotta commented Aug 10, 2021

perrotta commented Aug 10, 2021

makortel Jul 12, 2021 •

edited