Backport: Light by light modifier PR 103X

[rchudasa]

PR description:

@slava77 @echapon
This PR is a backport of following PRs.

• updating SC reg corrector for UL regression #26763 updating SC reg corrector for UL (May 16)
• Adapting EGRegression Modifiers to 106X and adding V3 modifier #26892 adding V3 modifier (Jun 13)
• Light by light modifier 1100 pre1 #27633 Light by light modifier (Aug 22)
• Filtering on pixel and strip clusters #27871 tracker cluster cut (Aug 27)
• Adding the regression config for light by light #28051 low-pt regression (Oct 15)

This PR is submitted to reconstruct low pT photons down to 2 GeV for light by light scattering.

[cmsbuild]

A new Pull Request was created by @rchudasa for CMSSW_10_3_X.

It involves the following packages:

Configuration/ProcessModifiers
RecoEcal/EgammaClusterProducers
RecoEgamma/EgammaElectronProducers
RecoEgamma/EgammaIsolationAlgos
RecoEgamma/EgammaPhotonProducers
RecoLocalCalo/Configuration
RecoLocalTracker/Configuration
RecoParticleFlow/PFProducer
RecoParticleFlow/PFTracking
RecoTracker/TkSeedGenerator

@perrotta, @kpedro88, @cmsbuild, @franzoni, @slava77, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks.
@echabert, @felicepantaleo, @jainshilpi, @pieterdavid, @argiro, @Martin-Grunewald, @mschrode, @lgray, @threus, @mmusich, @seemasharmafnal, @mmarionncern, @makortel, @varuns23, @apsallid, @yduhm, @Sam-Harper, @GiacomoSguazzoni, @rovere, @VinInn, @hatakeyamak, @alesaggio, @ebrondol, @dgulhan, @gbenelli, @cbernet, @gpetruc, @bachtis this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

• Backported from Light by light modifier 1100 pre1 #27633

[kpedro88]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/2261/console Started: 2019/08/29 16:50

[cmsbuild]

+1
Tested at: d829f0c
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-649856/2261/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-649856/2261/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 32
• DQMHistoTests: Total histograms compared: 3203312
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3203114
• DQMHistoTests: Total skipped: 197
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 31 files compared)
• Checked 134 log files, 14 edm output root files, 32 DQM output files

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/2944/console Started: 2019/10/15 14:28

[cmsbuild]

+1
Tested at: 4d4f51c
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-649856/2944/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-649856/2944/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 32
• DQMHistoTests: Total histograms compared: 3203312
• DQMHistoTests: Total failures: 2
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3203113
• DQMHistoTests: Total skipped: 197
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 31 files compared)
• Checked 134 log files, 14 edm output root files, 32 DQM output files

[slava77]

@rchudasa
please update the PR description to reflect the full list of backports.
Here is what I see, including the merge dates in the master:

• updating SC reg corrector for UL regression #26763 updating SC reg corrector for UL (May 16)
• Adapting EGRegression Modifiers to 106X and adding V3 modifier #26892 adding V3 modifier (Jun 13)
• Light by light modifier 1100 pre1 #27633 Light by light modifier (Aug 22)
• Filtering on pixel and strip clusters #27871 tracker cluster cut (Aug 27)
• Adding the regression config for light by light #28051 low-pt regression (Oct 15)

[slava77]

+1

for #27898 4d4f51c

• backports of updating SC reg corrector for UL regression #26763, Adapting EGRegression Modifiers to 106X and adding V3 modifier #26892, Light by light modifier 1100 pre1 #27633, Filtering on pixel and strip clusters #27871, Adding the regression config for light by light #28051 are correct (the combined diff is not the same due to some differences introduced in the context, e.g. in RecoEgamma/EgammaElectronProducers cleanup #27477 and FillDescriptions for all 3 GsfElectronProducers #25574)
• code changes are (somewhat) in line with the PR description; more backports were added since the PR was opened Backport: Light by light modifier PR 103X #27898 (comment). This PR introduces changes needed for a special workflow to be enabled with --procModifiers egamma_lowPt_exclusive and does not affect the standard workflows.
• jenkins tests pass and comparisons with the baseline show no differences
• local tests with a reference file from HIForward2018A run 326854 using 103X_dataRun2_Prompt_LowPtPhotonReg_v1 shows roughly expected behavior
  • CPU time is generally down, mostly by trimming the processing tails in tracking: average time on 10K events is down from 0.43 s/ev to 0.35 s/ev and the max time from 613 to 13 s/ev
  • max RSS on 8-thread job is down from 5.4 GiB to 3.1 GiB (due to trimmings starting from tracking)
  • AOD output size increased by about 50% from 20 to 32 kiB/ev driven by caloTowers and siPixelClusters

perhaps the most striking is the increase in the number of gedPhotons

Here in 103X with this PR and 103X_dataRun2_Prompt_LowPtPhotonReg_v1 we have about 0.68 photons per event, compared to my last test in 11X/master of #28051 using 110X_dataRun2_PromptLike_HI_Candidate_2019_09_30_11_30_43 with 0.53 photons per event (on the same input file). It seems like either some of the egamma calibrations are substantially different or there may be some other differences affecting the yield of photons in the two releases.

Comparing the distributions in the 11X version with 103X, the caloTowers look fairly similar.
It seems like most of the differences are in the endcaps.
Here are a few plots comparing the master version (in black) with this PR (in red)
The EE SC energy shape looks similar

EE SC corrected E uncertainty is much larger in 103X

The differences are possibly due to peculiarities in the GTs used in the tests. The GTs are set outside of the reco PRs.

[fabiocos]

+operations

the backport of the Configuration/ProcessModifier update is coherent with the master version

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_10_3_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_11_0_X is complete. This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

[fabiocos]

+1