Egamma PFID based on Tensorflow DNNs #35403

valsdav · 2021-09-24T13:30:17Z

PR description:

This PR introduces the DNN based Egamma particle flow ID evaluation based on the Tensorflow internal support.
Two helper classes, ElectronDNNEstimator and PhotonDNNEstimator, are included to handle the loading and evaluation of the DNN models for GsfElectron and GEDPhotons candidates.

Multiple models are evaluated for electrons and photons in different detector and energy regions. Batch evaluation is implemented to call each Tensorflow session only 1 time for event. The evaluation of the DNN models is configurable from python config and txt files: the list of variables to be used, their scaling parameters, the model files can be changed without changes to the code.

The DNNs are evaluated before the electron and photons candidates pass through the PFEgammaFilters, where the DNN values are available in the candidates to be used for the selection. Thresholds configuration for the PFid have been added to the PFEGammaFilter producer.

In the PR, the DNN based PFid has been activated by default and default models files have been included for the testing. Final model file will be provided in a separate PR after the finalization on the CMSSW_12_0 samples.

Presentations and documentation:

Physics talk in PPD: https://indico.cern.ch/event/1084340/contributions/4560857/attachments/2328176/3966582/PPDMeeting_14Oct2021.pdf
Integration of PF EGM ID in CMSSW https://indico.cern.ch/event/1081058/#4-integration-of-pf-egm-id-in
EGM PF validation studies https://indico.cern.ch/event/1081058/#5-egm-pf-validation-studies

PR validation:

Standard PF validation: https://akanugan.web.cern.ch/akanugan/PF_Validation_PFEGammaID_PR35403/
List of MINIAOD files and a few summary slides looking at the PF electrons & gedGsfElectrons in test vs reference are at:
https://cernbox.cern.ch/index.php/s/ki533KAwuWW1k3T

cmsbuild · 2021-09-24T13:38:15Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35403/25508

This PR adds an extra 868KB to repository
Found files with invalid states:
- RecoEgamma/ElectronIdentification/data/ElectronDNNScaler.txt:
  - Added: 7cd103b
  - Deleted: 9f23cd2
- RecoEgamma/ElectronIdentification/data/Ele_PFID_dnn/ElectronDNNScaler.txt:
  - Added: 3c2b7ba
  - Modified: dc76c55, 3e385ac, 9f23cd2
  - Deleted: 3caa230
- RecoEgamma/PhotonIdentification/data/Photon_PFID_dnn/PhotonDNNScaler.txt:
  - Added: dc76c55
  - Modified: 9f23cd2, 87e2640
  - Deleted: 3caa230
There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/PatCandidates/src/classes_def_objects.xml modified in PR(s): Add track-based isolation, recHit energy sum to IsolatedTrack class #31399
- File RecoParticleFlow/PFProducer/interface/PFEGammaFilters.h modified in PR(s): [EGM] Eta-Extended-Electrons (EEE) #35309
- File RecoParticleFlow/PFProducer/python/particleFlow_cff.py modified in PR(s): [EGM] Eta-Extended-Electrons (EEE) #35309
- File RecoParticleFlow/PFProducer/src/PFEGammaFilters.cc modified in PR(s): [EGM] Eta-Extended-Electrons (EEE) #35309

cmsbuild · 2021-09-24T13:38:41Z

A new Pull Request was created by @valsdav (Davide Valsecchi) for master.

It involves the following packages:

DataFormats/EgammaCandidates (reconstruction)
DataFormats/ParticleFlowCandidate (reconstruction)
DataFormats/PatCandidates (reconstruction)
RecoEgamma/EgammaElectronAlgos (reconstruction)
RecoEgamma/EgammaElectronProducers (reconstruction)
RecoEgamma/EgammaIsolationAlgos (reconstruction)
RecoEgamma/EgammaPhotonProducers (reconstruction)
RecoEgamma/ElectronIdentification (reconstruction)
RecoEgamma/PhotonIdentification (reconstruction)
RecoParticleFlow/PFProducer (reconstruction)

@jpata, @cmsbuild, @slava77 can you please review it and eventually sign? Thanks.
@mmarionncern, @Sam-Harper, @sobhatta, @afiqaize, @jainshilpi, @cbernet, @rovere, @lgray, @gouskos, @lecriste, @hatakeyamak, @gpetruc, @wrtabb, @varuns23, @seemasharmafnal, @ram1123 this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

slava77 · 2021-09-24T14:06:56Z

it would be nice to squash a few minor commits in the history.
42 is a bit large to start.

slava77 · 2021-09-24T14:09:45Z

Files in RecoEgamma/ElectronIdentification/data/Ele_PFID_dnn/ should be removed from the commit history and instead proposed as a PR to https://github.com/cms-data/RecoEgamma-ElectronIdentification

I notice that the model files are as small as 15 kB. Is the model really so small/simple or are these "dummy" files?

slava77 · 2021-09-24T14:10:31Z

@cmsbuild please test

slava77 · 2021-09-24T14:27:54Z

abort test

slava77 · 2021-09-24T14:30:02Z

enable profiling

slava77 · 2021-09-24T14:30:11Z

@cmsbuild please test

valsdav · 2021-09-24T14:49:20Z

HI @slava77

it would be nice to squash a few minor commits in the history.
42 is a bit large to start.

Of course, this can be done. Should I wait the end of the tests?

Files in RecoEgamma/ElectronIdentification/data/Ele_PFID_dnn/ should be removed from the commit history and instead proposed as a PR to https://github.com/cms-data/RecoEgamma-ElectronIdentification

I see, we can move them there.

I notice that the model files are as small as 15 kB. Is the model really so small/simple or are these "dummy" files?

They are so small because the models are quite simple (DNN with 128-64-64-32-16-1 nodes) and they have been saved in tensorflow graph format with the optimization performed by this nice tool from the CMS ML group: https://cms-ml.github.io/documentation/inference/tensorflow2.html#saving-your-model

slava77 · 2021-09-24T15:34:57Z

it would be nice to squash a few minor commits in the history.
42 is a bit large to start.

Of course, this can be done. Should I wait the end of the tests?

the build was already done, so the commits can be updated any time now.

cmsbuild · 2021-09-24T17:16:38Z

-1

Failed Tests: UnitTests RelVals AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a0d08a/18898/summary.html
COMMIT: 035207b
CMSSW: CMSSW_12_1_X_2021-09-24-1100/slc7_amd64_gcc900
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/35403/18898/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test testRecoMETMETProducers had ERRORS
---> test testTauEmbeddingProducers had ERRORS

RelVals

140.53

----- Begin Fatal Exception 24-Sep-2021 17:43:19 CEST-----------------------
An exception of category 'InvalidGraphDef' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=GsfElectronProducer label='gedGsfElectronsTmp'
Exception Message:
error while loading graphDef from 'RecoEgamma/ElectronIdentification/data/Ele_PFID_dnn/lowpT/lowpT_modelDNN.pb': Not found: RecoEgamma/ElectronIdentification/data/Ele_PFID_dnn/lowpT/lowpT_modelDNN.pb; No such file or directory
----- End Fatal Exception -------------------------------------------------

5.1

----- Begin Fatal Exception 24-Sep-2021 17:43:19 CEST-----------------------
An exception of category 'InvalidGraphDef' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=GsfElectronProducer label='gedGsfElectronsTmp'
Exception Message:
error while loading graphDef from 'RecoEgamma/ElectronIdentification/data/Ele_PFID_dnn/lowpT/lowpT_modelDNN.pb': Not found: RecoEgamma/ElectronIdentification/data/Ele_PFID_dnn/lowpT/lowpT_modelDNN.pb; No such file or directory
----- End Fatal Exception -------------------------------------------------

135.4

----- Begin Fatal Exception 24-Sep-2021 17:43:19 CEST-----------------------
An exception of category 'InvalidGraphDef' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=GsfElectronProducer label='gedGsfElectronsTmp'
Exception Message:
error while loading graphDef from 'RecoEgamma/ElectronIdentification/data/Ele_PFID_dnn/lowpT/lowpT_modelDNN.pb': Not found: RecoEgamma/ElectronIdentification/data/Ele_PFID_dnn/lowpT/lowpT_modelDNN.pb; No such file or directory
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

AddOn Tests

fastsim

----- Begin Fatal Exception 24-Sep-2021 17:43:14 CEST-----------------------
An exception of category 'InvalidGraphDef' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=GsfElectronProducer label='gedGsfElectronsTmp'
Exception Message:
error while loading graphDef from 'RecoEgamma/ElectronIdentification/data/Ele_PFID_dnn/lowpT/lowpT_modelDNN.pb': Not found: RecoEgamma/ElectronIdentification/data/Ele_PFID_dnn/lowpT/lowpT_modelDNN.pb; No such file or directory
----- End Fatal Exception -------------------------------------------------

fastsim1

----- Begin Fatal Exception 24-Sep-2021 17:43:14 CEST-----------------------
An exception of category 'InvalidGraphDef' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=GsfElectronProducer label='gedGsfElectronsTmp'
Exception Message:
error while loading graphDef from 'RecoEgamma/ElectronIdentification/data/Ele_PFID_dnn/lowpT/lowpT_modelDNN.pb': Not found: RecoEgamma/ElectronIdentification/data/Ele_PFID_dnn/lowpT/lowpT_modelDNN.pb; No such file or directory
----- End Fatal Exception -------------------------------------------------

fastsim2

----- Begin Fatal Exception 24-Sep-2021 17:43:14 CEST-----------------------
An exception of category 'InvalidGraphDef' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=GsfElectronProducer label='gedGsfElectronsTmp'
Exception Message:
error while loading graphDef from 'RecoEgamma/ElectronIdentification/data/Ele_PFID_dnn/lowpT/lowpT_modelDNN.pb': Not found: RecoEgamma/ElectronIdentification/data/Ele_PFID_dnn/lowpT/lowpT_modelDNN.pb; No such file or directory
----- End Fatal Exception -------------------------------------------------

Expand to see more addon errors ...

slava77 · 2021-09-24T17:27:58Z

RecoEgamma/ElectronIdentification/src/ElectronDNNEstimator.cc

+  tensorflow::setLogging(cfg_.log_level);
+  // load the graph definition
+  LogDebug("EleDNNPFid") << "Loading " << nModels_ << " graphs";
+  for (const auto& model_file : cfg_.models_files) {


I think that these need to be FileInPath instead of strings; apparently the tests are not running from /src directory.

hatakeyamak · 2021-09-24T21:41:20Z

Just tagging @akanugan here who is running some check of this PR in addition to jenkins.
BTW, he is seeing the same error as reported above from jenkins..

cmsbuild · 2021-10-14T16:06:55Z

Pull request #35403 was updated. @jpata, @cmsbuild, @slava77 can you please check and sign again.

jpata · 2021-10-15T12:04:32Z

@cmsbuild please test

jpata · 2021-10-15T14:03:05Z

@valsdav please update the PR description to make sure that all the slides (latest PF validations etc) are linked there

cmsbuild · 2021-10-16T14:05:44Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a0d08a/19675/summary.html
COMMIT: f45b48a
CMSSW: CMSSW_12_1_X_2021-10-15-2300/slc7_amd64_gcc900
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/35403/19675/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 5285 differences found in the comparisons
DQMHistoTests: Total files compared: 40
DQMHistoTests: Total histograms compared: 2751113
DQMHistoTests: Total failures: 3647
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2747444
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 39 files compared)
Checked 170 log files, 37 edm output root files, 40 DQM output files
TriggerResults: no differences found

jpata · 2021-10-18T07:42:15Z

+reconstruction

only technical changes since Egamma PFID based on Tensorflow DNNs #35403 (comment)

cmsbuild · 2021-10-18T07:42:36Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2021-10-18T12:50:05Z

+1

Needed externals for electrons and photons already merged

vlimant · 2023-10-05T12:35:56Z

RecoEgamma/EgammaPhotonProducers/python/gedPhotons_cfi.py

-    checkHcalStatus = cms.bool(True)
+    checkHcalStatus = cms.bool(True),
+    PhotonDNNPFid = cms.PSet(
+        enabled = cms.bool(False),


was this meant to be enabled ? ref #42949

its enabled via era, as it is meant to work only for run 3.

cmssw/RecoEgamma/EgammaPhotonProducers/python/gedPhotonSequence_cff.py

Lines 58 to 62 in aca7593

# Activate the Egamma PFID dnn only for Run3

from Configuration.Eras.Modifier_run3_common_cff import run3_common

run3_common.toModify(gedPhotonsTmp.PhotonDNNPFid,

enabled = True

)

cmsbuild added this to the CMSSW_12_1_X milestone Sep 24, 2021

cmsbuild added code-checks-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Sep 24, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Sep 24, 2021

cmsbuild added tests-started and removed tests-pending labels Sep 24, 2021

cmsbuild added tests-pending and removed tests-started labels Sep 24, 2021

cmsbuild added tests-started and removed tests-pending labels Sep 24, 2021

cmsbuild added tests-rejected and removed tests-started labels Sep 24, 2021

slava77 reviewed Sep 24, 2021

View reviewed changes

cmsbuild removed the tests-rejected label Sep 25, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 14, 2021

cmsbuild added tests-started and removed tests-pending labels Oct 15, 2021

cmsbuild added tests-approved and removed tests-started labels Oct 16, 2021

cmsbuild mentioned this pull request Oct 16, 2021

Clean unused dependencies from BuildFiles #35701

Merged

cmsbuild added fully-signed reconstruction-approved and removed reconstruction-pending pending-signatures labels Oct 18, 2021

cmsbuild added orp-approved and removed orp-pending labels Oct 18, 2021

cmsbuild merged commit 5c87671 into cms-sw:master Oct 18, 2021

makortel mentioned this pull request Oct 20, 2021

Module IB broken because of tensorflow/core/framework/tensor.h #35754

Closed

iarspider mentioned this pull request Oct 21, 2021

[ASAN] heap-buffer-overflow in GsfElectronProducer::produce #35760

Closed

swagata87 mentioned this pull request Nov 4, 2021

Fixing memory bug in GsfElectronProducer #35984

Merged

slava77 mentioned this pull request Nov 4, 2021

reco comparisons: add DNN ID plots for electrons and photons cms-sw/cms-bot#1659

Merged

swagata87 mentioned this pull request Nov 11, 2021

Appropriately fill PFCandidate::dnn_gamma_ data member in the constructors and copy operator #36096

Merged

a-kapoor mentioned this pull request May 10, 2022

Particle flow identification of electrons in extended eta #37880

Merged

valsdav mentioned this pull request May 10, 2022

DeepSC algorithm for ECAL SuperClusters #37115

Merged

swagata87 mentioned this pull request Oct 5, 2023

ID DNN score not properly filled for photons #42949

Closed

vlimant reviewed Oct 5, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Egamma PFID based on Tensorflow DNNs #35403

Egamma PFID based on Tensorflow DNNs #35403

valsdav commented Sep 24, 2021 •

edited

cmsbuild commented Sep 24, 2021

cmsbuild commented Sep 24, 2021

slava77 commented Sep 24, 2021

slava77 commented Sep 24, 2021

slava77 commented Sep 24, 2021

slava77 commented Sep 24, 2021

slava77 commented Sep 24, 2021

slava77 commented Sep 24, 2021

valsdav commented Sep 24, 2021

slava77 commented Sep 24, 2021

cmsbuild commented Sep 24, 2021

slava77 Sep 24, 2021

hatakeyamak commented Sep 24, 2021

cmsbuild commented Oct 14, 2021

jpata commented Oct 15, 2021

jpata commented Oct 15, 2021 •

edited

cmsbuild commented Oct 16, 2021

jpata commented Oct 18, 2021

cmsbuild commented Oct 18, 2021

perrotta commented Oct 18, 2021

vlimant Oct 5, 2023

swagata87 Oct 5, 2023

	# Activate the Egamma PFID dnn only for Run3
	from Configuration.Eras.Modifier_run3_common_cff import run3_common
	run3_common.toModify(gedPhotonsTmp.PhotonDNNPFid,
	enabled = True
	)

Egamma PFID based on Tensorflow DNNs #35403

Egamma PFID based on Tensorflow DNNs #35403

Conversation

valsdav commented Sep 24, 2021 • edited

PR description:

PR validation:

cmsbuild commented Sep 24, 2021

cmsbuild commented Sep 24, 2021

slava77 commented Sep 24, 2021

slava77 commented Sep 24, 2021

slava77 commented Sep 24, 2021

slava77 commented Sep 24, 2021

slava77 commented Sep 24, 2021

slava77 commented Sep 24, 2021

valsdav commented Sep 24, 2021

slava77 commented Sep 24, 2021

cmsbuild commented Sep 24, 2021

Unit Tests

RelVals

AddOn Tests

slava77 Sep 24, 2021

Choose a reason for hiding this comment

hatakeyamak commented Sep 24, 2021

cmsbuild commented Oct 14, 2021

jpata commented Oct 15, 2021

jpata commented Oct 15, 2021 • edited

cmsbuild commented Oct 16, 2021

Comparison Summary

jpata commented Oct 18, 2021

cmsbuild commented Oct 18, 2021

perrotta commented Oct 18, 2021

vlimant Oct 5, 2023

Choose a reason for hiding this comment

swagata87 Oct 5, 2023

Choose a reason for hiding this comment

valsdav commented Sep 24, 2021 •

edited

jpata commented Oct 15, 2021 •

edited