MLPF: first integration of EDProducer and training code

[jpata]

PR description:

This PR introduces an EDProducer using the MLPF approach, which create reco::PFCandidates, which can be used successfully in downstream CMSSW reconstruction, e.g. AK4 jet clustering.

It's not enabled by default, so nothing should be affected in standard workflows.

Here's an overview of the changes:

• in RecoParticleFlow/PFProducer, we add MLPFProducer using TensorFlow-CPU, (enabled only with the mlpf modifier, does not affect standard reco outputs even if enabled)
• the training & sample generation code is located in RecoParticleFlow/PFProducer/test/mlpf_training/
• in Validation/RecoParticleFlow, we modify the PFAnalysisNtuplizer, which was developed for and is currently used only by MLPF to generate training samples
• We add a process modifier mlpf and a new workflow 11834.13 where the TF-cpu evaluator of MLPF runs after standard PF and produces a new collection of PFCandidates in the RECO output.

Further work is planned to improve the reconstruction quality by retraining the model, and by integrating the ML-produced PFCandidates further with downstream reconstruction. The goal of this PR is to make the MLPF reconstruction approach easily testable and reviewable by the PF group & CMS more widely.

PR validation:

The latest status slides are here: https://indico.cern.ch/event/963885/#18-update-on-mlpf.

Can be validated using

$ runTheMatrix.py -l 11834.13
...
$ edmDumpEventContent --all 11834.13_*/step3_inRECOSIM.root | grep 'recoPFCandidates_mlpf'
vector<reco::PFCandidate>             "mlpfProducer"              ""                "RECO"         recoPFCandidates_mlpfProducer__RECO

Needs cms-data/RecoParticleFlow-PFProducer#2.

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-32048/19620

• This PR adds an extra 104KB to repository

• Found files with invalid states:

  • RecoParticleFlow/PFProducer/test/mlpf_training/run_tf.sh:
    • Added: 820662a
    • Deleted: 19c3309

[cmsbuild]

A new Pull Request was created by @jpata (Joosep Pata) for master.

It involves the following packages:

RecoParticleFlow/PFProducer
Validation/RecoParticleFlow

@perrotta, @andrius-k, @kmaeshima, @ErnestaP, @cmsbuild, @jfernan2, @fioriNTU, @slava77, @jpata can you please review it and eventually sign? Thanks.
@mmarionncern, @cbernet, @lgray, @seemasharmafnal, @hatakeyamak this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[jfernan2]

please test

[cmsbuild]

The tests are being triggered in jenkins.

• CMSSW_11_2_X_2020-11-06-1100/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/10554/console Started: 2020/11/06 16:43

[kpedro88]

This is a partial review, focusing on the SONIC part. (Some of the comments on the SONIC producer could also apply to the native producer.)

[cmsbuild]

+1
Tested at: 34a7931
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7ae4e2/10554/summary.html
CMSSW: CMSSW_11_2_X_2020-11-06-1100
SCRAM_ARCH: slc7_amd64_gcc820

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7ae4e2/10554/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 35
• DQMHistoTests: Total histograms compared: 2544144
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2544121
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 34 files compared)
• Checked 149 log files, 22 edm output root files, 35 DQM output files

[jpata]

+reconstruction

• implements the MLPF EDProducer (with the mlpf process modifier) and includes the GNN model training code
• adds a new workflow 11834.13 that enables the mlpf modifier for further studies
• igprof studies reported in MLPF: first integration of EDProducer and training code #32048 (comment)
• requires add MLPF weight file cms-data/RecoParticleFlow-PFProducer#2

[jfernan2]

+1

[kpedro88]

+upgrade

[chayanit]

+1

[jpata]

@cms-sw/operations-l2 kindly pinging

[qliphy]

+operations

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[qliphy]

+1