Change the default phase 2 PF reconstruction for endcap from simPF to pfTICL #32766

hatakeyamak · 2021-01-29T06:31:25Z

PR description:

This PR will change the default phase 2 PF reconstruction for endcap from simPF to pfTICL. This is based on the agreed plan between the PF group and HGCAL DPG that we will make this switch when #32291 is integrated (which happened a couple of days ago). The simPFProducer is still run for now, but not injected to the PF candidate list.

PR validation:

I ran this PR with the default setting (pfTICL) and also with --customise RecoParticleFlow/Configuration/RecoParticleFlow_cff.replaceTICLwithSimPF, and compared PF candidate outputs (actually packedCandidates) with those without this PR with the default (SimPF) and pfTICL injected. The test was done with 23234.0_TTbar 2026D49 without PU. I confirmed identical results when pfTICL or simPF is used for both without and with this PR.

if this PR is a backport please specify the original PR and why you need to backport that PR:

This is not a backport.

@bendavid @rovere @felicepantaleo @jnsandhya @zeratul87

…TICL.

cmsbuild · 2021-01-29T06:39:47Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-32766/20927

This PR adds an extra 16KB to repository

cmsbuild · 2021-01-29T06:40:13Z

A new Pull Request was created by @hatakeyamak (Kenichi Hatakeyama) for master.

It involves the following packages:

RecoHGCal/TICL
RecoParticleFlow/Configuration
RecoParticleFlow/PFProducer

@perrotta, @kpedro88, @cmsbuild, @srimanob, @slava77, @jpata can you please review it and eventually sign? Thanks.
@mmarionncern, @felicepantaleo, @rovere, @lgray, @apsallid, @sobhatta, @clelange, @lecriste, @cbernet, @seemasharmafnal this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

srimanob · 2021-01-29T07:09:38Z

Please test

cmsbuild · 2021-01-29T09:52:25Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0416e2/12594/summary.html
COMMIT: 914f1a9
CMSSW: CMSSW_11_3_X_2021-01-28-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32766/12594/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 3700 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2716596
DQMHistoTests: Total failures: 14353
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2702221
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 36 files compared)
Checked 156 log files, 37 edm output root files, 37 DQM output files

slava77 · 2021-01-29T14:33:10Z

enable profiling

slava77 · 2021-01-29T14:34:58Z

@cmsbuild please test

cmsbuild · 2021-01-29T17:35:29Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0416e2/12600/summary.html
COMMIT: 914f1a9
CMSSW: CMSSW_11_3_X_2021-01-28-2300/slc7_amd64_gcc900
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32766/12600/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 3700 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2716596
DQMHistoTests: Total failures: 14353
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2702221
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 36 files compared)
Checked 156 log files, 37 edm output root files, 37 DQM output files

jpata · 2021-02-02T11:42:36Z

Some remarks:

significantly fewer PFCandidates produced in the range 1.8<|eta|<3.0: https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_11_3_X_2021-01-28-2300+0416e2/41030/validateJR/all_OldVSNew_TTbar14TeV2026D49wf23234p0/all_OldVSNew_TTbar14TeV2026D49wf23234p0c_recoPFCandidates_particleFlow__RECO_obj_eta.png, consequently downstream quantities also change. @hatakeyamak can you confirm this is expected from the change?
only 2026D60 and 2026D49 workflows are affected
miniAOD size in 23434.21 goes up by ~3%, driven by patIsolatedTracks_isolatedTracks__PAT: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0416e2/12600/profiling/23434.21/products_miniAOD_sizes_compare_23434.21.txt, not clear how this is consistent with 1)
AOD size doesn't change
code changes LGTM

kpedro88 · 2021-02-02T14:45:26Z

Putting some plots here for posterity:

eta:

pt (showing that the decrease is mostly at low pt):

hatakeyamak · 2021-02-02T16:34:22Z

@jpata @kpedro88 thank you.

I certainly see smaller number of PF candidates in my private test with TICL. I believe the changes are mostly coming from lowpt neutrals which are sensitive to reconstruction details (that's what I see in my private test). @kpedro88 can the above plots be made for charged candidates and neutral candidates separately?

miniAOD size in 23434.21 goes up by ~3%, driven by patIsolatedTracks_isolatedTracks__PAT: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0416e2/12600/profiling/23434.21/products_miniAOD_sizes_compare_23434.21.txt, not clear how this is consistent with 1)

@jpata I think this is consistent with the fact that there a little reduced neutrals, which lead to somewhat increased number of isolated tracks.

I wonder @rovere has any additional comments/insight to offer.

kpedro88 · 2021-02-02T16:56:33Z

Here are the available plots generated for the comparison (of course, one can also download the output files from the matrix tests and make any plots):

pt (charged hadron)

pt (electron)

pt (photon)

pt (neutral hadron)

pt (HF hadron)

The changes are indeed concentrated in the photon and neutral hadron categories.

jpata · 2021-02-09T09:10:46Z

test parameters:

enable_tests = none

jpata · 2021-02-09T09:28:55Z

@cmsbuild please test

cmsbuild · 2021-02-09T18:16:09Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0416e2/12775/summary.html
COMMIT: 25bf846
CMSSW: CMSSW_11_3_X_2021-02-08-2300/slc7_amd64_gcc900
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32766/12775/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 3906 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2751765
DQMHistoTests: Total failures: 13939
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2737803
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.004 KiB( 36 files compared)
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 156 log files, 37 edm output root files, 37 DQM output files

smuzaffar · 2021-02-09T18:16:52Z

Profiling job again timeed out after 6 hours. For now I have explicitly marked the profiling status to pass so that you can get the pr test results

smuzaffar · 2021-02-09T18:30:26Z

@jpata , #32766 (comment) was the correct way to disable the profiling but there was a bug due to which bot was not overriding the #32766 (comment) request. This has been fixed now

jpata · 2021-02-10T07:12:27Z

+reconstruction

changes the default phase 2 PF reconstruction for endcap from simPF to pfTICL
reco changes observed: less neutrals and photons, changes to jet resolution, these are all expected
miniAOD size in 23434.21 goes up by ~3%
code changes are OK

srimanob · 2021-02-10T10:32:19Z

+Upgrade

OK for the code. But not clear on validation plan. What is the validation plan?
Should validation based on the same RAW, and compare between simPF and pfTICL with pre-release? High statistics?

cmsbuild · 2021-02-10T10:32:40Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

hatakeyamak · 2021-02-10T10:41:00Z

OK for the code. But not clear on validation plan. What is the validation plan?
Should validation based on the same RAW, and compare between simPF and pfTICL with pre-release? High statistics?

If this one goes into 11_3_0_pre3, the comparison w.r.t. pre2 will become a good validation for this change.

On Monday's PPD coordination, we requested:

if possible, flat QCD for phase 2: increase statistics from usual 50k to 100k events (both noPIU and PU)
add VBF H invisible with 200k events

for 11_3_0_pre3 and I think PdmV is looking into it. The same RAW (i.e. RECO only) will be helpful.

silviodonato · 2021-02-10T12:56:29Z

+1

gartung · 2021-02-10T19:24:19Z

Profiling job again timeed out after 6 hours. For now I have explicitly marked the profiling status to pass so that you can get the pr test results

@smuzaffar I added a timeout of 4 hours to the igprof commands in the profiling scripts.

Change the default phase 2 reconstruction for endcap from simPF to pf…

914f1a9

…TICL.

cmsbuild added this to the CMSSW_11_3_X milestone Jan 29, 2021

cmsbuild added code-checks-pending orp-pending pending-signatures reconstruction-pending tests-pending upgrade-pending labels Jan 29, 2021

hatakeyamak changed the title ~~Change the default phase 2 reconstruction for endcap from simPF to pfTICL~~ Change the default phase 2 PF reconstruction for endcap from simPF to pfTICL Jan 29, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Jan 29, 2021

cmsbuild added tests-started and removed tests-pending labels Jan 29, 2021

cmsbuild added tests-approved and removed tests-started labels Jan 29, 2021

cmsbuild added tests-started and removed tests-approved labels Jan 29, 2021

cmsbuild added tests-approved and removed tests-started labels Jan 29, 2021

cmsbuild added the tests-pending label Feb 9, 2021

cmsbuild added tests-started and removed tests-pending labels Feb 9, 2021

cmsbuild added tests-approved and removed tests-started labels Feb 9, 2021

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Feb 10, 2021

cmsbuild added fully-signed upgrade-approved and removed pending-signatures upgrade-pending labels Feb 10, 2021

cmsbuild added orp-approved and removed orp-pending labels Feb 10, 2021

cmsbuild merged commit 8377aea into cms-sw:master Feb 10, 2021

cmsbuild added tests-started tests-approved and removed tests-approved tests-started labels Feb 10, 2021

jpata mentioned this pull request Mar 5, 2021

PFBlock slower by factor 2x in Phase II #33083

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change the default phase 2 PF reconstruction for endcap from simPF to pfTICL #32766

Change the default phase 2 PF reconstruction for endcap from simPF to pfTICL #32766

hatakeyamak commented Jan 29, 2021 •

edited

cmsbuild commented Jan 29, 2021

cmsbuild commented Jan 29, 2021

srimanob commented Jan 29, 2021

cmsbuild commented Jan 29, 2021

slava77 commented Jan 29, 2021

slava77 commented Jan 29, 2021

cmsbuild commented Jan 29, 2021

jpata commented Feb 2, 2021

kpedro88 commented Feb 2, 2021

hatakeyamak commented Feb 2, 2021 •

edited

kpedro88 commented Feb 2, 2021

jpata commented Feb 9, 2021

jpata commented Feb 9, 2021

cmsbuild commented Feb 9, 2021

smuzaffar commented Feb 9, 2021 •

edited

smuzaffar commented Feb 9, 2021

jpata commented Feb 10, 2021

srimanob commented Feb 10, 2021

cmsbuild commented Feb 10, 2021

hatakeyamak commented Feb 10, 2021 •

edited

silviodonato commented Feb 10, 2021

gartung commented Feb 10, 2021

Change the default phase 2 PF reconstruction for endcap from simPF to pfTICL #32766

Change the default phase 2 PF reconstruction for endcap from simPF to pfTICL #32766

Conversation

hatakeyamak commented Jan 29, 2021 • edited

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented Jan 29, 2021

cmsbuild commented Jan 29, 2021

srimanob commented Jan 29, 2021

cmsbuild commented Jan 29, 2021

Comparison Summary

slava77 commented Jan 29, 2021

slava77 commented Jan 29, 2021

cmsbuild commented Jan 29, 2021

Comparison Summary

jpata commented Feb 2, 2021

kpedro88 commented Feb 2, 2021

hatakeyamak commented Feb 2, 2021 • edited

kpedro88 commented Feb 2, 2021

jpata commented Feb 9, 2021

jpata commented Feb 9, 2021

cmsbuild commented Feb 9, 2021

Comparison Summary

smuzaffar commented Feb 9, 2021 • edited

smuzaffar commented Feb 9, 2021

jpata commented Feb 10, 2021

srimanob commented Feb 10, 2021

cmsbuild commented Feb 10, 2021

hatakeyamak commented Feb 10, 2021 • edited

silviodonato commented Feb 10, 2021

gartung commented Feb 10, 2021

hatakeyamak commented Jan 29, 2021 •

edited

hatakeyamak commented Feb 2, 2021 •

edited

smuzaffar commented Feb 9, 2021 •

edited

hatakeyamak commented Feb 10, 2021 •

edited