PFlow fix/improvement in the HF region for phase2

[hatakeyamak]

PR description:

This PR introduces the first attempt of track-HF cluster matching in PFAlgo for phase 2. It shouldn't impact Run2 and Run3 reconstruction, but improves phase2 reconstruction. Before this PR, tracks and HF clusters in |eta|=3-4 region have been heavily double-counting charged particles in this region, which will be largely remedied in this PR. This PR introduces kdtree-based linking between tracks and HF clusters, and links are utilized in mostly createCandidatesHF of PFAlgo.

Also, the propagator is fixed to make the track propagation to HF to work for particles with pt<3 GeV also (which was broken in the past, as it didn't matter).

Also, the HF face is updated to be in agreement with the actual geometry.

PR validation:

http://hep.baylor.edu/hatake/PFval/PFForwardTrack/stack_offset_SinglePiPt50Eta3p45_3p54_2026D41_PU200_v0_vs_v5.pdf
shows the improved offset quantities. [Horns around |eta|=3-4 much reduced with this update.]

I made sure, using PF validation code, that 2018 QCD plots show no differences.

if this PR is a backport please specify the original PR:

This is not a backport.

@bendavid @jpata @iashvili @mariadalfonso

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28442/12853

• This PR adds an extra 180KB to repository

• Found files with invalid states:

  • RecoParticleFlow/PFProducer/plugins/kdtrees/KDTreeLinkerTrackHF.h:
    • Added: c494d1e
    • Deleted: a339298

Code check has found code style and quality issues which could be resolved by applying following patch(s)

• code-checks:
  https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28442/12853/code-checks.patch
  e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28442/12853/code-checks.patch | patch -p1
  You can also run scram build code-checks to apply code checks directly

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28442/12854

• This PR adds an extra 216KB to repository

• Found files with invalid states:

  • RecoParticleFlow/PFProducer/plugins/kdtrees/KDTreeLinkerTrackHF.h:
    • Added: c494d1e
    • Deleted: a339298

[cmsbuild]

A new Pull Request was created by @hatakeyamak (Kenichi Hatakeyama) for master.

It involves the following packages:

CommonTools/BaseParticlePropagator
RecoParticleFlow/PFProducer
RecoParticleFlow/PFTracking

@perrotta, @ssekmen, @lveldere, @sbein, @civanch, @mdhildreth, @cmsbuild, @slava77 can you please review it and eventually sign? Thanks.
@mmarionncern, @makortel, @rovere, @lgray, @seemasharmafnal, @cbernet, @bachtis this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

[perrotta]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/3568/console Started: 2019/11/21 09:18

[cmsbuild]

-1

Tested at: 2af5fe5

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f8d12e/3568/summary.html

I found follow errors while testing this PR

Failed tests: RelVals

• RelVals:

When I ran the RelVals I found an error in the following workflows:
20034.0 step3

runTheMatrix-results/20034.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D35_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D35+RecoFullGlobal_2026D35+HARVESTFullGlobal_2026D35/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D35_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D35+RecoFullGlobal_2026D35+HARVESTFullGlobal_2026D35.log

[cmsbuild]

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

[slava77]

the crash looks related

PFAlgo::createCandidatesHF(const reco::PFBlock&, reco::PFBlock::LinkData&, c...): 
Assertion `inds.hfEmIs.size() + inds.hfHadIs.size() == elements.size()'

[slava77]

looking at 1000 events in wf 23243.0 (QCD flat-PT, D49, no PU), things are roughly as expected (red is with this PR; black is CMSSW_11_1_X_2019-12-17-1100 out of the box)

• there are more charged hadrons and significantly fewer HF hadrons (and moderately fewer HF EMs) in the |eta| 3-4 range ; charged hadron appearance is restricted to pt>~3 GeV (as in the code)
  
  
  

almost all tracks in eta 3-4 are now attached to PF candidates, at least for the "lost tracks" selections (unclear if this may be a bit of an overkill)

pfMET is not affected significantly, although it looks a bit closer to gen

The jet response is down in the forward eta region

the fraction of the reco response well above 1 is much lower now, which is apparently in the right direction (removing double-counting of objects).

a simple jet distribution does not have horns in the eta 3-4 region anymore

[slava77]

I added a few more plots to the post above.
Almost complete disappearance of the lost tracks in eta 3-4 seems a bit suspicious https://user-images.githubusercontent.com/4676718/71046511-614d1100-20ed-11ea-86fc-57a1328467c8.png
either the tracks are so pure or the track-HF linking manages to link fakes successfully.
Note that the high-purity fake rate is still not zero in the eta range:

[slava77]

+1

for #28442 b07fda1

• code changes are in line with the PR description and the follow up review. The energy flow double-counting issue fixed in this PR is effectively a bug-fix and qualifies for a backport to 11_0.
• jenkins tests pass and comparisons with the baseline show differences only in the phase-2 workflows in plots related to PF candidates, concentrated in the eta range of 3 to 4. This is in agreement with the changes in the code.
• plots made in an extended test of 1K QCD flat-pt events show roughly expected behavior, in line with the PR description and the reco meeting https://indico.cern.ch/event/866456/contributions/3651206/attachments/1954041/3245106/PF_Reco_Nov29-2019.pdf

[Martin-Grunewald]

+1
(pending answer to my question)

[hatakeyamak]

+1
(pending answer to my question)

I added the answer above, but yes, your understanding is correct.

[Martin-Grunewald]

thanks

[kpedro88]

+upgrade

[fabiocos]

@ssekmen @sbein please have a look at this PR, it affects tangentially fastsim and it has been discussed for fast integration into 11_1_X and backport to 11_0_X in view of the HLT TDR production. I would like to integrate it asap, so please check and comment (in case of hot remarks) or sign it

[sbein]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

[fabiocos]

@hatakeyamak in order to prepare the backport requested by @fwyzard could you please provide already the PR there, so as we can launch the tests?

[slava77]

@hatakeyamak in order to prepare the backport requested by @fwyzard could you please provide already the PR there, so as we can launch the tests?

it looks like the same topic branch will work
https://github.com/cms-sw/cmssw/compare/CMSSW_11_0_X...hatakeyamak:Phase2PFTracksForward_v6?expand=1

[hatakeyamak]

@hatakeyamak in order to prepare the backport requested by @fwyzard could you please provide already the PR there, so as we can launch the tests?

it looks like the same topic branch will work
https://github.com/cms-sw/cmssw/compare/CMSSW_11_0_X...hatakeyamak:Phase2PFTracksForward_v6?expand=1

I had impression that I don't have to do anything as @slava77 indicated, but please let me know if anything is necessary for backport.

[slava77]

I had impression that I don't have to do anything as @slava77 indicated, but please let me know if anything is necessary for backport.

you still need to submit a pull request (even though you do not need to create a new branch)

[hatakeyamak]

I had impression that I don't have to do anything as @slava77 indicated, but please let me know if anything is necessary for backport.

you still need to submit a pull request (even though you do not need to create a new branch)

Ah, ok. Right, of course. I just did it. #28650

[fabiocos]

+1