CHS and PUPPI track-vertex-association for Run3 #36639

ahinzmann · 2022-01-06T13:31:42Z

PR description:

Followup from #33885 and #35583.

Update track-vertex-association optimized for Run3 using uniform criteria for CHS jets, CHS MET, PUPPI jets and PUPPI MET.
Avoid running dedicated track-vertex-association module for JME to address issue slowdown in PFCandidatePrimaryVertexSorter in CMSSW_12_1_0_pre2 -> CMSSW_12_1_0_pre3 #35986. Instead use original modules for JME-specific track-vertex-association in PFPileUp.cc (CHS jets in AOD), PuppiProducer.cc (PUPPI jets in AOD) and CandPtrSelector (CHS MET in MiniAOD), taking parameter values from a single configuration file pfNoPileUpJME_cff.py.

PR validation:

scram b runtests
runTheMatrix -l limited

Validation on Run3 RelVal samples to be presented at JMAR on Jan 11th:
https://indico.cern.ch/event/1110013/

if this PR is a backport please specify the original PR and why you need to backport that PR:

No. Only meant for Run3.

cmsbuild · 2022-01-06T13:40:10Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36639/27656

This PR adds an extra 68KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File PhysicsTools/PatAlgos/python/slimming/miniAOD_tools.py modified in PR(s): Adding muon MVA ID #36179
- File PhysicsTools/PatAlgos/python/tools/jetCollectionTools.py modified in PR(s): [JMENanoAOD] Rework for Prompt #35614

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-checks:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36639/27656/code-checks.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36639/27656/code-checks.patch | patch -p1
You can also run scram build code-checks to apply code checks directly
code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36639/27656/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36639/27656/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2022-01-06T14:01:14Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36639/27657

ERROR: Build errors found during clang-tidy run.

CommonTools/ParticleFlow/plugins/PFPileUp.cc:180:58: error: expected unqualified-id [clang-diagnostic-error]
            (!(fNumOfPUVtxsForCharged_ > 0 && !vertices->->empty() && PVOrig.key() <= fNumOfPUVtxsForCharged_ &&
                                                         ^
Suppressed 1323 warnings (1319 in non-user code, 3 NOLINT, 1 with check filters).
--
gmake: *** [config/SCRAM/GMake/Makefile.coderules:129: code-checks] Error 2
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

ahinzmann · 2022-01-06T14:38:37Z

For the record, code-checks didn't work in this case, it proposed "->->", not sure if this can be fixed in code-checks.

`diff --git a/CommonTools/ParticleFlow/plugins/PFPileUp.cc b/CommonTools/ParticleFlow/plugins/PFPileUp.cc
index a473439807d4..a47f1965f2b7 100644
--- a/CommonTools/ParticleFlow/plugins/PFPileUp.cc
+++ b/CommonTools/ParticleFlow/plugins/PFPileUp.cc
@@ -177,7 +177,7 @@ void PFPileUp::produce(Event& iEvent, const EventSetup& iSetup) {
const reco::VertexRef& PVOrig = associatedPV[p];
int quality = associationQuality[p];
if ((PVOrig.isNonnull() && PVOrig.key() > 0 && PVOrig->ndof() > 4.0 && quality >= vertexAssociationQuality_) &&

       (!(fNumOfPUVtxsForCharged_ > 0 && vertices->size() > 0 && PVOrig.key() <= fNumOfPUVtxsForCharged_ &&

       (!(fNumOfPUVtxsForCharged_ > 0 && !vertices->->empty() && PVOrig.key() <= fNumOfPUVtxsForCharged_ &&
          std::abs(p->vertex().z() - vertices->at(0).z()) < fDzCutForChargedFromPUVtxs_)))
     pfCandidatesFromPU.push_back(p);
 }

`

cmsbuild · 2022-01-06T14:44:55Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36639/27658

This PR adds an extra 16KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File PhysicsTools/PatAlgos/python/slimming/miniAOD_tools.py modified in PR(s): Adding muon MVA ID #36179
- File PhysicsTools/PatAlgos/python/tools/jetCollectionTools.py modified in PR(s): [JMENanoAOD] Rework for Prompt #35614

cmsbuild · 2022-01-06T14:45:18Z

A new Pull Request was created by @ahinzmann for master.

It involves the following packages:

CommonTools/ParticleFlow (reconstruction)
CommonTools/PileupAlgos (reconstruction)
CommonTools/RecoAlgos (reconstruction)
PhysicsTools/PatAlgos (reconstruction)
PhysicsTools/PatUtils (reconstruction)
RecoMET/Configuration (reconstruction)

@jpata, @cmsbuild, @clacaputo, @slava77 can you please review it and eventually sign? Thanks.
@rappoccio, @gouskos, @abbiendi, @emilbols, @mbluj, @demuller, @seemasharmafnal, @mmarionncern, @missirol, @jhgoh, @jdolen, @azotz, @schoef, @jdamgov, @nhanvtran, @gkasieczka, @hatakeyamak, @mariadalfonso, @AlexDeMoor, @JyothsnaKomaragiri, @gpetruc, @andrzejnovak this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

jpata · 2022-01-06T15:42:15Z

enable profiling

jpata · 2022-01-06T15:46:56Z

@cmsbuild please test

cmsbuild · 2022-01-06T18:17:48Z

-1

Failed Tests: RelVals RelVals-INPUT AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-11f204/21555/summary.html
COMMIT: 01f7cf7
CMSSW: CMSSW_12_3_X_2022-01-06-1100/slc7_amd64_gcc10
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36639/21555/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

135.4

----- Begin Fatal Exception 06-Jan-2022 17:16:39 CET-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'prevalidation_step'
   [2] Prefetching for module JetFlavourClustering/'myak4JetFlavourInfos'
   [3] Prefetching for module FastjetJetProducer/'ak4PFJetsCHS'
   [4] Prefetching for module TPPFCandidatesOnPFCandidates/'pfNoPileUpJME'
   [5] Calling method for module PFPileUp/'pfPileUpJME'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: edm::Association<std::vector<reco::Vertex> >
Looking for module label: primaryVertexAssociation
Looking for productInstanceName: original

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "SkipEvent = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

5.1

----- Begin Fatal Exception 06-Jan-2022 17:16:41 CET-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'prevalidation_step'
   [2] Prefetching for module JetFlavourClustering/'myak4JetFlavourInfos'
   [3] Prefetching for module FastjetJetProducer/'ak4PFJetsCHS'
   [4] Prefetching for module TPPFCandidatesOnPFCandidates/'pfNoPileUpJME'
   [5] Calling method for module PFPileUp/'pfPileUpJME'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: edm::Association<std::vector<reco::Vertex> >
Looking for module label: primaryVertexAssociation
Looking for productInstanceName: original

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "SkipEvent = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

2018.1

----- Begin Fatal Exception 06-Jan-2022 17:22:22 CET-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'prevalidation_step'
   [2] Prefetching for module JetFlavourClustering/'myak4JetFlavourInfos'
   [3] Prefetching for module FastjetJetProducer/'ak4PFJetsCHS'
   [4] Prefetching for module TPPFCandidatesOnPFCandidates/'pfNoPileUpJME'
   [5] Calling method for module PFPileUp/'pfPileUpJME'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: edm::Association<std::vector<reco::Vertex> >
Looking for module label: primaryVertexAssociation
Looking for productInstanceName: original

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "SkipEvent = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

RelVals-INPUT

138.4138.4_PromptCollisions+RunMinimumBias2021+ALCARECOPROMPTR3+HARVESTDPROMPTR3/step2_PromptCollisions+RunMinimumBias2021+ALCARECOPROMPTR3+HARVESTDPROMPTR3.log
138.5138.5_ExpressCollisions+RunMinimumBias2021+TIER0EXPRUN3+ALCARECOEXPR3+HARVESTDEXPR3/step2_ExpressCollisions+RunMinimumBias2021+TIER0EXPRUN3+ALCARECOEXPR3+HARVESTDEXPR3.log
140.54140.54_RunPA2013+RunPA2013+RECO_PPbData+HARVEST_PPbData/step2_RunPA2013+RunPA2013+RECO_PPbData+HARVEST_PPbData.log

Expand to see more relval errors ...

AddOn Tests

fastsim

----- Begin Fatal Exception 06-Jan-2022 17:16:59 CET-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 3 stream: 3
   [1] Running path 'prevalidation_step'
   [2] Prefetching for module JetFlavourClustering/'myak4JetFlavourInfos'
   [3] Prefetching for module FastjetJetProducer/'ak4PFJetsCHS'
   [4] Prefetching for module TPPFCandidatesOnPFCandidates/'pfNoPileUpJME'
   [5] Calling method for module PFPileUp/'pfPileUpJME'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: edm::Association<std::vector<reco::Vertex> >
Looking for module label: primaryVertexAssociation
Looking for productInstanceName: original

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "SkipEvent = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

fastsim1

----- Begin Fatal Exception 06-Jan-2022 17:16:59 CET-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 2 stream: 3
   [1] Running path 'prevalidation_step'
   [2] Prefetching for module JetFlavourClustering/'myak4JetFlavourInfos'
   [3] Prefetching for module FastjetJetProducer/'ak4PFJetsCHS'
   [4] Prefetching for module TPPFCandidatesOnPFCandidates/'pfNoPileUpJME'
   [5] Calling method for module PFPileUp/'pfPileUpJME'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: edm::Association<std::vector<reco::Vertex> >
Looking for module label: primaryVertexAssociation
Looking for productInstanceName: original

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "SkipEvent = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

fastsim2

----- Begin Fatal Exception 06-Jan-2022 17:17:02 CET-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 1
   [1] Running path 'prevalidation_step'
   [2] Prefetching for module JetFlavourClustering/'myak4JetFlavourInfos'
   [3] Prefetching for module FastjetJetProducer/'ak4PFJetsCHS'
   [4] Prefetching for module TPPFCandidatesOnPFCandidates/'pfNoPileUpJME'
   [5] Calling method for module PFPileUp/'pfPileUpJME'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: edm::Association<std::vector<reco::Vertex> >
Looking for module label: primaryVertexAssociation
Looking for productInstanceName: original

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "SkipEvent = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

Expand to see more addon errors ...

cmsbuild · 2022-01-17T12:46:08Z

Pull request #36639 was updated. @jpata, @cmsbuild, @clacaputo, @slava77 can you please check and sign again.

jpata · 2022-01-17T12:55:45Z

@cmsbuild enable none

jpata · 2022-01-17T12:56:41Z

@cmsbuild please test

cmsbuild · 2022-01-17T23:31:11Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-11f204/21754/summary.html
COMMIT: 692bae3
CMSSW: CMSSW_12_3_X_2022-01-16-2300/slc7_amd64_gcc10
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36639/21754/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 7355 differences found in the comparisons
DQMHistoTests: Total files compared: 43
DQMHistoTests: Total histograms compared: 3464734
DQMHistoTests: Total failures: 5800
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3458911
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 42 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 181 log files, 42 edm output root files, 43 DQM output files
TriggerResults: found differences in 1 / 42 workflows

jpata · 2022-01-18T14:30:15Z

+reconstruction

on top of CHS and PUPPI track-vertex-association for Run3 #36639 (comment), fixes Enable->enable for some modules
reco changes limited to jet and MET (and downstream quantities) as expected

cmsbuild · 2022-01-18T14:30:34Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

qliphy · 2022-01-24T08:35:20Z

please test
to refresh the results

cmsbuild · 2022-01-24T14:41:14Z

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-11f204/21950/summary.html
COMMIT: 692bae3
CMSSW: CMSSW_12_3_X_2022-01-23-2300/slc7_amd64_gcc10
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/36639/21950/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

4.764.76_ZMuSkim2012D+ZMuSkim2012D+HLTDSKIM2+RECODR1reHLT2+HARVESTDR1reHLT/step2_ZMuSkim2012D+ZMuSkim2012D+HLTDSKIM2+RECODR1reHLT2+HARVESTDR1reHLT.log

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 7668 differences found in the comparisons
DQMHistoTests: Total files compared: 43
DQMHistoTests: Total histograms compared: 3449324
DQMHistoTests: Total failures: 5056
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3444246
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 42 files compared)
Checked 181 log files, 42 edm output root files, 43 DQM output files
TriggerResults: found differences in 1 / 42 workflows

qliphy · 2022-01-25T02:07:20Z

merge
addressing #35583 (comment)

cmsbuild added this to the CMSSW_12_3_X milestone Jan 6, 2022

cmsbuild added code-checks-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Jan 6, 2022

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jan 6, 2022

ahinzmann force-pushed the run3chs branch from abb6570 to c100e7d Compare January 6, 2022 13:51

cmsbuild added code-checks-pending and removed code-checks-rejected labels Jan 6, 2022

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jan 6, 2022

CHS and PUPPI track-vertex-association for Run3

01f7cf7

ahinzmann force-pushed the run3chs branch from c100e7d to 01f7cf7 Compare January 6, 2022 14:36

cmsbuild added code-checks-pending and removed code-checks-rejected labels Jan 6, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Jan 6, 2022

cmsbuild added tests-started and removed tests-pending labels Jan 6, 2022

cmsbuild removed the tests-started label Jan 6, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Jan 17, 2022

cmsbuild added tests-started and removed tests-pending labels Jan 17, 2022

cmsbuild added tests-approved and removed tests-started labels Jan 17, 2022

cmsbuild added fully-signed reconstruction-approved and removed reconstruction-pending pending-signatures labels Jan 18, 2022

cmsbuild added tests-started and removed tests-approved labels Jan 24, 2022

cmsbuild added tests-rejected and removed tests-started labels Jan 24, 2022

cmsbuild added orp-approved and removed orp-pending labels Jan 25, 2022

cmsbuild merged commit 5787b89 into cms-sw:master Jan 25, 2022

qliphy mentioned this pull request Jan 25, 2022

CHS/Puppi vertex unification follow-up #35583

Closed

missirol mentioned this pull request Jan 26, 2022

HLT menu development for 12_3_X #36807

Merged

cmsbuild mentioned this pull request Jan 26, 2022

Try a second time if pop fails in ReusableObjectHolder #36813

Closed

jpata mentioned this pull request Feb 8, 2022

slowdown in PFCandidatePrimaryVertexSorter in CMSSW_12_1_0_pre2 -> CMSSW_12_1_0_pre3 #35986

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CHS and PUPPI track-vertex-association for Run3 #36639

CHS and PUPPI track-vertex-association for Run3 #36639

ahinzmann commented Jan 6, 2022 •

edited

cmsbuild commented Jan 6, 2022

cmsbuild commented Jan 6, 2022

ahinzmann commented Jan 6, 2022 •

edited

cmsbuild commented Jan 6, 2022

cmsbuild commented Jan 6, 2022

jpata commented Jan 6, 2022

jpata commented Jan 6, 2022

cmsbuild commented Jan 6, 2022

cmsbuild commented Jan 17, 2022

jpata commented Jan 17, 2022 •

edited

jpata commented Jan 17, 2022

cmsbuild commented Jan 17, 2022

jpata commented Jan 18, 2022

cmsbuild commented Jan 18, 2022

qliphy commented Jan 24, 2022

cmsbuild commented Jan 24, 2022

qliphy commented Jan 25, 2022

CHS and PUPPI track-vertex-association for Run3 #36639

CHS and PUPPI track-vertex-association for Run3 #36639

Conversation

ahinzmann commented Jan 6, 2022 • edited

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented Jan 6, 2022

cmsbuild commented Jan 6, 2022

ahinzmann commented Jan 6, 2022 • edited

cmsbuild commented Jan 6, 2022

cmsbuild commented Jan 6, 2022

jpata commented Jan 6, 2022

jpata commented Jan 6, 2022

cmsbuild commented Jan 6, 2022

RelVals

RelVals-INPUT

AddOn Tests

cmsbuild commented Jan 17, 2022

jpata commented Jan 17, 2022 • edited

jpata commented Jan 17, 2022

cmsbuild commented Jan 17, 2022

Comparison Summary

jpata commented Jan 18, 2022

cmsbuild commented Jan 18, 2022

qliphy commented Jan 24, 2022

cmsbuild commented Jan 24, 2022

RelVals-INPUT

Comparison Summary

qliphy commented Jan 25, 2022

ahinzmann commented Jan 6, 2022 •

edited

ahinzmann commented Jan 6, 2022 •

edited

jpata commented Jan 17, 2022 •

edited