Add possibility to apply latest MET Phi corrections during MiniAOD processing #38461

michaelwassmer · 2022-06-22T07:56:42Z

PR description:

This PR is intended to allow the application of the latest MET Phi corrections when running over MiniAOD. The MET Phi corrections are intended to mitigate the observed azimuthal angle modulation of missing transverse momentum (MET). The corrections were derived by @lathomas and can be found at the following link: https://lathomas.web.cern.ch/lathomas/METStuff/XYCorrections/
This PR is just the CMSSW implementation of those corrections using an already existing module with some adaptions.
The effect of the PR can be seen in more detail in the following two presentations, which were given in the MET meeting:

Important: The PR is designed such that it shouldn't affect the default/current behavior at all.
For the PR to actually do something, the newly added 'campaign' and 'era' input arguments of the MET tool need to be explicitly used. This is actually one thing, where the input of experts would be greatly appreciated. It would probably be preferred to automatically retrieve the campaign and era, but at the moment I'm not sure how this could be achieved.

Changes:

JetMETCorrections/Type1MET/plugins/MultShiftMETcorrInputProducer.cc

Added possibility to also use the total number of primary vertices as the variable the Phi corrections depend on
Removed the PF candidate loop if the corrections are done as a function of a variable which doesn't use the PF candidates

Added several python configuration files containing the MET Phi corrections
PhysicsTools/PatUtils/python/patPFMETCorrections_cff.py

Added the corrections from the configuration files as objects to be able to use them

PhysicsTools/PatUtils/python/tools/runMETCorrectionsAndUncertainties.py

Allow to use the new corrections in the MET module if the newly added 'era' and 'campaign' input arguments are explicitly used
The default behavior should stay untouched for now

PR validation:

Successfully performed code-checks and code-format.
Besides some file read/open errors, the runTheMatrix.py -l limited -i all command also seemed to work successfully.

…ertices

cmsbuild · 2022-06-22T08:04:18Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-38461/30671

This PR adds an extra 44KB to repository

cmsbuild · 2022-06-22T08:04:38Z

A new Pull Request was created by @michaelwassmer for master.

It involves the following packages:

JetMETCorrections/Type1MET (reconstruction)
PhysicsTools/PatUtils (reconstruction)

@jpata, @cmsbuild, @clacaputo can you please review it and eventually sign? Thanks.
@AlexDeMoor, @rappoccio, @gouskos, @jdolen, @ahinzmann, @schoef, @mmarionncern, @jdamgov, @JyothsnaKomaragiri, @nhanvtran, @gkasieczka, @clelange, @emilbols, @hatakeyamak, @gpetruc, @andrzejnovak, @demuller, @mariadalfonso, @seemasharmafnal this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

jpata · 2022-06-22T11:29:37Z

@cmsbuild please test

cmsbuild · 2022-06-22T15:49:03Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ebeba3/25695/summary.html
COMMIT: 3421775
CMSSW: CMSSW_12_5_X_2022-06-22-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/38461/25695/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3659099
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3659069
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 49 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

jpata

This is actually one thing, where the input of experts would be greatly appreciated. It would probably be preferred to automatically retrieve the campaign and era, but at the moment I'm not sure how this could be achieved.

Don't we usually configure this on the fly with the era-dependent toModify operations?

cmssw/PhysicsTools/PatAlgos/python/slimming/miniAOD_tools.py

Line 38 in 6d2f660

    
           run2_muon_2016.toModify( process.patMuons, effectiveAreaVec = [0.0735,0.0619,0.0465,0.0433,0.0577])

jpata · 2022-06-27T07:15:55Z

PhysicsTools/PatUtils/python/tools/runMETCorrectionsAndUncertainties.py

-            }
-
-        getattr(process, "patPFMetTxyCorr"+postfix).parameters = xyTags[corScheme+"_25ns"]
+        print("DEBUG ",campaign,datamc,era)


unintentionally left in?

Yes, I forgot that line. I just removed it and updated the PR.
Regarding the modifiers, are they also capable of differentiating between 'suberas' e.g. 2018B and 2018C? The corrections also differ between the these suberas.

jpata · 2022-06-28T06:40:32Z

Having a centrally available correction (option 2) would be preferred to a custom MiniAOD recipe.

tvami · 2022-06-28T12:37:21Z

@michaelwassmer let me understand the final goal here, is this supposed to be backported to 10_6_X so it's used for UL ?

michaelwassmer · 2022-06-30T09:02:45Z

@michaelwassmer let me understand the final goal here, is this supposed to be backported to 10_6_X so it's used for UL ?

This could be one use case to have phi corrected MET in the next MiniAOD interation out of the box. However, I asked the JME/MET conveners to join on Friday so we have their perspective as well.

jpata · 2022-07-04T08:26:00Z

@cmsbuild please test

cmsbuild · 2022-07-04T12:19:04Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ebeba3/25964/summary.html
COMMIT: cf9bcca
CMSSW: CMSSW_12_5_X_2022-07-03-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/38461/25964/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3654771
DQMHistoTests: Total failures: 14
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3654735
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

jpata · 2022-07-07T07:27:08Z

+reconstruction

possibility to enable MET phi corrections in private MINIAOD production
not enabled by default
a follow-up for enabling this via the global tag is necessary

tvami · 2022-07-07T11:33:27Z

@jpata I think your own hold is still at place :)

clacaputo · 2022-07-15T09:26:20Z

Can hold be removed only by the person that triggers it?

perrotta · 2022-07-15T09:28:53Z

unhold

cmsbuild · 2022-07-15T09:29:18Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2022-07-15T09:30:33Z

Can hold be removed only by the person that triggers it?

I was able to unhold, as I understanf this was a reco request

qliphy · 2022-07-16T01:03:54Z

please test
to refresh the results

cmsbuild · 2022-07-16T08:03:57Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ebeba3/26281/summary.html
COMMIT: cf9bcca
CMSSW: CMSSW_12_5_X_2022-07-15-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38461/26281/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3654053
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3654023
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 49 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

perrotta · 2022-07-16T13:00:47Z

+1

Intended for private analyses and productions

Michael Wassmer added 10 commits June 22, 2022 09:36

Added cfis for new MET XY corrections into CMSSW

4e92c62

Do not perform the loop over PF candidates if not necessary

e8dd8e7

change naming scheme for xy corrections

9ffaa8b

Introduce new varType for MET XY corrections with number of primary v…

486b851

…ertices

introduce new inputs for met correction tool

eb60218

Upload MET Phi corrections for PuppiMET for UL MC

bd0cbc2

Upload MET Phi corrections for PuppiMET for UL DATA

4bf300b

Puppi XY corrections

c4b6a5c

allow XY MET corrections also for Puppi

5660e18

code-format

3421775

cmsbuild added this to the CMSSW_12_5_X milestone Jun 22, 2022

cmsbuild added code-checks-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Jun 22, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 22, 2022

michaelwassmer marked this pull request as ready for review June 22, 2022 10:18

cmsbuild added tests-started and removed tests-pending labels Jun 22, 2022

cmsbuild added tests-approved and removed tests-started labels Jun 22, 2022

jpata reviewed Jun 27, 2022

View reviewed changes

Remove forgotten debug output in runMETCorrectionsAndUncertainties.py

cf9bcca

cmsbuild removed the tests-approved label Jun 27, 2022

cmsbuild added tests-started and removed tests-pending labels Jul 4, 2022

cmsbuild added tests-approved and removed tests-started labels Jul 4, 2022

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Jul 7, 2022

cmsbuild added fully-signed and removed pending-signatures hold labels Jul 15, 2022

cmsbuild added tests-started and removed tests-approved labels Jul 16, 2022

cmsbuild added tests-approved and removed tests-started labels Jul 16, 2022

cmsbuild added orp-approved and removed orp-pending labels Jul 16, 2022

cmsbuild merged commit 8ab252c into cms-sw:master Jul 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add possibility to apply latest MET Phi corrections during MiniAOD processing #38461

Add possibility to apply latest MET Phi corrections during MiniAOD processing #38461

michaelwassmer commented Jun 22, 2022 •

edited

cmsbuild commented Jun 22, 2022

cmsbuild commented Jun 22, 2022

jpata commented Jun 22, 2022

cmsbuild commented Jun 22, 2022

jpata left a comment

jpata Jun 27, 2022

michaelwassmer Jun 27, 2022

jpata commented Jun 28, 2022

tvami commented Jun 28, 2022

michaelwassmer commented Jun 30, 2022

jpata commented Jul 4, 2022

cmsbuild commented Jul 4, 2022

jpata commented Jul 7, 2022

tvami commented Jul 7, 2022

clacaputo commented Jul 15, 2022

perrotta commented Jul 15, 2022

cmsbuild commented Jul 15, 2022

perrotta commented Jul 15, 2022

qliphy commented Jul 16, 2022

cmsbuild commented Jul 16, 2022

perrotta commented Jul 16, 2022

Add possibility to apply latest MET Phi corrections during MiniAOD processing #38461

Add possibility to apply latest MET Phi corrections during MiniAOD processing #38461

Conversation

michaelwassmer commented Jun 22, 2022 • edited

PR description:

PR validation:

cmsbuild commented Jun 22, 2022

cmsbuild commented Jun 22, 2022

jpata commented Jun 22, 2022

cmsbuild commented Jun 22, 2022

Comparison Summary

jpata left a comment

Choose a reason for hiding this comment

jpata Jun 27, 2022

Choose a reason for hiding this comment

michaelwassmer Jun 27, 2022

Choose a reason for hiding this comment

jpata commented Jun 28, 2022

tvami commented Jun 28, 2022

michaelwassmer commented Jun 30, 2022

jpata commented Jul 4, 2022

cmsbuild commented Jul 4, 2022

Comparison Summary

jpata commented Jul 7, 2022

tvami commented Jul 7, 2022

clacaputo commented Jul 15, 2022

perrotta commented Jul 15, 2022

cmsbuild commented Jul 15, 2022

perrotta commented Jul 15, 2022

qliphy commented Jul 16, 2022

cmsbuild commented Jul 16, 2022

Comparison Summary

perrotta commented Jul 16, 2022

michaelwassmer commented Jun 22, 2022 •

edited