Adding MVA ID to nanoAOD code #39355

cramonal · 2022-09-09T09:39:25Z

PR description:

This PR propagates the new muon identification approach using a MVA introduced in this PR to the nanoAOD:

This MVA has been presented in several Muon POG meetings: https://indico.cern.ch/event/1034615/#4-update-on-mva-id
This is already available in miniAOD, but in order to simplify the usage of this ID at analysis level it would be of interest to have this also available in nanoAOD.

In the output there are 3 new branches:

Muon_mvaIDMuon -> contains the MVA score reading it from the miniAOD.
Muon_mvaIDMuon_WP -> contains the miniAOD selectors with the definition of the proposed Working Points.
Muon_mvaIDMuon_nano -> contains the MVA score, recalculating it when running the nanoAOD.

PR validation:

We check the output of the new branch and Working Points running runTheMatrix and everything looks fine.
We execute the basic tests suggested in the CMSSW PR instructions

cmsbuild · 2022-09-09T09:48:13Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39355/32053

This PR adds an extra 16KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39355/32053/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39355/32053/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2022-09-09T10:20:03Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39355/32058

This PR adds an extra 20KB to repository

cmsbuild · 2022-09-09T10:20:21Z

A new Pull Request was created by @cramonal for master.

It involves the following packages:

PhysicsTools/NanoAOD (xpog)

@cmsbuild, @swertz, @vlimant can you please review it and eventually sign? Thanks.
@gpetruc, @swertz this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

swertz · 2022-09-12T14:10:55Z

Thanks @cramonal , a couple of things:

The new branches should be added to the DQM: https://github.com/cms-sw/cmssw/blob/master/PhysicsTools/NanoAOD/python/nanoDQM_cfi.py#L489
I do not understand why we need the MVA score to be both taken from mini and recomputed in nano, and stored twice. There should be a single branch, configured using modifiers (see e.g. an example here) to either take the existing value from mini, or to recompute it when running on older mini versions which don't have the MVAID stored. (the goal is to support the MVA ID both for Run 2 and Run 3, or only Run 3? please clarify)
You now have two different muon "MVA IDs" in nano, mvaId and mvaIDMuon, it's impossible for people to know which one to use. Is the first one still supported? If not, it should be removed. If it is, please clarify in the branch name or documentation which is which.
Can you clarify why it's need to have a dedicated EvaluateMuonMVAID module, can't the MuonMvaIDEstimator (from Adding muon MVA ID #36179) be reused somehow instead of duplicating the logic here?

cramonal · 2022-09-13T16:36:43Z

Thanks for the prompt review @swertz.

* I do not understand why we need the MVA score to be both taken from mini _and_ recomputed in nano, and stored twice. There should be a single branch, configured using modifiers (see e.g. an example [here](https://github.com/cms-sw/cmssw/blob/master/PhysicsTools/NanoAOD/python/muons_cff.py#L152)) to either take the existing value from mini, or to recompute it when running on older mini versions which don't have the MVAID stored. (the goal is to support the MVA ID both for Run 2 and Run 3, or only Run 3? please clarify)

I agree with this comment. Can you point me to the era modifier needed for processing miniv2?

swertz · 2022-09-14T08:16:32Z

Can you point me to the era modifier needed for processing miniv2?

Before I can answer that, could you clarify whether that new mvaID will be supported also for Run 2 UL (which would clearly be good; but means it needs to be re-run on Run 2 miniV2 samples) or whether it is only made available for Run 3?

cramonal · 2022-09-14T08:30:08Z

Before I can answer that, could you clarify whether that new mvaID will be supported also for Run 2 UL (which would clearly be good; but means it needs to be re-run on Run 2 miniV2 samples) or whether it is only made available for Run 3?

We would like this to be supported for Run 2 UL besides Run 3.

swertz · 2022-09-14T12:34:54Z

In that case you would need to insert the on-the-fly evaluation for these modifiers:

run2_miniAOD_80XLegacy | run2_nanoAOD_92X | run2_nanoAOD_94X2016 | run2_nanoAOD_94XMiniAODv1 |  run2_nanoAOD_94XMiniAODv2 | run2_nanoAOD_102Xv1 | run2_nanoAOD_106Xv1 | run2_nanoAOD_106Xv2 | run3_nanoAOD_122

While by default, for anything more recent (>=123X) it would be taken from mini.

swertz · 2022-09-28T07:33:51Z

@cramonal , any update on this? Is it clear to you how to proceed with the modifiers?

swertz · 2022-10-11T13:48:45Z

Hi, kind ping again @cramonal , or perhaps @cms-sw/muon-pog-l2 - what is the plan for this?

cramonal · 2022-10-11T15:27:22Z

Hi, kind ping again @cramonal , or perhaps @cms-sw/muon-pog-l2 - what is the plan for this?

Hi, we are working on this. Will update in the following days

swertz

This time the rebase worked @cramonal , but some small changes are still needed.

swertz · 2022-11-15T07:00:06Z

PhysicsTools/NanoAOD/python/muons_cff.py

@@ -45,6 +71,10 @@
     ),
 )

+(run2_miniAOD_80XLegacy | run2_nanoAOD_92X | run2_nanoAOD_94X2016 | run2_nanoAOD_94XMiniAODv1 | run2_nanoAOD_94XMiniAODv2\


This and the other cases below should just be run2_nanoAOD_106Xv2 | run3_nanoAOD_122

swertz · 2022-11-15T07:00:48Z

PhysicsTools/NanoAOD/python/muons_cff.py

        mvaLowPtId = Var("passed('LowPtMvaLoose')+passed('LowPtMvaMedium')","uint8", doc="Low Pt Mva ID from miniAOD selector (1=LowPtMvaLoose, 2=LowPtMvaMedium)"),
        miniIsoId = Var("passed('MiniIsoLoose')+passed('MiniIsoMedium')+passed('MiniIsoTight')+passed('MiniIsoVeryTight')","uint8",doc="MiniIso ID from miniAOD selector (1=MiniIsoLoose, 2=MiniIsoMedium, 3=MiniIsoTight, 4=MiniIsoVeryTight)"),
+        mvaIDMuon = Var("mvaIDValue()",float,doc="MVA-based ID score ",precision=6),
+        mvaIDMuon_WP = Var("passed('MvaIDwpMedium')+passed('MvaIDwpTight')","uint8",doc="MVA-based ID selector WPs (1=MVAIDwpMedium,2=MVAIDwpTight)"),


didn't we agree to remove the WP variable?

cramonal · 2022-11-15T12:56:07Z

Dear @swertz,

Sorry for the noise I was finalizing the changes. Everything should be in now.

Please note that we want to keep the WP branch but we want this to be computed in nanoAOD for all the cases.

Also I have renamed the branches in order to avoid having the expresion "Muon_*" twice in the name of the branch.

cmsbuild · 2022-11-15T13:01:04Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39355/33054

This PR adds an extra 32KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File PhysicsTools/NanoAOD/python/muons_cff.py modified in PR(s): HLT_Mu17_Photon30_IsoCaloId_v trigger filters backport to CMSSW_12_4_X #40034, HLT_Mu17_Photon30_IsoCaloId_v trigger filters backport to CMSSW_12_4_X #40054
- File PhysicsTools/NanoAOD/python/nanoDQM_cfi.py modified in PR(s): Run3 ele ID is added to the cleaned 126 nanoAOD electron module + DQM (Run3 EffArea is applied only to the EGamma supported variables) #40045, HLT_Mu17_Photon30_IsoCaloId_v trigger filters backport to CMSSW_12_4_X #40034, HLT_Mu17_Photon30_IsoCaloId_v trigger filters backport to CMSSW_12_4_X #40054, Cut based photon id run iii winter22 #39963

cmsbuild · 2022-11-15T13:01:51Z

Pull request #39355 was updated. @cmsbuild, @swertz, @vlimant can you please check and sign again.

swertz · 2022-11-15T15:07:20Z

please test

cmsbuild · 2022-11-16T11:45:08Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-774eca/29036/summary.html
COMMIT: e86bd08
CMSSW: CMSSW_12_6_X_2022-11-15-2300/el8_amd64_gcc10
Additional Tests: NANO
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39355/29036/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 8 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3417074
DQMHistoTests: Total failures: 9
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3417043
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 4.224 KiB( 47 files compared)
DQMHistoSizes: changed ( 11634.0,... ): 0.530 KiB Physics/NanoAODDQM
DQMHistoSizes: changed ( 13234.0,... ): 0.348 KiB Physics/NanoAODDQM
Checked 206 log files, 48 edm output root files, 48 DQM output files
TriggerResults: no differences found

NANO Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-774eca/2500.311_mc106Xul17v2+TTbarMINIAOD10.6_UL17v2+NANO_mc10.6ul17v2+HRV_NANO_mc
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-774eca/2500.312_mc106Xul17v2+TTbarMINIAOD10.6_UL18v2+NANO_mc10.6ul18v2+HRV_NANO_mc
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-774eca/2500.31_mc106Xul17v2+TTbarMINIAOD10.6_UL16v2+NANO_mc10.6ul16v2+HRV_NANO_mc
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-774eca/2500.331_data106Xul17v2+MuonEG2017MINIAOD10.6v2+NANO_data10.6ul17v2+HRV_NANO_data
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-774eca/2500.332_data106Xul18v2+MuonEG2018MINIAOD10.6v2+NANO_data10.6ul18v2+HRV_NANO_data
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-774eca/2500.33_data106Xul16v2+MuonEG2016MINIAOD10.6v2+NANO_data10.6ul16v2+HRV_NANO_data
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-774eca/2500.401_mc122Xrun3+TTbarMINIAOD12.2+NANO_mc12.2+HRV_NANO_mc
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-774eca/2500.4_mc122Xrun3_v10+TTbarMINIAOD12.2+NANO_mc12.2_v10+HRV_NANO_mc
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-774eca/2500.501_mc124Xrun3+TTbarMINIAOD12.4+NANO_mc12.4+HRV_NANO_mc
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-774eca/2500.511_data124Xrun3+MuonEG2022MINIAOD12.4+NANO_data12.4+HRV_NANO_data
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-774eca/2500.51_data124Xrun3_v10+MuonEG2022MINIAOD12.4+NANO_data12.4_v10+HRV_NANO_data
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-774eca/2500.5_mc124Xrun3_v10+TTbarMINIAOD12.4+NANO_mc12.4_v10+HRV_NANO_mc
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-774eca/2500.6_mc126X_v10+TTBarMINIAOD12.6+NANO_mc12.6_v10+HRV_NANO_mc

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 15
DQMHistoTests: Total histograms compared: 14740
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 14740
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 6.146 KiB( 14 files compared)
DQMHistoSizes: changed ( 2500.311,... ): 0.530 KiB Physics/NanoAODDQM
DQMHistoSizes: changed ( 2500.331,... ): 0.348 KiB Physics/NanoAODDQM
Checked 31 log files, 0 edm output root files, 15 DQM output files

Nano size comparison Summary:

Sample	kb/ev	ref kb/ev	diff kb/ev	ev/s/thd	ref ev/s/thd	diff rate	mem/thd	ref mem/thd
2500.31	2.206	2.204	0.002 ( +0.1% )	9.15	11.16	-18.0%	1.538	1.738
2500.311	2.327	2.324	0.003 ( +0.1% )	8.82	10.77	-18.2%	1.908	2.143
2500.312	2.279	2.276	0.002 ( +0.1% )	8.95	11.73	-23.7%	1.900	2.131
2500.33	1.102	1.099	0.003 ( +0.3% )	21.29	27.95	-23.8%	1.678	1.838
2500.331	1.399	1.396	0.004 ( +0.3% )	15.56	18.95	-17.9%	1.838	1.992
2500.332	1.331	1.328	0.003 ( +0.3% )	17.52	21.17	-17.2%	1.784	1.928
2500.4	2.115	2.115	0.000 ( +0.0% )	9.78	11.91	-17.9%	1.442	1.680
2500.401	2.116	2.113	0.003 ( +0.1% )	9.89	12.08	-18.2%	1.258	1.458
2500.5	0.302	0.302	0.000 ( +0.0% )	50.01	69.27	-27.8%	1.181	1.385
2500.501	0.302	0.302	0.000 ( +0.0% )	50.65	68.55	-26.1%	1.201	1.377
2500.51	1.087	1.087	0.000 ( +0.0% )	27.98	39.20	-28.6%	1.439	1.661
2500.511	1.095	1.092	0.003 ( +0.3% )	27.43	36.14	-24.1%	1.474	1.656
2500.6	1.418	1.418	0.000 ( +0.0% )	23.60	32.93	-28.3%	1.156	1.376
2500.601	1.424	1.422	0.002 ( +0.2% )	23.09	32.42	-28.8%	1.186	1.377

swertz · 2022-11-16T12:01:25Z

+xpog

New branches are visible in non-V10 workflows; no other changes as expected.

cmsbuild · 2022-11-16T12:02:02Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

rappoccio · 2022-11-16T13:10:16Z

+1

cmsbuild added this to the CMSSW_12_6_X milestone Sep 9, 2022

cmsbuild added code-checks-pending orp-pending pending-signatures tests-pending xpog-pending labels Sep 9, 2022

cmsbuild added code-checks-rejected code-checks-pending and removed code-checks-pending code-checks-rejected labels Sep 9, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Sep 9, 2022

swertz mentioned this pull request Sep 12, 2022

new features for V11 nanoAOD cms-nanoAOD/cmssw#602

Closed

1 task

cmsbuild mentioned this pull request Sep 20, 2022

LHE generic mass filter extension #39458

Closed

swertz mentioned this pull request Sep 26, 2022

MuonMVAid in nano cms-nanoAOD/cmssw#588

Closed

cmsbuild mentioned this pull request Oct 6, 2022

HLT Tau DQM from MiniAOD #39647

Closed

cramonal force-pushed the muonMVAID-inNano branch from a60ca3c to 8cc1144 Compare October 15, 2022 09:14

cmsbuild added code-checks-pending and removed code-checks-approved labels Oct 15, 2022

cramonal and others added 2 commits November 14, 2022 23:51

fix error in DQM plot

ae53a53

store muon mvaID as userFloat, evaluate WPs manually for Run2

62a59e2

swertz reviewed Nov 15, 2022

View reviewed changes

implement missing comments

e86bd08

cramonal force-pushed the muonMVAID-inNano branch from bdfca86 to e86bd08 Compare November 15, 2022 12:51

cmsbuild added code-checks-pending and removed code-checks-approved labels Nov 15, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 15, 2022

cmsbuild added tests-started and removed tests-pending labels Nov 15, 2022

cmsbuild added tests-approved and removed tests-started labels Nov 16, 2022

cmsbuild added fully-signed xpog-approved and removed pending-signatures xpog-pending labels Nov 16, 2022

cmsbuild added orp-approved and removed orp-pending labels Nov 16, 2022

cmsbuild merged commit f991752 into cms-sw:master Nov 16, 2022

This was referenced Nov 17, 2022

[ROOT626] Updated root to tip of branch v6-26-00-patches cms-sw/cmsdist#8179

Closed

[MkFit] Phase2 geometry & String configurable standard functions #39866

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding MVA ID to nanoAOD code #39355

Adding MVA ID to nanoAOD code #39355

cramonal commented Sep 9, 2022

cmsbuild commented Sep 9, 2022

cmsbuild commented Sep 9, 2022

cmsbuild commented Sep 9, 2022

swertz commented Sep 12, 2022

cramonal commented Sep 13, 2022

swertz commented Sep 14, 2022

cramonal commented Sep 14, 2022

swertz commented Sep 14, 2022

swertz commented Sep 28, 2022

swertz commented Oct 11, 2022

cramonal commented Oct 11, 2022

swertz left a comment

swertz Nov 15, 2022

swertz Nov 15, 2022

cramonal commented Nov 15, 2022

cmsbuild commented Nov 15, 2022

cmsbuild commented Nov 15, 2022

swertz commented Nov 15, 2022

cmsbuild commented Nov 16, 2022

swertz commented Nov 16, 2022

cmsbuild commented Nov 16, 2022

rappoccio commented Nov 16, 2022

Adding MVA ID to nanoAOD code #39355

Adding MVA ID to nanoAOD code #39355

Conversation

cramonal commented Sep 9, 2022

cmsbuild commented Sep 9, 2022

cmsbuild commented Sep 9, 2022

cmsbuild commented Sep 9, 2022

swertz commented Sep 12, 2022

cramonal commented Sep 13, 2022

swertz commented Sep 14, 2022

cramonal commented Sep 14, 2022

swertz commented Sep 14, 2022

swertz commented Sep 28, 2022

swertz commented Oct 11, 2022

cramonal commented Oct 11, 2022

swertz left a comment

Choose a reason for hiding this comment

swertz Nov 15, 2022

Choose a reason for hiding this comment

swertz Nov 15, 2022

Choose a reason for hiding this comment

cramonal commented Nov 15, 2022

cmsbuild commented Nov 15, 2022

cmsbuild commented Nov 15, 2022

swertz commented Nov 15, 2022

cmsbuild commented Nov 16, 2022

Comparison Summary

NANO Comparison Summary

swertz commented Nov 16, 2022

cmsbuild commented Nov 16, 2022

rappoccio commented Nov 16, 2022