[NanoAOD, JMENanoAOD] Remove jercVars variables, update puId config #33824

nurfikri89 · 2021-05-24T17:49:13Z

PR description:

This PR aims to

Remove certain jercVars variables from the main NanoAOD but still keep them in the JMENano.
Update configurations for pileup Id in JMENano.

With this PR, the event size for the main Nano is reduced from 2.90 kb/event to 2.86 kb/event. The comparison was made using 5K events from a TTJets RunIISummer20UL17MiniAODv2 sample. No event size changes for JMENano.

if this PR is a backport please specify the original PR and why you need to backport that PR:

Once merged, this PR needs to be backported to 11_2_X and 10_6_X, for the Ultra-Legacy NanoAODv9 production.

…E nano. Update puId configuration for JME nano.

nurfikri89 · 2021-05-24T17:50:22Z

FYI @kirschen

cmsbuild · 2021-05-24T17:56:14Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-33824/22834

This PR adds an extra 28KB to repository

cmsbuild · 2021-05-24T17:56:34Z

A new Pull Request was created by @nurfikri89 (Nurfikri Norjoharuddeen) for master.

It involves the following packages:

PhysicsTools/NanoAOD

@cmsbuild, @mariadalfonso, @gouskos, @fgolf can you please review it and eventually sign? Thanks.
@gpetruc, @swertz this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

mariadalfonso · 2021-05-24T21:50:38Z

please test

cmsbuild · 2021-05-25T00:49:07Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-daf735/15277/summary.html
COMMIT: e3e8e94
CMSSW: CMSSW_12_0_X_2021-05-24-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/33824/15277/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 102 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2650486
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2650457
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 36 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 155 log files, 37 edm output root files, 37 DQM output files
TriggerResults: no differences found

mariadalfonso · 2021-05-25T08:07:39Z

please test workflow 25202.15,10224.15,11024.15

(to test JMEnano workflow)

cmsbuild · 2021-05-25T14:08:22Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-daf735/15285/summary.html
COMMIT: e3e8e94
CMSSW: CMSSW_12_0_X_2021-05-24-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/33824/15285/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-daf735/10224.15_TTbar_13+2017PU_JMENano+TTbar_13TeV_TuneCUETP8M1_GenSim+DigiPU+RecoFakeHLTPU+HARVESTFakeHLTPU+Nano
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-daf735/11024.15_TTbar_13+2018PU_JMENano+TTbar_13TeV_TuneCUETP8M1_GenSim+DigiPU+RecoFakeHLTPU+HARVESTFakeHLTPU+Nano
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-daf735/25202.15_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25+NANOUP15MC_PU25_JME

Summary:

No significant changes to the logs found
Reco comparison results: 99 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2650486
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2650463
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 36 files compared)
Checked 155 log files, 37 edm output root files, 37 DQM output files
TriggerResults: no differences found

mariadalfonso · 2021-05-25T14:16:00Z

PhysicsTools/NanoAOD/python/custom_jme_cff.py

  run2_jme_2016.toModify(proc.jetTable.variables, puIdDisc = Var("userFloat('puId80XDisc')",float,doc="Pilup ID discriminant with 80X (2016) training",precision=10))
+  run2_jme_2016.toModify(proc.jetTable.variables, puId = Var("userInt('puId80XfullId')",int,doc="Pileup ID flags with 80X (2016) training"))


what do you intent to do in the jmenano ?

the full set of puget Id as stored in central nano are more complex depending on the nano version
https://github.com/cms-sw/cmssw/blob/master/PhysicsTools/NanoAOD/python/jets_cff.py#L268-L278

For the reclustered AK4 CHS jets in JMEnano, we would like the Pileup Id discriminant and flags to be set according to their respective training campaigns. The settings here from the main (https://github.com/cms-sw/cmssw/blob/master/PhysicsTools/NanoAOD/python/jets_cff.py#L268-L278) does not need to be repeated again in JMEnano. This line is needed at the moment because the UL2016 training is still not included in the main nano.

mariadalfonso · 2021-05-25T14:16:37Z

PhysicsTools/NanoAOD/python/jets_cff.py

-         chFPV1EF = cms.InputTag("jercVars:chargedFromPV1EnergyFraction"),
-         chFPV2EF = cms.InputTag("jercVars:chargedFromPV2EnergyFraction"),
-         chFPV3EF = cms.InputTag("jercVars:chargedFromPV3EnergyFraction"),


These variable need to be removed from the DQM as well

Apologies. Will remove them from the DQM.

cmsbuild · 2021-05-25T15:41:39Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-33824/22853

This PR adds an extra 24KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File PhysicsTools/NanoAOD/python/jets_cff.py modified in PR(s): Including jetId criteria for UL16 #33826

cmsbuild · 2021-05-25T15:42:07Z

Pull request #33824 was updated. @cmsbuild, @mariadalfonso, @gouskos, @fgolf can you please check and sign again.

mariadalfonso · 2021-05-25T15:55:09Z

please test

mariadalfonso · 2021-05-25T15:56:15Z

this PR save 1 or 2% of space
you can prepare the backport in the mean time

cmsbuild · 2021-05-25T18:49:11Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-daf735/15297/summary.html
COMMIT: 8677d78
CMSSW: CMSSW_12_0_X_2021-05-24-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/33824/15297/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 103 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2650486
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2650457
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -2.973 KiB( 36 files compared)
DQMHistoSizes: changed ( 1325.81 ): -1.857 KiB Physics/NanoAODDQM
DQMHistoSizes: changed ( 136.8523 ): -1.116 KiB Physics/NanoAODDQM
Checked 155 log files, 37 edm output root files, 37 DQM output files
TriggerResults: no differences found

mariadalfonso · 2021-05-26T08:24:01Z

+xpog

moved variable from central nano to jmenano
https://gitlab.cern.ch/cms-nanoAOD/nanoAOD-integration/-/issues/90

cmsbuild · 2021-05-26T08:24:23Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

qliphy · 2021-05-26T08:43:51Z

+1

…cVars_PuId [NanoAOD, JMENanoAOD] Backport of #33824 (Remove jercVars variables, update puId config) to 11_2_X

…cVars_PuId [NanoAOD, JMENanoAOD] Backport of #33824 (Remove jercVars variables, update puId config) to 10_6_X

Remove certain jercVars variables from main nano and store only in JM…

e3e8e94

…E nano. Update puId configuration for JME nano.

cmsbuild added this to the CMSSW_12_0_X milestone May 24, 2021

cmsbuild added code-checks-pending orp-pending pending-signatures tests-pending xpog-pending labels May 24, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels May 24, 2021

cmsbuild added tests-started and removed tests-pending labels May 24, 2021

mariadalfonso mentioned this pull request May 24, 2021

features for V9 nanoAOD cms-nanoAOD/cmssw#555

Closed

55 tasks

cmsbuild added tests-approved and removed tests-started labels May 25, 2021

cmsbuild added tests-started and removed tests-approved labels May 25, 2021

cmsbuild added tests-approved and removed tests-started labels May 25, 2021

mariadalfonso reviewed May 25, 2021

View reviewed changes

cmsbuild mentioned this pull request May 25, 2021

Including jetId criteria for UL16 #33826

Merged

Remove some jercVars from nanoDQM

8677d78

cmsbuild removed code-checks-approved tests-approved labels May 25, 2021

cmsbuild added code-checks-pending tests-pending labels May 25, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels May 25, 2021

cmsbuild added tests-started and removed tests-pending labels May 25, 2021

This was referenced May 25, 2021

[NanoAOD, JMENanoAOD] Backport of #33824 (Remove jercVars variables, update puId config) to 11_2_X #33834

Merged

[NanoAOD, JMENanoAOD] Backport of #33824 (Remove jercVars variables, update puId config) to 10_6_X #33835

Merged

cmsbuild added tests-approved and removed tests-started labels May 25, 2021

cmsbuild mentioned this pull request May 25, 2021

Fix fatJetTable genJetAK8Idx matching #33839

Merged

cmsbuild added fully-signed xpog-approved and removed pending-signatures xpog-pending labels May 26, 2021

cmsbuild added orp-approved and removed orp-pending labels May 26, 2021

cmsbuild merged commit 7971fe9 into cms-sw:master May 26, 2021

cmsbuild added a commit that referenced this pull request May 26, 2021

Merge pull request #33834 from nurfikri89/portFrom120XTo112X_nano_jer…

0d904de

…cVars_PuId [NanoAOD, JMENanoAOD] Backport of #33824 (Remove jercVars variables, update puId config) to 11_2_X

cmsbuild mentioned this pull request May 26, 2021

Rename pixelVertexCUDA|SoA to pixelVerticesCUDA|SoA #33844

Merged

cmsbuild added a commit that referenced this pull request May 27, 2021

Merge pull request #33835 from nurfikri89/portFrom120XTo106X_nano_jer…

2b9e19f

…cVars_PuId [NanoAOD, JMENanoAOD] Backport of #33824 (Remove jercVars variables, update puId config) to 10_6_X

nurfikri89 deleted the from120X_202105201100_nano_jercVars_PuId branch July 15, 2022 04:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[NanoAOD, JMENanoAOD] Remove jercVars variables, update puId config #33824

[NanoAOD, JMENanoAOD] Remove jercVars variables, update puId config #33824

nurfikri89 commented May 24, 2021

nurfikri89 commented May 24, 2021

cmsbuild commented May 24, 2021

cmsbuild commented May 24, 2021

mariadalfonso commented May 24, 2021

cmsbuild commented May 25, 2021

mariadalfonso commented May 25, 2021

cmsbuild commented May 25, 2021

mariadalfonso May 25, 2021

nurfikri89 May 25, 2021 •

edited

mariadalfonso May 25, 2021

nurfikri89 May 25, 2021

cmsbuild commented May 25, 2021

cmsbuild commented May 25, 2021

mariadalfonso commented May 25, 2021

mariadalfonso commented May 25, 2021

cmsbuild commented May 25, 2021

mariadalfonso commented May 26, 2021

cmsbuild commented May 26, 2021

qliphy commented May 26, 2021

		run2_jme_2016.toModify(proc.jetTable.variables, puIdDisc = Var("userFloat('puId80XDisc')",float,doc="Pilup ID discriminant with 80X (2016) training",precision=10))
		run2_jme_2016.toModify(proc.jetTable.variables, puId = Var("userInt('puId80XfullId')",int,doc="Pileup ID flags with 80X (2016) training"))

[NanoAOD, JMENanoAOD] Remove jercVars variables, update puId config #33824

[NanoAOD, JMENanoAOD] Remove jercVars variables, update puId config #33824

Conversation

nurfikri89 commented May 24, 2021

PR description:

if this PR is a backport please specify the original PR and why you need to backport that PR:

nurfikri89 commented May 24, 2021

cmsbuild commented May 24, 2021

cmsbuild commented May 24, 2021

mariadalfonso commented May 24, 2021

cmsbuild commented May 25, 2021

Comparison Summary

mariadalfonso commented May 25, 2021

cmsbuild commented May 25, 2021

Comparison Summary

mariadalfonso May 25, 2021

Choose a reason for hiding this comment

nurfikri89 May 25, 2021 • edited

Choose a reason for hiding this comment

mariadalfonso May 25, 2021

Choose a reason for hiding this comment

nurfikri89 May 25, 2021

Choose a reason for hiding this comment

cmsbuild commented May 25, 2021

cmsbuild commented May 25, 2021

mariadalfonso commented May 25, 2021

mariadalfonso commented May 25, 2021

cmsbuild commented May 25, 2021

Comparison Summary

mariadalfonso commented May 26, 2021

cmsbuild commented May 26, 2021

qliphy commented May 26, 2021

nurfikri89 May 25, 2021 •

edited