GT Updates : PhaseII GT updated with the new Jet Energy Corrections

[arunhep]

Summary of changes in Global Tags

Upgrade

• PhaseII 2023 realistic scenario : 93X_upgrade2023_realistic_v2 as 93X_upgrade2023_realistic_v5 with the following changes:
  • update of Jet Energy Corrections mentioned here : GT Updates : PhaseII GT updated with the new Jet Energy Corrections #21370 (comment)

[cmsbuild]

A new Pull Request was created by @arunhep (Arun Kumar) for CMSSW_9_3_X.

It involves the following packages:

Configuration/AlCa

@ghellwig, @arunhep, @cerminar, @cmsbuild, @franzoni, @lpernie can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @mmusich, @ghellwig, @tocheng this is something you requested to watch as well.
@davidlange6, @slava77 you are the release manager for this.

cms-bot commands are listed here

[arunhep]

attn @kpedro88

[arunhep]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/24530/console Started: 2017/11/20 06:58

[cmsbuild]

-1

Tested at: 6eaa636

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
a52b8b5
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21370/24530/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21370/24530/git-merge-result

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21370/24530/summary.html

I found follow errors while testing this PR

Failed tests: RelVals

• RelVals:

When I ran the RelVals I found an error in the following worklfows:
20034.0 step3

runTheMatrix-results/20034.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D17_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D17+RecoFullGlobal_2023D17+HARVESTFullGlobal_2023D17/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D17_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D17+RecoFullGlobal_2023D17+HARVESTFullGlobal_2023D17.log

20434.0 step3

runTheMatrix-results/20434.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D19_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D19+RecoFullGlobal_2023D19+HARVESTFullGlobal_2023D19/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D19_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D19+RecoFullGlobal_2023D19+HARVESTFullGlobal_2023D19.log

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
a52b8b5
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21370/24530/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21370/24530/git-merge-result

[cmsbuild]

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

[arunhep]

@kpedro88
this is the error on step3 related to the JECs update

Failed to propagate track: Pt = 4.89723, eta = -3.07669, phi = -0.890748 to ECAL entrance !!
%MSG
----- Begin Fatal Exception 20-Nov-2017 08:29:05 CET-----------------------
An exception of category 'InvalidInput' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'MINIAODSIMoutput_step'
   [2] Prefetching for module PoolOutputModule/'MINIAODSIMoutput'
   [3] Prefetching for module PATMETSlimmer/'slimmedMETs'
   [4] Prefetching for module PATMETProducer/'patMETs'
   [5] Prefetching for module CorrectedPFMETProducer/'pfMetT1'
   [6] Prefetching for module PFJetMETcorrInputProducer/'corrPfMetType1'
   [7] Prefetching for module ChainedJetCorrectorProducer/'ak4PFCHSL1FastL2L3ResidualCorrector'
   [8] Calling method for module LXXXCorrectorProducer/'ak4PFCHSResidualCorrector'
Exception Message:
 cannot find key 8 in the JEC payload, this usually means you have to change the global tag
----- End Fatal Exception -------------------------------------------------
Another exception was caught while trying to clean up files after the primary fatal exception.
20-Nov-2017 08:29:08 CET  Closed file file:step2.root

[kpedro88]

@aperloff have you seen this error before?

[aperloff]

Yes, I have. It means that the L2L3Residual payload is missing from the GT (https://github.com/cms-sw/cmssw/blob/master/CondFormats/JetMETObjects/interface/JetCorrectorParameters.h#L141). Are you running on MC? The MC workflows might have that turned on because what we normally do is put in a placeholder for the L2L3Residuals which just returns a multiplicative factor of 1. It's not as s safe a workflow, but for normal JECs it works just fine. Still, I'm surprised an official workflow would take that chance. I didn't put in the dummy files for L2L3 since that hasn't, in the past, been necessary for PhaseII. Do I need to recreate the GT and re-upload?

[kpedro88]

@aperloff from the exception message, it looks like this is something for MET. Probably it doesn't need it, but if it's straightforward to remake the JEC records with dummy residual corrections, that would probably be faster than trying to debug the miniAOD configs.

[kpedro88]

Okay, I think this is actually a prefetching issue similar to #20057 - the product of ak4PFCHSL1FastL2L3ResidualCorrector is never actually requested because the request is behind a conditional (https://github.com/cms-sw/cmssw/blob/CMSSW_9_3_X/JetMETCorrections/Type1MET/interface/PFJetMETcorrInputProducerT.h#L169-L175), but it's still marked as a consume, so the framework attempts to prefetch it anyway in newer CMSSW versions. For now, easiest just to put in the dummy corrections.

[arunhep]

@kpedro88 @aperloff
any chance that these dummy tags are prepared and queued to 93X, 94X and 10.0.X queues just like the other ones today?

[kpedro88]

@arunhep I asked @aperloff to remake the records, but have not heard back yet. Hopefully soon.

[aperloff]

@arunhep @kpedro88 This is next on my list. Will be done soon.

[aperloff]

@arunhep @kpedro88 I have made the new tags and uploaded them to both the production and prep databases. They are:
JetCorrectorParametersCollection_PhaseIIFall17_V4_MC_AK8PFPuppi
JetCorrectorParametersCollection_PhaseIIFall17_V4_MC_AK8PFchs
JetCorrectorParametersCollection_PhaseIIFall17_V4_MC_AK8PF
JetCorrectorParametersCollection_PhaseIIFall17_V4_MC_AK4PFPuppi
JetCorrectorParametersCollection_PhaseIIFall17_V4_MC_AK4PFchs
JetCorrectorParametersCollection_PhaseIIFall17_V4_MC_AK4PF
@kpedro88 I have not queued them yet, nor have I requested a GT.

[kpedro88]

@aperloff thanks
@arunhep I submitted a new request with these tags

[cmsbuild]

Pull request #21370 was updated. @ghellwig, @arunhep, @cerminar, @cmsbuild, @franzoni, @lpernie can you please check and sign again.

[arunhep]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/24686/console Started: 2017/11/27 06:31

[arunhep]

@kpedro88 done !
To avoid too many iterations, its always good to check locally by running the PhaseII wf.

[kpedro88]

Yes, this was much more tedious than I expected. I will consider posting an issue for the MET code to add some triggers so unneeded/dummy JECs can be avoided in the future.

[cmsbuild]

+1
Tested at: 491d4be
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21370/24686/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
1e8e6c7
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21370/24686/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21370/24686/git-merge-result

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21370/24686/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 96 differences found in the comparisons
• DQMHistoTests: Total files compared: 25
• DQMHistoTests: Total histograms compared: 2624448
• DQMHistoTests: Total failures: 4174
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2620097
• DQMHistoTests: Total skipped: 177
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
• Checked 104 log files, 8 edm output root files, 25 DQM output files

[arunhep]

@kpedro88 @aperloff can you validate the changes observed in PhaseII workflows because of these changes in JECs?
https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_9_3_X_2017-11-26-0000+21370/23958/20034.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023D17_GenSimHLBeamSpotFull14+DigiFullTrigger_2023D17+RecoFullGlobal_2023D17+HARVESTFullGlobal_2023D17/

once you give me green light, I will sign this PR.

[kpedro88]

To me, the changes look consistent with the application of better JECs.

[kpedro88]

@arunhep please sign

[arunhep]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_9_3_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_10_0_X is complete. This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar (and backports should be raised in the release meeting by the corresponding L2)

[davidlange6]

hi all
coming back to loose ends - whats the plan for getting these changes into 10x. That should happen first.

[arunhep]

@davidlange6

hi all
coming back to loose ends - whats the plan for getting these changes into 10x. That should happen first.

here you go : #21647

[davidlange6]

+1