Integration of ParticleNET-HLT AK4 and AK8 DQM Monitoring

[rgerosa]

Dear All,

Description

this PR is meant to integrate in CMSSW a DQM workflow for monitoring the efficiency of AK4 and AK8 ParticleNet taggers at HLT with respect to their offline implementation. Having such monitor is important to assess the stability of the performance of the new triggers implemented for run3 dedicated to HH4b and boosted Hbb topologies. This PR contains a new DQM producer called ParticleJetTagMonitor.cc that handles the efficiency computation (definition of numerator and denominator) in the following way:

a) For AK4 PNET b-tagging, an enriched sample in ttbar emu events are selected offline requiring a emu-pair and at least 2 central jets with offline PNET b-tagging score above a certain threshold. These events are used to assess the HLT PNET efficiency by taking the ratio between these two triggers (already present in the current HLT menù for both data and MC): HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_PFDiJet30_v and HLT_Mu8_TrkIsoVVL_Ele23_CaloIdL_TrackIdL_IsoVL_DZ_PFDiJet30_PFBTagParticleNet_2BTagSum0p65_v

b) For AK8 PNET H(bb)-tagging, given that an enriched control sample of boosted H(bb) jets cannot be isolated in data with a good purity, the measurement is performed in a sample of boosted ttbar semi-leptonic events selected offline requiring one isolated lepton (muon or electron), at least two b-tagged central jets, and one AK8 jet not-in-overlap with the leptons with certain pT, soft-drop-mass, and offline PNET AK8 Hbb score. The efficiency is then measured by the ratio of these triggers: denominator [HLT_IsoMu50_AK8PFJet230_SoftDropMass40_v, HLT_Ele50_CaloIdVT_GsfTrkIdT_AK8PFJet230_SoftDropMass40_v], numerator [HLT_IsoMu50_AK8PFJet230_SoftDropMass40_PFAK8ParticleNetBB0p35_v, HLT_Ele50_CaloIdVT_GsfTrkIdT_AK8PFJet230_SoftDropMass40_PFAK8ParticleNetBB0p35_v]

Backport

Once approved, we will need a backport to the release cycles used in the T0 operation for first Run3 collisions (124X and maybe 123X)

Validation:

we verified that the workflow works by re-emulating the HLT on MC and Data events as described here [0], then re-running the RECO+DQM and Harvesting steps.

[0] https://twiki.cern.ch/twiki/bin/viewauth/CMS/HLTValidationAndDQM

Caveats:

1. In order to perform the measurement of the trigger efficiency in the most robust way, we would need to match the offline jets passing the selection (leading PNET score jets) with those present at the trigger level. This can be performed in the Validation workflow by adding the PNET valueMaps for the score in the HLT event content (as expressed in HLTrigger_EventContent_cff.py). However we are not sure this can be performed when the prompt-reco is run at T0 hence, if this HLT collections won't be present, we would need to remove this matching by changing the value of requireHLTOfflineJetMatching in ParticleNetJetTagMonitoring_cfi.py

2. We would like to select events offline on the basis of the offline PNET in order to minimise misalignments between online and offline. However, currently, PNET AK4 and AK8 are not inferred in the RECO step but only in the MINIAOD/NanoAOD one. Hence, we modified the RecoBTag sequence in order to infere AK8 and AK4 PNET. However, in order to gain speed, we restrict the inference of the AK4 network only to jets with |eta|<2.5 and pT > 25 GeV while, in miniAOD no eta-boundaries are applied and inference runs for all jets above 10 GeV. We hope this can be a good compromise and can cope with the constraints of the standard reconstruction.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-38748/31066

• This PR adds an extra 52KB to repository

[cmsbuild]

A new Pull Request was created by @rgerosa (raffaele gerosa) for master.

It involves the following packages:

• DQMOffline/Trigger (dqm)
• HLTrigger/Configuration (hlt)
• L1Trigger/Configuration (l1)
• RecoBTag/Configuration (reconstruction)
• RecoBTag/ONNXRuntime (reconstruction)

@Martin-Grunewald, @rekovic, @epalencia, @emanueleusai, @ahmad3213, @cmsbuild, @missirol, @jfernan2, @clacaputo, @jpata, @pmandrik, @micsucmed, @cecilecaillol, @rvenditti can you please review it and eventually sign? Thanks.
@AlexDeMoor, @emilbols, @HuguesBrun, @missirol, @silviodonato, @mtosi, @jhgoh, @Martin-Grunewald, @JyothsnaKomaragiri, @sscruz, @Fedespring, @andrzejnovak, @demuller, @trocino, @cericeci, @rociovilar this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy, @rappoccio you are the release manager for this.

cms-bot commands are listed here

[Martin-Grunewald]

This file is automatically generated, so these changes will be lost. Please remove them and instead file a CMSHLT JIRA request and explanation why this addition is needed, I note the wildcard in the keep adding all module instances starting with hltParticleNet*

[rgerosa]

@Martin-Grunewald thanks I didn't know that! I am going to update now the PR taking out that keep and I'll open a JIRA ticket explaining why we need that. In addition, to be more precise, we just need to add hltParticleNetDiscriminators* cause eventually we just look at the final discriminators for the efficiency measurement. Testing this now before opening a JIRA ticket.

[Martin-Grunewald]

This file is automatically generated, so these changes will be lost. Please remove them and instead file a CMSHLT JIRA request and explanation why this addition is needed, I note the wildcard in the keep adding all module instances starting with hltParticleNet*

[Martin-Grunewald]

-1
Remove changes to HLTrigger EventContent

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-38748/31067

• This PR adds an extra 52KB to repository

[cmsbuild]

Pull request #38748 was updated. @epalencia, @emanueleusai, @ahmad3213, @cmsbuild, @rekovic, @jfernan2, @clacaputo, @jpata, @pmandrik, @micsucmed, @cecilecaillol, @rvenditti can you please check and sign again.

[Martin-Grunewald]

Link to CMSHLT Jira ticket for EventContent: https://its.cern.ch/jira/browse/CMSHLT-2402

[emanueleusai]

please test

[emanueleusai]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

[qliphy]

please test
to refresh

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b09ddb/26676/summary.html
COMMIT: f77cb4f
CMSSW: CMSSW_12_5_X_2022-08-06-1100/el8_amd64_gcc10
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/38748/26676/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 51
• DQMHistoTests: Total histograms compared: 3691612
• DQMHistoTests: Total failures: 2
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3691588
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 233.51399999999995 KiB( 50 files compared)
• DQMHistoSizes: changed ( 11634.0,... ): 38.919 KiB HLT/HIG
• Checked 212 log files, 49 edm output root files, 51 DQM output files
• TriggerResults: no differences found

[qliphy]

+1

[Martin-Grunewald]

Hi,

Looks like this PR creates crashes in the HIon tests of HLT - is there some config update for HIon missing? The error is the same for MC:

RelVal --step=RAW2DIGI,L1Reco,RECO,PAT,VALIDATION,DQM --conditions=auto:run3_mc_HIon --filein=file:RelVal_DigiL1RawHLT_HIon_MC.root --custom_conditions= --fileout=RelVal_RECO_HIon_MC.root --number=100 --mc --no_exec --datatier GEN-SIM-RECO,MINIAODSIM,DQMIO --eventcontent=RECOSIM,MINIAODSIM,DQM --customise=HLTrigger/Configuration/CustomConfigs.Base --era=Run3_pp_on_PbPb --customise= --scenario=pp --python_filename=RelVal_RECO_HIon_MC.py --processName=RECO

----- Begin Fatal Exception 10-Aug-2022 19:43:31 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 3 event: 605 stream: 3
   [1] Running path 'dqmoffline_step'
   [2] Prefetching for module ParticleNetJetTagMonitor/'ParticleNetAK8HbbTagMonitoring'
   [3] Prefetching for module BTagProbabilityToDiscriminator/'pfMassDecorrelatedParticleNetDiscriminatorsJetTags'
   [4] Prefetching for module BoostedJetONNXJetTagsProducer/'pfMassDecorrelatedParticleNetJetTags'
   [5] Calling method for module DeepBoostedJetTagInfoProducer/'pfParticleNetTagInfos'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for a container with elements of type: reco::Jet
Looking for module label: ak8PFJetsPuppi
Looking for productInstanceName:

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "SkipEvent = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

and for real data:

 RelVal --step=RAW2DIGI,L1Reco,RECO,DQM --conditions=auto:run3_data_HIon --filein=file:RelVal_HLT_HIon_DATA.root --custom_conditions= --fileout=RelVal_RECO_HIon_DATA.root --number=25 --data --no_exec --datatier RECO,MINIAOD,DQMIO --eventcontent=RECO,MINIAOD,DQM --customise=HLTrigger/Configuration/CustomConfigs.Base --era=Run3_pp_on_PbPb --customise= --scenario=pp --python_filename=RelVal_RECO_HIon_DATA.py --processName=RECO1 

----- Begin Fatal Exception 10-Aug-2022 19:22:04 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 355558 lumi: 224 event: 215655207 stream: 0
   [1] Running path 'dqmoffline_step'
   [2] Prefetching for module ParticleNetJetTagMonitor/'ParticleNetAK8HbbTagMonitoring'
   [3] Prefetching for module BTagProbabilityToDiscriminator/'pfMassDecorrelatedParticleNetDiscriminatorsJetTags'
   [4] Prefetching for module BoostedJetONNXJetTagsProducer/'pfMassDecorrelatedParticleNetJetTags'
   [5] Calling method for module DeepBoostedJetTagInfoProducer/'pfParticleNetTagInfos'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for a container with elements of type: reco::Jet
Looking for module label: ak8PFJetsPuppi
Looking for productInstanceName:

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "SkipEvent = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

[Martin-Grunewald]

I verified this with CMSSW_12_5_X_2022-08-07-2300 out of the box where tests run fine, and then adding this PR on top and tests fail with the above.

[rgerosa]

@Martin-Grunewald

sorry but I did not test the PR for HI since I did not know it was needed. As far as I see from the error, the problem is that the ak8PFJetsPuppi collection is not present in HI data (and it makes sense given that puppi is not tuned for HI). Therefore, I think there are two possible solutions: either we switch the collection from ak8PFJetsPuppi to ak8PFJets for HI or we don't infere PNET in HI data.

Is there a specific modifier for run3 HI? I would say that probably is better just to not run PNET in Run3 HI since it is not meant to be use for HI. The DQM monitoring element should then just return 0 for every event since come collections won't be present.

[Martin-Grunewald]

@rgerosa

The HIon era is Run3_pp_on_PbPb, see:
https://cmssdt.cern.ch/lxr/source/Configuration/Eras/python/Era_Run3_pp_on_PbPb_cff.py
where some exclusions are alreay used (to get it from the Run3 era).
Perhaps one could put the removal there?

[rgerosa]

@Martin-Grunewald thanks!

I suppose that I would need to open a new PR with the fix. I'll try to implement this in the upcoming days and test it. Could you please send me, if not here in a private email, the cmsDriver command you are using to test over the run3 HI scenario in data or MC?

[Martin-Grunewald]

Here is a recipe: in a developer area, check out HLTrigger/Configuration, run a script creating various cfg files and then run a subset of those to reproduce the errors:

cmsrel CMSSW_12_5_X_2022-08-10-2300
cd CMSSW_12_5_X_2022-08-10-2300/src
cmsenv
git cms-addpkg HLTrigger/Configuration
scram b
cd HLTrigger/Configuration/test/
./cmsDriver.csh HIon

# For the test of MC:

cmsRun RelVal_DigiL1RawHLT_HIon_MC.py >& RelVal_DigiL1RawHLT_HIon_MC.log
cmsRun RelVal_RECO_HIon_MC.py >& RelVal_RECO_HIon_MC.log
echo $?

# For the test of real data:

cmsRun RelVal_L1RePack_HIon_DATA.py >& RelVal_L1RePack_HIon_DATA.log
ln -s RelVal_L1RePack_HIon_DATA.root RelVal_Raw_HIon_DATA.root
cmsRun RelVal_HLT_HIon_DATA.py >& RelVal_HLT_HIon_DATA.log
cmsRun RelVal_RECO_HIon_DATA.py >& RelVal_RECO_HIon_DATA.log
echo $?

In each case (MC and data), the last cmsRun step is the one crashing!

[rgerosa]

Thanks @Martin-Grunewald

I'll implement a fix using your recipe and open a new PR. Once merged, we will need also a back port to 12-4-X as this monitoring needs to go into the pp collision monitoring. Unfortunately, I am about to leave for vacation so if not today I'll do it as soon as I am back in a week time scale

[missirol]

#39051 provides a fix for this (I prefer to have the HLT-Validation tests running correctly asap). A better solution can be implemented by Raffaele once he is back, if needed.

[rgerosa]

@missirol just had a quick look from the phone at your PR and I think it is just fine. By taking out the DQM monitoring in HI one prevents the inference of PNET to be run as these products are not stored in AOD. Therefore, it looks good to me! Once your PR will be merged, I'll open a back-port to 12-4-X.

[missirol]

@rgerosa , thanks for having a look. Then I guess you can backport this PR and #39051 together in one go to 12_4_X.