[NGT] Add workflow to produce Phase2 NANOAOD with L1 and HLT objects by elenavernazza · Pull Request #51004 · cms-sw/cmssw

elenavernazza · 2026-05-20T13:32:33Z

PR description:

This PR introduces a new Phase-2 workflow to produce NanoAOD samples containing both L1 and HLT objects.
It combines the configurations of:

L1 NanoAOD based on NANO:@Phase2L1DPGwithGen flavour from Phase2 L1Nano instructions
HLT NanoAOD based on NANO:@NGTScoutingVal flavour from Phase2 HLTNano instructions

Without this PR, the L1+HLT NanoAOD sample could only be produced starting from GEN-SIM-DIGI-RAW-MINIAOD inputs from the TSG Spring24 samples, since the L1 NanoAOD assumes the presence of PAT collections in the input data.
With this PR, it is possible to produce the L1+HLT NanoAOD sample directly from GEN-SIM inputs, without the need to run all the reconstruction chain up to MINIAOD.

A dedicated workflow (suffix .774) has been introduced to monitor possible failures during future developments.

runTheMatrix.py -l 34434.774 -w upgrade --nEvents 10 -i all

The workflow is based on the procModifier nano_l1_hlt, which

disables all GEN sequences that depend on PAT collections to avoid crashes (here)
disables the hpsTauTable since the corresponding input collection was present in the Spring24 samples but is no longer available in more recent L1 menus (here)
removes the dstTriggerAcceptFilter filter for the production of the HLT Tables (here) and removes the event selection in the output (here): these changes allow us to store all events in the output NanoAOD, regardless of whether they pass the L1 and HLT selections. This feature provides a change compared to the NANO:@NGTScoutingVal flavour, where only the events passing the OR of the L1 seeds are stored in the output.

Instructions to run the L1+HLT NanoAOD production:

Starting from GEN-SIM input:

cmsDriver.py step2 -n 10 \
 -s DIGI:pdigi_valid,L1TrackTrigger,L1,L1P2GT,DIGI2RAW,HLT:NGTScouting,VALIDATION:@hltValidation,NANO:@NGTScoutingVal+@Phase2L1DPGwithGen \
 --procModifiers ngtScouting,nano_l1_hlt \
 --conditions auto:phase2_realistic_T35 --geometry ExtendedRun4D121  --era Phase2C22I13M9 \
 --datatier NANOAODSIM --eventcontent NANOAODSIM \
 --customise SLHCUpgradeSimulations/Configuration/aging.customise_aging_1000 \
 --filein file:step1.root --fileout file:step2.root

Starting from GEN-SIM-DIGI-RAW-MINIAOD input:

cmsDriver.py step2 -n 10 \
 -s L1,L1TrackTrigger,L1P2GT,HLT:NGTScouting,NANO:@NGTScoutingVal+@Phase2L1DPG,VALIDATION:@hltValidation \
 --procModifiers ngtScouting,nano_l1_hlt --processName HLTX \
 --datatier GEN-SIM-DIGI-RAW-MINIAOD --eventcontent NANOAODSIM \
 --conditions auto:phase2_realistic_T35 --geometry ExtendedRun4D110 --era Phase2C17I13M9 \
 --customise SLHCUpgradeSimulations/Configuration/aging.customise_aging_1000,L1Trigger/Configuration/customisePhase2TTOn110.customisePhase2TTOn110 \
 --filein /store/mc/Phase2Spring24DIGIRECOMiniAOD/TT_TuneCP5_14TeV-powheg-pythia8/GEN-SIM-DIGI-RAW-MINIAOD/PU200_AllTP_140X_mcRun4_realistic_v4-v1/2560000/11d1f6f0-5f03-421e-90c7-b5815197fc85.root --fileout file:step2.root --python_filename rerun_L1_HLT_NANO_cfg.py \
 --mc --inputCommands='keep *, drop *_hlt*_*_HLT, drop triggerTriggerFilterObjectWithRefs_l1t*_*_HLT, drop *_simEcalUnsuppressedDigis_*_*, drop l1tPFJets_*_*_*, drop l1tTrackerMuons_l1tTkMuonsGmt*_*_HLT'

PR validation:

This PR has been validated by running the new workflow 34434.774:

runTheMatrix.py -l 34434.774 -w upgrade --ibeos --nEvents 10 -i all

The workflow produces correctly the output NanoAOD sample, containing the L1 and HLT Tables.
By adding the option --customise_commands='process.options.wantSummary = True', it is possible to verify that the Nano Tables are filled independently of the L1/HLT filters: in this example, 8 events out of 10 pass theDST_PFScouting filter (corresponding to the OR of all L1 seeds), but all 10 events are saved in the output.

TrigReport ---------- Path   Summary ------------
TrigReport  Trig Bit#   Executed     Passed     Failed      Error Name
TrigReport     1    1         10         10          0          0 L1TrackTrigger_step
TrigReport     1    3         10         10          0          0 Phase2L1GTProducer
TrigReport     1    4         10         10          0          0 Phase2L1GTAlgoBlockProducer
TrigReport     1    5         10          1          9          0 pDoubleEGEle37_24
...
TrigReport     1   45         10          0         10          0 pTripleTkMuon_5_3p5_2p5_OS_Mass5to17
TrigReport     1   47         10          8          2          0 DST_PFScouting
TrigReport     1   48         10          0         10          0 HLTriggerFinalPath
TrigReport     1   49         10         10          0          0 prevalidation_step
TrigReport     1   50         10         10          0          0 nanoAOD_step0

The other two recipes for the L1+HLT NanoAOD production based on GEN-SIM and GEN-SIM-DIGI-RAW-MINIAOD have also been successfully validated.

cmsbuild · 2026-05-20T13:33:00Z

cms-bot internal usage

mmusich · 2026-05-20T13:49:54Z

type ngt

mmusich · 2026-05-20T13:52:12Z

A dedicated workflow (suffix .774) has been introduced to monitor possible failures during future developments.

@elenavernazza can you add it to the ph2_hlt matrix so that it gets automatically tested when we use that matrix in PR tests ? it would need to be added here:

cmssw/Configuration/PyReleaseValidation/python/relval_Run4.py

Line 60 in be7a514

# Phase-2 HLT tests

and here:

cmssw/Configuration/PyReleaseValidation/scripts/runTheMatrix.py

Line 161 in be7a514

'ph2_hlt' : [prefixDet+34.75, # HLT phase-2 timing menu

Finally can you please update the documentation at:

https://github.com/cms-sw/cmssw/blob/master/Configuration/PyReleaseValidation/README.md

for the new .774 offset?

bfonta · 2026-05-20T13:52:32Z

+    '--datatier':'GEN-SIM-DIGI-RAW,NANOAODSIM',
+    '--procModifiers': 'ngtScouting,nano_l1_hlt',
+    '--eventcontent':'FEVTDEBUGHLT,NANOAODSIM'
+}


Thinking about manual configuration, will this step run if the user forgets to specify @Phase2L1DPGwithGen, despite having nano_l1_hlt active?

why would a user do that? Following wrong recipes results typically in wrong results, no?

Sure; I was just suggesting that adding some protection might help.

The NANO:@NGTScoutingVal+@Phase2L1DPGwithGen step should be the only ingredient required to produce the L1+HLT NanoAOD sample.
The nano_l1_hlt procModifier is introduced as a protection against missing PAT collections that are required in some of the L1Nano sequences. The alternative would have been to reorganize the L1Nano configuration itself; however, this would have modified the standard L1Nano format used by existing L1 workflows and users, which I wanted to avoid.

@elenavernazza It is clear what nano_l1_hlt achieves. My comment was rather addressing future users that might use this new process modifier without fully knowing the required inputs. I don't know what happens if the user specifies only NANO:@NGTScoutingVal, but the potential crash might be cryptic. I was suggesting you could add some checks / protections that throw a meaningful error message if the user forgets to add @Phase2L1DPGwithGen.

No, why should it?

Please see my comment.

Please see #51004 (comment).

by this token, should you test all possible workflow combinations in the matrix ?

I never suggested this.
I tested NANO:@NGTScoutingVal with nano_l1_hlt, since the bot does not test it, and since I thought that it could lead to a crash, due to the absence of @Phase2L1DPG.
There is no crash, so all is well.

since the bot does not test it,

It is straightforward to see from the code changes that this should not lead to a runtime crash.

The only modifications affecting NANO:@NGTScoutingVal are in HLTrigger/NGTScouting/python/HLTNanoProducer_cff.py and are limited to:

removing the dstTriggerAcceptFilter filter for the production of the HLT tables

removing the event selection in the output

As already stated in the PR description, these changes simply allow all events to be written to the output NanoAOD, independently of the L1/HLT decision. No producer dependencies or runtime execution paths are modified in a way that could plausibly introduce a crash.

Because of that, I am a bit puzzled that this specific combination of steps/process modifiers was singled out in the review, while there are already many other untested combinations present in CMSSW that would be equally (or more) likely to expose runtime issues.

bfonta · 2026-05-20T13:53:58Z

    process.l1tPh2NanoTask.add(p2L1TablesTask)
+
+    # This modifier excludes the hpsTauTable, which is based on the l1tHPSPFTauProducerPuppi collection, no longer available in the L1 menu.
+    # This allows to run the NANOAOD production from L1 and HLT steps, whitout assuming old inputs from the Spring24 datasets.


Just a nitpick: what do you mean by "assuming" here? Maybe "considering" or "assuming the existence"? Same below.

In case you change something else (otherwise ignore this comment): there is also a typo in "whitout".

cmsbuild · 2026-05-20T13:56:15Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51004/49420

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Rpc digi dev v9 #47447, update phase-2 HLT timing script and use alpaka modifier in NGT workflows #49821, [WIP] Prototype TruthInfo graph for truth history navigation and association studies #50913
- File HLTrigger/NGTScouting/python/HLTNanoProducer_cff.py modified in PR(s): Phase2 Single_Tau_Trigger Path Added #49637, Add TrackingParticle <-> reco-track association tables to NanoAOD (Phase-2 HLT) #50976

cmsbuild · 2026-05-20T13:56:37Z

A new Pull Request was created by @elenavernazza for master.

It involves the following packages:

Configuration/ProcessModifiers (operations)
Configuration/PyReleaseValidation (pdmv)
DPGAnalysis/Phase2L1TNanoAOD (l1, xpog)
HLTrigger/NGTScouting (hlt)
PhysicsTools/NanoAOD (xpog)

@AdrianoDee, @BenjaminRS, @DickyChant, @Martin-Grunewald, @antoniovagnerini, @battibass, @cmsbuild, @davidlange6, @fabiocos, @ftenchini, @ftorrresd, @kfjack, @mandrenguyen, @miquork, @mmusich, @quinnanm, @sroychow can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @fabiocos, @gpetruc, @makortel, @missirol, @mmusich, @slomeo this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2026-05-20T14:10:01Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-51004/49422

There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/relval_Run4.py modified in PR(s): [20_0_X] Disable RelVal workflows that read existing non-RAW files #50954
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Rpc digi dev v9 #47447, update phase-2 HLT timing script and use alpaka modifier in NGT workflows #49821, [WIP] Prototype TruthInfo graph for truth history navigation and association studies #50913
- File Configuration/PyReleaseValidation/scripts/runTheMatrix.py modified in PR(s): [Do not merge] For cms-bot PR testing #48637, [20_0_X] Disable RelVal workflows that read existing non-RAW files #50954
- File HLTrigger/NGTScouting/python/HLTNanoProducer_cff.py modified in PR(s): Phase2 Single_Tau_Trigger Path Added #49637, Add TrackingParticle <-> reco-track association tables to NanoAOD (Phase-2 HLT) #50976

cmsbuild · 2026-05-20T14:10:33Z

Pull request #51004 was updated. @AdrianoDee, @BenjaminRS, @DickyChant, @Martin-Grunewald, @antoniovagnerini, @battibass, @cmsbuild, @davidlange6, @fabiocos, @ftenchini, @ftorrresd, @kfjack, @mandrenguyen, @miquork, @mmusich, @quinnanm, @sroychow can you please check and sign again.

mmusich · 2026-05-20T14:12:27Z

test parameters:

workflows = ph2_hlt

mmusich · 2026-05-20T14:13:35Z

@cmsbuild, please test

mmusich · 2026-05-20T14:55:18Z

the L1+HLT NanoAOD sample could only be produced starting from GEN-SIM-DIGI-RAW-MINIAOD inputs from the TSG Spring24 samples, since the L1 NanoAOD assumes the presence of PAT collections in the input data.

Taking the liberty to tag @artlbv, just to make sure our understanding in this respect is correct.

cmsbuild · 2026-05-20T16:43:01Z

+1

Size: This PR adds an extra 72KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b1e200/53387/summary.html
COMMIT: 2cbc11f
CMSSW: CMSSW_17_0_X_2026-05-20-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/51004/53387/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 66
DQMHistoTests: Total histograms compared: 4596347
DQMHistoTests: Total failures: 65
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4596262
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 65 files compared)
Checked 274 log files, 233 edm output root files, 66 DQM output files
TriggerResults: no differences found

bfonta · 2026-05-20T16:52:55Z

it is possible to verify that the Nano Tables are filled independently of the L1/HLT filters

Is there a way to estimate of how much this PR will affect storage needs?

mmusich · 2026-05-20T17:59:33Z

Is there a way to estimate of how much this PR will affect storage needs?

somewhat of an academical question as this workflow is not used in any production?

mmusich · 2026-05-20T19:26:21Z

+hlt

elenavernazza added 3 commits May 20, 2026 12:23

Add L1 + HLT NanoAOD workflow

0d5b238

Apply procModifier to enable L1 + HLT NanoAOD

b6de68e

Protect track associators from missing collection

74ef3bd

cmsbuild added this to the CMSSW_17_0_X milestone May 20, 2026

cmsbuild added l1-pending hlt-pending operations-pending pending-signatures tests-pending orp-pending pdmv-pending code-checks-pending xpog-pending labels May 20, 2026

elenavernazza changed the title ~~[NGT] Add workflow to produces Phase2 NANOAOD with L1 and HLT objects~~ [NGT] Add workflow to produce Phase2 NANOAOD with L1 and HLT objects May 20, 2026

cmsbuild added the ngt label May 20, 2026

bfonta reviewed May 20, 2026

View reviewed changes

cmsbuild added code-checks-approved and removed code-checks-pending labels May 20, 2026

elenavernazza added 2 commits May 20, 2026 16:02

Add .774 workflow to ph2_hlt test matrix

31b674f

Clarify comment

2cbc11f

cmsbuild added code-checks-pending and removed code-checks-approved labels May 20, 2026

cmsbuild removed the code-checks-pending label May 20, 2026

cmsbuild added the code-checks-approved label May 20, 2026

cmsbuild added tests-started and removed tests-pending labels May 20, 2026

cmsbuild added operations-approved tests-approved and removed operations-pending tests-started labels May 20, 2026

cmsbuild added hlt-approved and removed hlt-pending labels May 20, 2026

This was referenced May 21, 2026

Displaced Particle Gun #51006

Open

add phase-2 HLT nano workflows to the nano matrix #51013

Open

add hybrid PU workflow to PR tests #51022

Open

Conversation

elenavernazza commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR description:

Instructions to run the L1+HLT NanoAOD production:

PR validation:

Uh oh!

cmsbuild commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mmusich commented May 20, 2026

Uh oh!

mmusich commented May 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cmsbuild commented May 20, 2026

Uh oh!

cmsbuild commented May 20, 2026

Uh oh!

cmsbuild commented May 20, 2026

Uh oh!

cmsbuild commented May 20, 2026

Uh oh!

mmusich commented May 20, 2026

Uh oh!

mmusich commented May 20, 2026

Uh oh!

mmusich commented May 20, 2026

Uh oh!

cmsbuild commented May 20, 2026

Comparison Summary

Uh oh!

bfonta commented May 20, 2026

Uh oh!

mmusich commented May 20, 2026

Uh oh!

mmusich commented May 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

elenavernazza commented May 20, 2026 •

edited

Loading

cmsbuild commented May 20, 2026 •

edited

Loading