Issue in CMSSW_12_5_2_patch1 MC with L1TDigi2Raw/gmtStage2Digi

[aloeliger]

What

Taken from cms-sw#41317

One of our MC samples is failing the DIGI step on McM (McM link) with the fatal exception:

An exception of category 'StdException' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'HLTAnalyzerEndpath'
   [2] Prefetching for module L1TRawToDigi/'hltGtStage2Digis'
   [3] Prefetching for module RawDataCollectorByLabel/'rawDataCollector'
   [4] Calling method for module L1TDigiToRaw/'gmtStage2Raw'
Exception Message:
A std::exception was thrown.
array::at: __n (which is 6) >= _Nm (which is 6)

Recipe to recreate:

I was able to reproduce the same error locally by running the cmsDriver commands in CMSSW_12_5_2_patch1:

# producing the GS input, which works
cmsDriver.py Configuration/GenProduction/python/TSG-Phase2Fall22GS-00192-fragment.py --python_filename TSG-Phase2Fall22GS-00192_1_cfg.py --eventcontent RAWSIM --customise Configuration/DataProcessing/Utils.addMonitoring --datatier GEN-SIM --fileout file:TSG-Phase2Fall22GS-00192.root --conditions 125X_mcRun4_realistic_v2 --beamspot HLLHC14TeV --customise_commands process.source.numberEventsInLuminosityBlock="cms.untracked.uint32(100)" --step GEN,SIM --geometry Extended2026D88 --era Phase2C17I13M9 --no_exec --mc -n 294

# DR+MiniAOD step, which fails
cmsDriver.py  --python_filename TSG-Phase2Fall22DRMiniAOD-00056_1_cfg.py --eventcontent FEVTDEBUGHLT --pileup 'AVE_200_BX_25ns' --customise SLHCUpgradeSimulations/Configuration/aging.customise_aging_1000,Configuration/DataProcessing/Utils.addMonitoring --datatier GEN-SIM-DIGI-RAW --fileout file:TSG-Phase2Fall22DRMiniAOD-00056_0.root --pileup_input "dbs:/MinBias_TuneCP5_14TeV-pythia8/Phase2Fall22GS-HCalDetIDFix_125X_mcRun4_realistic_v2-v1/GEN-SIM" --conditions 125X_mcRun4_realistic_v2 --step DIGI,L1TrackTrigger,L1,DIGI2RAW,HLT:@fake2 --geometry Extended2026D88 --nStreams 3 --filein file:TSG-Phase2Fall22GS-00192.root --era Phase2C17I13M9 --no_exec --mc -n 8

[eyigitba]

This is GMT packer. Crash could be related to the HMT data format update which caused the unpackers to crash as well.

[aloeliger]

This is GMT packer. Crash could be related to the HMT data format update which caused the unpackers to crash as well.

Seems unlikely. This is on CMSSW_12_5_2_patch1.

[eyigitba]

oof. didn't see that. nevermind then

[dinyar]

I suspect we crash here: https://github.com/cms-sw/cmssw/blob/CMSSW_12_5_X/EventFilter/L1TRawToDigi/plugins/implementations_stage2/RegionalMuonGMTPacker.cc#L106-L115

The packer assumes that we get three muons (per link) and I guess that in Phase-2 this changed..

Cheers,
Dinyar

[DickyChant]

Hi all,

We seem to have seen similar issues for Run3 SUEP requests as well.

Reported from KIT:

----- Begin Fatal Exception 11-Jun-2024 22:12:45 CEST-----------------------
An exception of category 'StdException' occurred while
   [0] Processing  Event run: 1 lumi: 2032 event: 1425948 stream: 0
   [1] Running path 'HLTAnalyzerEndpath'
   [2] Prefetching a for module L1TRawToDigi/'hltGtStage2Digis'
   [3] Prefetching for module RawDataCollectorByLabel/'rawDataCollector'
   [4] Calling method for module L1TDigiToRaw/'gmtStage2Raw'
Exception Message:
A std::exception was thrown.
array::at: __n (which is 6) >= _Nm (which is 6)
----- End Fatal Exception -------------------------------------------------

The wf and campaign are 'workflow': 'cmsunified_task_EXO-Run3Summer22EEGS-00721__v1_T_240508_172730_6267', 'campaignName': 'Run3Summer22EEGS,Run3Summer22EEDRPremix,Run3Summer22EEMiniAODv4,Run3Summer22EENanoAODv12'

Would appreciate any follow up on this!