Add GEM unpacker in L1Repack FullMC and temporary load of GEM geometry to hltGetConfiguration #34788

srimanob · 2021-08-05T10:39:24Z

PR description:

It was #34785
This PR is to fix two issues when L1REPACK is used for Run3. The error messages w/o this PR is mentioned at the end of the PR description.
(1) no GEM unpacker defined in SimL1EmulatorRepack. It is solved in this PR.
(2) when hltGetConfiguration is used, GEM geometry is not loaded properly. This PR introduced a temporary fix.

Next step:
Following the comment on #34785 (comment)

PR validation:

To test (1), the following cmsDriver run fine.
cmsDriver.py --process reHLT -s L1REPACK:FullMC,HLT:@relval2021 --conditions auto:phase1_2021_realistic --datatier GEN-SIM-RAW -n -1 --eventcontent RAWSIM --geometry DB:Extended --era Run3 --filein file:step2.root --fileout file:step3_reHLT.root --python reHLT_ProdLike_2021.py --no_exec

To test (2), the following config can run fine.
hltGetConfiguration /dev/CMSSW_12_1_0/GRun/V1 --process reHLT --globaltag auto:phase1_2021_realistic --mc --unprescale --paths HLTriggerFirstPath,HLT_IsoMu24_v13,HLT_Mu50_v13,HLTriggerFinalPath --output minimal --input file:step2.root --l1-emulator FullMC > tmp.py

Test cover also DD4hep (wf 11634.911), as GEMGeometryESModule will be loaded twice.

if this PR is a backport please specify the original PR and why you need to backport that PR:

Not a backport, and no need of backport

Note

Issue of (1): solved in this PR

----- Begin Fatal Exception 05-Aug-2021 05:33:59 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'HLTAnalyzerEndpath'
   [2] Prefetching for module L1TRawToDigi/'hltGtStage2Digis'
   [3] Prefetching for module RawDataCollectorByLabel/'rawDataCollector'
   [4] Prefetching for module L1TDigiToRaw/'packGmtStage2'
   [5] Prefetching for module L1TMuonOverlapTrackProducer/'simOmtfDigis'
   [6] Prefetching for module CSCTriggerPrimitivesProducer/'simCscTriggerPrimitiveDigis'
   [7] Prefetching for module GEMPadDigiClusterProducer/'simMuonGEMPadDigiClusters'
   [8] Calling method for module GEMPadDigiProducer/'simMuonGEMPadDigis'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: MuonDigiCollection<GEMDetId,GEMDigi>
Looking for module label: simMuonGEMDigis
Looking for productInstanceName:

Issue of (2): temporary solution is provided for now

----- Begin Fatal Exception 05-Aug-2021 05:59:31 CEST-----------------------
An exception of category 'NoProxyException' occurred while
   [0] Processing  stream begin Run run: 1 stream: 1
   [1] Calling method for module GEMPadDigiClusterProducer/'simMuonGEMPadDigiClusters'
Exception Message:
No data of type "GEMGeometry" with label "" in record "MuonGeometryRecord"
 Please add an ESSource or ESProducer to your job which can deliver this data.
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 05-Aug-2021 05:59:31 CEST-----------------------
An exception of category 'NoProxyException' occurred while
   [0] Processing  stream begin Run run: 1 stream: 3
   [1] Calling method for module GEMPadDigiProducer/'simMuonGEMPadDigis'
Exception Message:
No data of type "GEMGeometry" with label "" in record "MuonGeometryRecord"
 Please add an ESSource or ESProducer to your job which can deliver this data.
----- End Fatal Exception -------------------------------------------------

Martin-Grunewald · 2021-08-05T10:46:40Z

Still same conflict - need to start from most recent 12_1 IB....

srimanob · 2021-08-05T10:51:44Z

Still same conflict - need to start from most recent 12_1 IB....

Should be OK now. I started from CMSSW_12_1_X_2021-08-04-1100, maybe the last merge was after that.

cmsbuild · 2021-08-05T10:56:24Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34788/24487

This PR adds an extra 20KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File HLTrigger/Configuration/python/customizeHLTforCMSSW.py modified in PR(s): Update DT DQM, calibration scripts, and local reco code to allow using "new" DB format #34612

cmsbuild · 2021-08-05T10:56:45Z

A new Pull Request was created by @srimanob (Phat Srimanobhas) for master.

It involves the following packages:

Configuration/StandardSequences (operations)
HLTrigger/Configuration (hlt)

@perrotta, @cmsbuild, @silviodonato, @Martin-Grunewald, @qliphy, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks.
@fabiocos, @makortel, @felicepantaleo, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @Martin-Grunewald, @lecriste, @mtosi, @ebrondol, @mmusich, @dgulhan, @slomeo this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy, @perrotta you are the release manager for this.

cms-bot commands are listed here

srimanob · 2021-08-05T11:02:45Z

Please test

cmsbuild · 2021-08-05T15:21:12Z

-1

Failed Tests: RelVals RelVals-INPUT AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eadd7/17561/summary.html
COMMIT: d7c1197
CMSSW: CMSSW_12_1_X_2021-08-04-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34788/17561/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eadd7/17561/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eadd7/17561/git-merge-result

RelVals

4.534.53_RunPhoton2012B+RunPhoton2012B+HLTD+RECODR1reHLT+HARVESTDR1reHLT/step2_RunPhoton2012B+RunPhoton2012B+HLTD+RECODR1reHLT+HARVESTDR1reHLT.log
136.731136.731_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2/step2_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2.log
136.793136.793_RunDoubleEG2017C+RunDoubleEG2017C+HLTDR2_2017+RECODR2_2017reHLT_skimDoubleEG_Prompt+HARVEST2017/step2_RunDoubleEG2017C+RunDoubleEG2017C+HLTDR2_2017+RECODR2_2017reHLT_skimDoubleEG_Prompt+HARVEST2017.log

Expand to see more relval errors ...

RelVals-INPUT

4.174.17_RunMinBias2011A+RunMinBias2011A+HLTD+RECODR1reHLT+HARVESTDR1reHLT+SKIMDreHLT/step2_RunMinBias2011A+RunMinBias2011A+HLTD+RECODR1reHLT+HARVESTDR1reHLT+SKIMDreHLT.log
4.264.26_ZMuSkim2011A+ZMuSkim2011A+HLTDSKIM+RECODR1reHLT+HARVESTDR1reHLT/step2_ZMuSkim2011A+ZMuSkim2011A+HLTDSKIM+RECODR1reHLT+HARVESTDR1reHLT.log
4.274.27_ZElSkim2011A+ZElSkim2011A+HLTDSKIM+RECODR1reHLT+HARVESTDR1reHLT/step2_ZElSkim2011A+ZElSkim2011A+HLTDSKIM+RECODR1reHLT+HARVESTDR1reHLT.log

Expand to see more relval errors ...

AddOn Tests

hlt_mc_Fake

----- Begin Fatal Exception 05-Aug-2021 16:55:23 CEST-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named /data/cmsbld/jenkins/workspace/ib-run-pr-addon/CMSSW_12_1_X_2021-08-04-2300/src/HLTrigger/Configuration/test/OnLine_HLT_Fake.py
Exception Message:
 unknown python problem occurred.
AttributeError: 'NoneType' object has no attribute 'fillProcessDesc'

At:
  <string>(2): <module>

----- End Fatal Exception -------------------------------------------------

hlt_mc_FakecmsDriver.py RelVal -s HLT:Fake,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run1_mc_Fake --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --processName=HLTRECO --filein file:RelVal_Raw_Fake_MC.root --fileout file:RelVal_Raw_Fake_MC_HLT_RECO.root : FAILED - time: date Thu Aug 5 16:55:53 2021-date Thu Aug 5 16:52:56 2021 s - exit: 256
hlt_mc_Fake1

----- Begin Fatal Exception 05-Aug-2021 16:55:25 CEST-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named /data/cmsbld/jenkins/workspace/ib-run-pr-addon/CMSSW_12_1_X_2021-08-04-2300/src/HLTrigger/Configuration/test/OnLine_HLT_Fake1.py
Exception Message:
 unknown python problem occurred.
AttributeError: 'NoneType' object has no attribute 'fillProcessDesc'

At:
  <string>(2): <module>

----- End Fatal Exception -------------------------------------------------

Expand to see more addon errors ...

cmsbuild · 2021-08-05T17:27:49Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34788/24518

This PR adds an extra 16KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File HLTrigger/Configuration/python/customizeHLTforCMSSW.py modified in PR(s): Update DT DQM, calibration scripts, and local reco code to allow using "new" DB format #34612

cmsbuild · 2021-08-05T17:28:10Z

Pull request #34788 was updated. @perrotta, @cmsbuild, @silviodonato, @Martin-Grunewald, @qliphy, @fabiocos, @davidlange6 can you please check and sign again.

srimanob · 2021-08-05T17:32:00Z

@cmsbuild Please test

cmsbuild · 2021-09-16T01:15:39Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34788/25313

This PR adds an extra 20KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File HLTrigger/Configuration/python/customizeHLTforCMSSW.py modified in PR(s): Move MSLayerKeeper* objects from thread-local into EventSetup #35269, Add support for the gpu modifier at HLT #34978

cmsbuild · 2021-09-16T01:16:03Z

Pull request #34788 was updated. @perrotta, @cmsbuild, @Martin-Grunewald, @qliphy, @fabiocos, @davidlange6 can you please check and sign again.

srimanob · 2021-09-16T01:24:33Z

@cmsbuild please test

cmsbuild · 2021-09-16T07:34:05Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-1eadd7/18649/summary.html
COMMIT: 6c2bedf
CMSSW: CMSSW_12_1_X_2021-09-14-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34788/18649/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 39
DQMHistoTests: Total histograms compared: 3000833
DQMHistoTests: Total failures: 12
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3000799
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
Checked 165 log files, 37 edm output root files, 39 DQM output files
TriggerResults: no differences found

Martin-Grunewald · 2021-09-20T07:07:12Z

Is there a dd4hep test where it fails in case the customisation is run?
IOW, I want to see why we need ~dd4hep rather than just process = customiseFor34788(process)

srimanob · 2021-09-20T07:15:36Z

Is there a dd4hep test where it fails in case the customisation is run?
IOW, I want to see why we need ~dd4hep rather than just process = customiseFor34788(process)

Hi Martin, yes we have.
#34788 (comment)

Martin-Grunewald · 2021-09-20T07:24:21Z

Yes, but that was before we used: process.load("Geometry.GEMGeometryBuilder.gemGeometryDB_cfi"), no?

srimanob · 2021-09-20T07:30:13Z

Yes, but that was before we used: process.load("Geometry.GEMGeometryBuilder.gemGeometryDB_cfi"), no?

Ah, my local test also shows this when I switched to process.load.

Martin-Grunewald · 2021-09-20T07:55:20Z

OK, I reproduced the problem. However, as you know, we are later going to put this (ES module) directly into the menu in ConfDb and the associated menu dumps, but this does not know about dd4hep vs non-dd4hep. So I am afraid I can not sign off on this PR; the problem must be solved at dd4hep/non-dd4hep level. I note no other ES module needs this special case.

Martin-Grunewald · 2021-09-20T07:58:29Z

-1
Calling the customisation routine must not depend on dd4hep vs non-dd4hep use case.

srimanob · 2021-09-20T08:10:20Z

No problem. This PR is on temporary to make GEM unpacking. If it is not OK on HLT side, we can wait. The original reason of this PR is to make sure we can do fullMC emulation. I hope when we have it integrated with CSC, we should not see any issues. I still have no idea why it is only GEM module which is called twice.

cmsbuild added this to the CMSSW_12_1_X milestone Aug 5, 2021

cmsbuild added code-checks-pending hlt-pending operations-pending orp-pending pending-signatures tests-pending labels Aug 5, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Aug 5, 2021

cmsbuild added tests-started and removed tests-pending labels Aug 5, 2021

srimanob mentioned this pull request Aug 5, 2021

Add GEM unpacker in L1Repack FullMC and temporary load of GEM geometry to hltGetConfiguration #34785

Closed

cmsbuild added tests-rejected and removed tests-started labels Aug 5, 2021

srimanob force-pushed the 121_AddGEMUnpackerL1REPACKandUpdateHLTGetConfig branch from d7c1197 to c6c934e Compare August 5, 2021 17:20

cmsbuild added code-checks-pending tests-pending and removed tests-rejected code-checks-approved labels Aug 5, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Aug 5, 2021

cmsbuild added tests-started and removed tests-pending labels Sep 15, 2021

add apply(process)

6c2bedf

cmsbuild added code-checks-pending tests-pending and removed tests-started code-checks-approved labels Sep 16, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Sep 16, 2021

cmsbuild added tests-started and removed tests-pending labels Sep 16, 2021

cmsbuild added tests-approved and removed tests-started labels Sep 16, 2021

This was referenced Sep 16, 2021

[EGM] Eta-Extended-Electrons (EEE) #35309

Merged

Update GPU workflows #35331

Merged

cleanup HLTHcalCalibTypeFilter #35315

Merged

cmsbuild added hlt-rejected and removed hlt-pending labels Sep 20, 2021

srimanob closed this Sep 20, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GEM unpacker in L1Repack FullMC and temporary load of GEM geometry to hltGetConfiguration #34788

Add GEM unpacker in L1Repack FullMC and temporary load of GEM geometry to hltGetConfiguration #34788

srimanob commented Aug 5, 2021 •

edited

Martin-Grunewald commented Aug 5, 2021

srimanob commented Aug 5, 2021 •

edited

cmsbuild commented Aug 5, 2021

cmsbuild commented Aug 5, 2021

srimanob commented Aug 5, 2021

cmsbuild commented Aug 5, 2021

cmsbuild commented Aug 5, 2021

cmsbuild commented Aug 5, 2021

srimanob commented Aug 5, 2021

cmsbuild commented Sep 16, 2021

cmsbuild commented Sep 16, 2021

srimanob commented Sep 16, 2021

cmsbuild commented Sep 16, 2021

Martin-Grunewald commented Sep 20, 2021

srimanob commented Sep 20, 2021

Martin-Grunewald commented Sep 20, 2021

srimanob commented Sep 20, 2021

Martin-Grunewald commented Sep 20, 2021

Martin-Grunewald commented Sep 20, 2021

srimanob commented Sep 20, 2021

Add GEM unpacker in L1Repack FullMC and temporary load of GEM geometry to hltGetConfiguration #34788

Add GEM unpacker in L1Repack FullMC and temporary load of GEM geometry to hltGetConfiguration #34788

Conversation

srimanob commented Aug 5, 2021 • edited

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

Note

Martin-Grunewald commented Aug 5, 2021

srimanob commented Aug 5, 2021 • edited

cmsbuild commented Aug 5, 2021

cmsbuild commented Aug 5, 2021

srimanob commented Aug 5, 2021

cmsbuild commented Aug 5, 2021

RelVals

RelVals-INPUT

AddOn Tests

cmsbuild commented Aug 5, 2021

cmsbuild commented Aug 5, 2021

srimanob commented Aug 5, 2021

cmsbuild commented Sep 16, 2021

cmsbuild commented Sep 16, 2021

srimanob commented Sep 16, 2021

cmsbuild commented Sep 16, 2021

Comparison Summary

Martin-Grunewald commented Sep 20, 2021

srimanob commented Sep 20, 2021

Martin-Grunewald commented Sep 20, 2021

srimanob commented Sep 20, 2021

Martin-Grunewald commented Sep 20, 2021

Martin-Grunewald commented Sep 20, 2021

srimanob commented Sep 20, 2021

srimanob commented Aug 5, 2021 •

edited

srimanob commented Aug 5, 2021 •

edited