Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[HGCAL trigger] Migration to new code structure - First pass #36337

Merged
merged 14 commits into from Feb 17, 2022

Conversation

jbsauvan
Copy link
Contributor

@jbsauvan jbsauvan commented Dec 2, 2021

PR description:

  • First round for the migration to an updated code structure.
  • This new code structure is developed in order to seamlessly run standalone emulator classes (bit-level emulation of the HGCAL TPG firmware, without any CMSSW dependencies) inside CMSSW.
  • Details of this new code structure can be found in this presentation: https://indico.cern.ch/event/1061177/#29-emulator-technical-developm
  • At the moment, both the old and new class structures are living together, with the old classes still being used as default
  • As this is just the beginning of a large refactoring of our codebase, this will be continued with other PRs in the future

Associated internal PRs and reviews:

This PR depends on external cms-data/L1Trigger-L1THGCal#23

PR validation:

Private tests in L1Trigger/L1THGCalUtilities/test:

cmsRun testHGCalL1T_RelValV11_cfg.py
cmsRun testHGCalL1T_newProcessors_V11_cfg.py
cmsRun testHGCalL1T_stage1Truncation_V11_cfg.py
cmsRun testHGCalL1T_multialgo_V11_cfg.py

Tested D49, D60, D68, D77, D86 workflows

runTheMatrix.py -w upgrade -l 23234.0 --maxSteps=2 
runTheMatrix.py -w upgrade -l 28234.0 --maxSteps=2 
runTheMatrix.py -w upgrade -l 31434.0 --maxSteps=2
runTheMatrix.py -w upgrade -l 35034.0 --maxSteps=2
runTheMatrix.py -w upgrade -l 38634.0 --maxSteps=2

@EmyrClement @portalesHEP FYI

@cmsbuild
Copy link
Contributor

cmsbuild commented Dec 2, 2021

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36337/27085

  • This PR adds an extra 216KB to repository

@cmsbuild
Copy link
Contributor

cmsbuild commented Dec 2, 2021

A new Pull Request was created by @jbsauvan (Jean-Baptiste Sauvan) for master.

It involves the following packages:

  • L1Trigger/L1THGCal (upgrade, l1)
  • L1Trigger/L1THGCalUtilities (upgrade, l1)

@cmsbuild, @AdrianoDee, @srimanob, @cecilecaillol, @rekovic can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @missirol, @amarini, @lgray this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

@rekovic
Copy link
Contributor

rekovic commented Dec 6, 2021

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Dec 6, 2021

-1

Failed Tests: RelVals RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5ba14e/21003/summary.html
COMMIT: 9717eb8
CMSSW: CMSSW_12_2_X_2021-12-05-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36337/21003/install.sh to create a dev area with all the needed externals and cmssw changes.

CMS deprecated warnings: 3 CMS deprecated warnings found, see summary page for details.

RelVals

----- Begin Fatal Exception 06-Dec-2021 10:03:25 CET-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named step2_DIGI_L1TrackTrigger_L1_DIGI2RAW_HLT.py
Exception Message:
 unknown python problem occurred.
RuntimeError: An exception of category 'FileInPathError' occurred.
Exception Message:
edm::FileInPath unable to find file L1Trigger/L1THGCal/data/hgcal_trigger_link_mapping_120links_v1.json anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36337/21003/CMSSW_12_2_X_2021-12-05-2300/poison:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36337/21003/CMSSW_12_2_X_2021-12-05-2300/src:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36337/21003/CMSSW_12_2_X_2021-12-05-2300/external/slc7_amd64_gcc900/data:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/poison:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/src:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/external/slc7_amd64_gcc900/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/runTheMatrix-results/23234.0_TTbar_14TeV+2026D49+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal


At:
  /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/python/FWCore/ParameterSet/Types.py(840): insertInto
  /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/python/FWCore/ParameterSet/Mixins.py(377): insertContentsInto
  /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/python/FWCore/ParameterSet/Types.py(925): insertContentsInto
  /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/python/FWCore/ParameterSet/Types.py(918): insertInto
  /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/python/FWCore/ParameterSet/Config.py(1108): _insertInto
  /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/python/FWCore/ParameterSet/Config.py(1322): fillProcessDesc
  <string>(2): <module>

----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 06-Dec-2021 10:05:41 CET-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named step2_DIGI_L1TrackTrigger_L1_DIGI2RAW_HLT.py
Exception Message:
 unknown python problem occurred.
RuntimeError: An exception of category 'FileInPathError' occurred.
Exception Message:
edm::FileInPath unable to find file L1Trigger/L1THGCal/data/hgcal_trigger_link_mapping_120links_v1.json anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36337/21003/CMSSW_12_2_X_2021-12-05-2300/poison:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36337/21003/CMSSW_12_2_X_2021-12-05-2300/src:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36337/21003/CMSSW_12_2_X_2021-12-05-2300/external/slc7_amd64_gcc900/data:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/poison:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/src:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/external/slc7_amd64_gcc900/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/runTheMatrix-results/35034.0_TTbar_14TeV+2026D77+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal


At:
  /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/python/FWCore/ParameterSet/Types.py(840): insertInto
  /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/python/FWCore/ParameterSet/Mixins.py(377): insertContentsInto
  /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/python/FWCore/ParameterSet/Types.py(925): insertContentsInto
  /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/python/FWCore/ParameterSet/Types.py(918): insertInto
  /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/python/FWCore/ParameterSet/Config.py(1108): _insertInto
  /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/python/FWCore/ParameterSet/Config.py(1322): fillProcessDesc
  <string>(2): <module>

----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 06-Dec-2021 10:07:27 CET-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named step2_DIGI_L1TrackTrigger_L1_DIGI2RAW_HLT.py
Exception Message:
 unknown python problem occurred.
RuntimeError: An exception of category 'FileInPathError' occurred.
Exception Message:
edm::FileInPath unable to find file L1Trigger/L1THGCal/data/hgcal_trigger_link_mapping_120links_v1.json anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36337/21003/CMSSW_12_2_X_2021-12-05-2300/poison:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36337/21003/CMSSW_12_2_X_2021-12-05-2300/src:/cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36337/21003/CMSSW_12_2_X_2021-12-05-2300/external/slc7_amd64_gcc900/data:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/poison:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/src:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/external/slc7_amd64_gcc900/data
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/runTheMatrix-results/28234.0_TTbar_14TeV+2026D60+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal


At:
  /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/python/FWCore/ParameterSet/Types.py(840): insertInto
  /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/python/FWCore/ParameterSet/Mixins.py(377): insertContentsInto
  /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/python/FWCore/ParameterSet/Types.py(925): insertContentsInto
  /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/python/FWCore/ParameterSet/Types.py(918): insertInto
  /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/python/FWCore/ParameterSet/Config.py(1108): _insertInto
  /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_2_X_2021-12-05-2300/python/FWCore/ParameterSet/Config.py(1322): fillProcessDesc
  <string>(2): <module>

----- End Fatal Exception -------------------------------------------------
Expand to see more relval errors ...

RelVals-INPUT

  • 23234.023234.0_TTbar_14TeV+2026D49+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14INPUT+DigiTrigger+RecoGlobal+HARVESTGlobal/step2_TTbar_14TeV+2026D49+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14INPUT+DigiTrigger+RecoGlobal+HARVESTGlobal.log
  • 23234.10323234.103_TTbar_14TeV+2026D49Aging3000+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14INPUT+DigiTrigger+RecoGlobal+HARVESTGlobal/step2_TTbar_14TeV+2026D49Aging3000+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14INPUT+DigiTrigger+RecoGlobal+HARVESTGlobal.log
  • 23234.2123234.21_TTbar_14TeV+2026D49_ProdLike+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14INPUT+DigiTrigger+RecoGlobal+MiniAOD/step2_TTbar_14TeV+2026D49_ProdLike+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14INPUT+DigiTrigger+RecoGlobal+MiniAOD.log
Expand to see more relval errors ...

@jbsauvan
Copy link
Contributor Author

jbsauvan commented Dec 6, 2021

It needs to be tested with cms-data/L1Trigger-L1THGCal#23

@perrotta
Copy link
Contributor

perrotta commented Dec 6, 2021

test parameters:

@perrotta
Copy link
Contributor

perrotta commented Dec 6, 2021

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Dec 6, 2021

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5ba14e/21009/summary.html
COMMIT: 9717eb8
CMSSW: CMSSW_12_2_X_2021-12-05-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36337/21009/install.sh to create a dev area with all the needed externals and cmssw changes.

CMS deprecated warnings: 3 CMS deprecated warnings found, see summary page for details.

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 2 differences found in the comparisons
  • DQMHistoTests: Total files compared: 42
  • DQMHistoTests: Total histograms compared: 3250608
  • DQMHistoTests: Total failures: 5
  • DQMHistoTests: Total nulls: 1
  • DQMHistoTests: Total successes: 3250580
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.004 KiB( 41 files compared)
  • DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
  • Checked 177 log files, 37 edm output root files, 42 DQM output files
  • TriggerResults: no differences found

@jbsauvan
Copy link
Contributor Author

Hello, are there comments on this PR?
Thanks!

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5ba14e/22456/summary.html
COMMIT: 882a50c
CMSSW: CMSSW_12_3_X_2022-02-15-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36337/22456/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5ba14e/22456/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5ba14e/22456/git-merge-result

CMS deprecated warnings: 3 CMS deprecated warnings found, see summary page for details.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-5ba14e/138.4_PromptCollisions+RunMinimumBias2021+ALCARECOPROMPTR3+HARVESTDPROMPTR3
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-5ba14e/138.5_ExpressCollisions+RunMinimumBias2021+TIER0EXPRUN3+ALCARECOEXPR3+HARVESTDEXPR3
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-5ba14e/139.001_RunMinimumBias2021+RunMinimumBias2021+HLTDR3_2021+RECODR3_MinBiasOffline+HARVESTD2021MB

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 4 differences found in the comparisons
  • DQMHistoTests: Total files compared: 49
  • DQMHistoTests: Total histograms compared: 3965143
  • DQMHistoTests: Total failures: 13
  • DQMHistoTests: Total nulls: 1
  • DQMHistoTests: Total successes: 3965107
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.004 KiB( 48 files compared)
  • DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
  • Checked 204 log files, 45 edm output root files, 49 DQM output files
  • TriggerResults: no differences found

@AdrianoDee
Copy link
Contributor

@jbsauvan the differences have disappeared, so they originated from it.

@jbsauvan
Copy link
Contributor Author

Thanks @AdrianoDee
I've added the commit back. We can test if the differences come back, to be 100% sure.
Then we can remove the commit or keep it before merging, as you prefer; for me at the moment it is not important to have it in.
Anyway I will bring the issue to the relevant people to see what is the status of deriving connection maps for the V16 geometry.

@AdrianoDee
Copy link
Contributor

@jbsauvan seems a good plan. Triggering them again.

@AdrianoDee
Copy link
Contributor

Please test

@cecilecaillol
Copy link
Contributor

+l1

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5ba14e/22479/summary.html
COMMIT: fc77fa5
CMSSW: CMSSW_12_3_X_2022-02-16-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36337/22479/install.sh to create a dev area with all the needed externals and cmssw changes.

CMS deprecated warnings: 3 CMS deprecated warnings found, see summary page for details.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-5ba14e/138.4_PromptCollisions+RunMinimumBias2021+ALCARECOPROMPTR3+HARVESTDPROMPTR3
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-5ba14e/138.5_ExpressCollisions+RunMinimumBias2021+TIER0EXPRUN3+ALCARECOEXPR3+HARVESTDEXPR3
  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-5ba14e/139.001_RunMinimumBias2021+RunMinimumBias2021+HLTDR3_2021+RECODR3_MinBiasOffline+HARVESTD2021MB

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 2 differences found in the comparisons
  • DQMHistoTests: Total files compared: 49
  • DQMHistoTests: Total histograms compared: 3965143
  • DQMHistoTests: Total failures: 12
  • DQMHistoTests: Total nulls: 1
  • DQMHistoTests: Total successes: 3965108
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.004 KiB( 48 files compared)
  • DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
  • Checked 204 log files, 45 edm output root files, 49 DQM output files
  • TriggerResults: no differences found

@AdrianoDee
Copy link
Contributor

Ok, now I don't see any difference still. I'm not completely understanding what has happened but this shows that the differences saw came from something unrelated to this PR. It's enough for me.

@AdrianoDee
Copy link
Contributor

+upgrade
Nothing new w.r.t. latest sign-off, investigations have shown that the differences that appeared at a certain point where unrelated to this PR.

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

@perrotta
Copy link
Contributor

perrotta commented Feb 17, 2022

Ok, now I don't see any difference still. I'm not completely understanding what has happened but this shows that the differences saw came from something unrelated to this PR. It's enough for me.

@AdrianoDee while I am perfectly fine with the previous explanation provided by @jbsauvan , I would like to correct you: the results of the last tests do show the same differences (probably only numerical fluctuations) in the L1T DQM for wf 38634.0 that we were discussing above in this thread.

@perrotta
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit 72a3ef1 into cms-sw:master Feb 17, 2022
@AdrianoDee
Copy link
Contributor

@perrotta you are right. I overlooked sorry. We will keep an eye on the next step. (Still some DQM plots that showed differences have gone)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet