Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HLT customisation for Patatrack pixel tracks and GPU offload #34956

Merged
merged 1 commit into from Aug 20, 2021

Conversation

fwyzard
Copy link
Contributor

@fwyzard fwyzard commented Aug 19, 2021

PR description:

Add a customisation script for the HLT:

  • enable the Patatrack pixel track reconstruction, with quadruplets or triplets, running on CPU or GPU
  • optionally, add the possibility to offload the ECAL unpacking and local reconstruction to run on GPU
  • optionally, add the possibility to offload the HCAL local reconstruction to run on GPU

The new customisation is not used by any workflows.

PR validation:

Validated in CMSSW_12_1_0_pre1 with

cmsDriver.py reHLT \
    --processName reHLT \
    -s HLT:@relval2021 \
    --conditions 120X_mcRun3_2021_realistic_v4 \
    --datatier GEN-SIM-DIGI-RAW \
    -n 10 \
    --eventcontent FEVTDEBUGHLT \
    --geometry DB:Extended \
    --era Run3 \
    --customise=HLTrigger/Configuration/customizeHLTforPatatrack.customizeHLTforPatatrackTriplets \
    --filein /store/relval/CMSSW_12_0_0_pre6/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_120X_mcRun3_2021_realistic_v4_JIRA_129-v1/00000/79c06ed5-929b-4a57-a4f2-1ae90e6b38c5.root

resulting in

19-Aug-2021 17:28:58 CEST  Initiating request to open file file:///gpu_data/store/relval/CMSSW_12_0_0_pre6/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_120X_mcRun3_2021_realistic_v4_JIRA_129-v1/00000/79c06ed5-929b-4a57-a4f2-1ae90e6b38c5.root
19-Aug-2021 17:29:00 CEST  Successfully opened file file:///gpu_data/store/relval/CMSSW_12_0_0_pre6/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_120X_mcRun3_2021_realistic_v4_JIRA_129-v1/00000/79c06ed5-929b-4a57-a4f2-1ae90e6b38c5.root
%MSG-e TkDetLayers:   TSGFromL2Muon:hltL3NoFiltersNoVtxTrajSeedOIHit@streamBeginRun  19-Aug-2021 17:29:42 CEST Run: 1 Stream: 0
 ForwardDiskSectorBuilderFromDet: Trying to build Petal Wedge from Dets at different z positions !! Delta_z = -0.951241
%MSG
Begin processing the 1st record. Run 1, Event 2, LumiSection 1 on stream 0 at 19-Aug-2021 17:29:43.362 CEST
#--------------------------------------------------------------------------
#                         FastJet release 3.4.0
#                 M. Cacciari, G.P. Salam and G. Soyez                  
#     A software package for jet finding and analysis at colliders      
#                           http://fastjet.fr                           
#	                                                                      
# Please cite EPJC72(2012)1896 [arXiv:1111.6097] if you use this package
# for scientific work and optionally PLB641(2006)57 [hep-ph/0512210].   
#                                                                       
# FastJet is provided without warranty under the GNU GPL v2 or higher.  
# It uses T. Chan's closest pair algorithm, S. Fortune's Voronoi code
# and 3rd party plugin jet algorithms. See COPYING file for details.
#--------------------------------------------------------------------------
Begin processing the 2nd record. Run 1, Event 1, LumiSection 1 on stream 0 at 19-Aug-2021 17:29:52.870 CEST
Begin processing the 3rd record. Run 1, Event 3, LumiSection 1 on stream 0 at 19-Aug-2021 17:30:00.396 CEST
Begin processing the 4th record. Run 1, Event 4, LumiSection 1 on stream 0 at 19-Aug-2021 17:30:03.761 CEST
Begin processing the 5th record. Run 1, Event 5, LumiSection 1 on stream 0 at 19-Aug-2021 17:30:07.514 CEST
Begin processing the 6th record. Run 1, Event 7, LumiSection 1 on stream 0 at 19-Aug-2021 17:30:11.653 CEST
Begin processing the 7th record. Run 1, Event 8, LumiSection 1 on stream 0 at 19-Aug-2021 17:30:18.739 CEST
Begin processing the 8th record. Run 1, Event 6, LumiSection 1 on stream 0 at 19-Aug-2021 17:30:20.226 CEST
Begin processing the 9th record. Run 1, Event 9, LumiSection 1 on stream 0 at 19-Aug-2021 17:30:28.354 CEST
Begin processing the 10th record. Run 1, Event 10, LumiSection 1 on stream 0 at 19-Aug-2021 17:30:33.571 CEST
19-Aug-2021 17:30:42 CEST  Closed file file:///gpu_data/store/relval/CMSSW_12_0_0_pre6/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_120X_mcRun3_2021_realistic_v4_JIRA_129-v1/00000/79c06ed5-929b-4a57-a4f2-1ae90e6b38c5.root

=============================================

MessageLogger Summary

 type     category        sev    module        subroutine        count    total
 ---- -------------------- -- ---------------- ----------------  -----    -----
    1 TkDetLayers          -e TSGFromL2Muon:hl                       1        1
    2 fileAction           -s file_close                             1        1
    3 fileAction           -s file_open                              2        2

 type    category    Examples: run/evt        run/evt          run/evt
 ---- -------------------- ---------------- ---------------- ----------------
    1 TkDetLayers          Run: 1 Stream: 0                  
    2 fileAction           PostGlobalEndRun                  
    3 fileAction           pre-events       pre-events       

Severity    # Occurrences   Total Occurrences
--------    -------------   -----------------
Error                   1                   1
System                  3                   3

dropped waiting message count 0

Tested successfully both with and without a GPU.
The physics validation and the GPU vs CPU comparison will follow after the customisation has been merged.

Add a customisation script for the HLT:
  - enable the Patatrack pixel track reconstruction, with quadruplets or triplets, running on CPU or GPU
  - optionally, add the possibility to offload the ECAL unpacking and local reconstruction to run on GPU
  - optionally, add the possibility to offload the HCAL local reconstruction to run on GPU

Co-authored-by: Andrea Massironi <massironi.andrea@gmail.com>
@fwyzard
Copy link
Contributor Author

fwyzard commented Aug 19, 2021

@cmsbuild, please test

To make the bot happy, even if the new customisation is not used anywhere.

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34956/24776

  • This PR adds an extra 20KB to repository

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @fwyzard (Andrea Bocci) for master.

It involves the following packages:

  • HLTrigger/Configuration (hlt)

@Martin-Grunewald can you please review it and eventually sign? Thanks.
@Martin-Grunewald this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

@fwyzard
Copy link
Contributor Author

fwyzard commented Aug 19, 2021

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a99d23/17896/summary.html
COMMIT: 88685fe
CMSSW: CMSSW_12_1_X_2021-08-19-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34956/17896/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 4 differences found in the comparisons
  • DQMHistoTests: Total files compared: 39
  • DQMHistoTests: Total histograms compared: 3000352
  • DQMHistoTests: Total failures: 6
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3000324
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
  • Checked 165 log files, 37 edm output root files, 39 DQM output files
  • TriggerResults: no differences found

@Martin-Grunewald
Copy link
Contributor

+1

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

@perrotta
Copy link
Contributor

+1

  • Taking for good the validation of the customization as reported in the PR description

@cmsbuild cmsbuild merged commit 69d2c87 into cms-sw:master Aug 20, 2021
@missirol
Copy link
Contributor

(Leaving a comment for @fwyzard, even though the PR is closed)

If using an HLT config with only a subset of triggers, it can happen that some of the customisations are silently skipped.

I noticed this when testing a config with (only) the path MC_PFMET, which contains HLTDoFullUnpackingEgammaEcalWithoutPreshowerSequence but not HLTDoFullUnpackingEgammaEcalSequence; in this case, the ECAL customisation function is (unnecessarily) skipped due to this check.

I guess one option is to just improve a bit the safeguards at the beginning of the functions (maybe just the ECAL one). I was wondering if it would be okay to remove the safeguards altogether (and then, if no path uses the customised sequences/tasks, those will be ignored in any case?).

@fwyzard
Copy link
Contributor Author

fwyzard commented Aug 23, 2021

@missirol thanks for the comment.

We cannot run the customisation unconditionally, as they would fail if the configuration is missing the modules it references, e.g.

process.hltEcalDigisLegacy = process.hltEcalDigis.clone()

will fail if hltEcalDigis is not available, etc.

It may be enough to change

-if not 'HLTDoFullUnpackingEgammaEcalSequence' in process.__dict__:
+if not any(seq in process.__dict__ for seq in ['HLTDoFullUnpackingEgammaEcalMFSequence', 'HLTDoFullUnpackingEgammaEcalWithoutPreshowerSequence', 'HLTDoFullUnpackingEgammaEcalSequence'):

Could you give it a try ?

@missirol
Copy link
Contributor

Thanks, @fwyzard . Yep, will try.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants