Skip invalid channels in the GPU HCAL RecHit producer [12.4.x]

[fwyzard]

PR description:

This PR consistes of two parts:

• the GPU code now marks as invalid (setting chi² = -9999.) the HCAL channels for which it cannot determine the SOI;
• the CPU code that converts from SoA to legacy format now skips all invalid channels.

See h#39693 for a discussion on the underlying issues.

PR validation:

Tested on top of CMSSW_12_4_10 over the events that caused the HLT to crash in runs 357898, 359998, and 360295.
Without these changes the original crash can be reproduced in multiple files:

----- Begin Fatal Exception 14-Oct-2022 20:27:30 CEST-----------------------
An exception of category 'Conditions not found' occurred while
   [0] Processing  Event run: 360295 lumi: 174 event: 264575349 stream: 7
   [1] Running path 'Path'
   [2] Calling method for module CaloTowersCreator/'hltTowerMakerForAll'
Exception Message:
Unavailable Conditions of type HcalChannelQuality for cell (0x0) 
----- End Fatal Exception -------------------------------------------------

With these changes the HLT jobs run to completion on almost all those files (barring one that is affected by a different, ECAL-related crash).

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Backport of #39738 for data taking.

[fwyzard]

type bugfix

[cmsbuild]

A new Pull Request was created by @fwyzard (Andrea Bocci) for CMSSW_12_4_X.

It involves the following packages:

• RecoLocalCalo/HcalRecProducers (reconstruction)

@cmsbuild, @mandrenguyen, @clacaputo can you please review it and eventually sign? Thanks.
@youyingli, @apsallid, @missirol, @bsunanda, @mariadalfonso, @abdoulline this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

• Backported from Skip invalid channels in the GPU HCAL RecHit producer #39738

[fwyzard]

backport #39738

[fwyzard]

please test

[cmsbuild]

Pull request #39740 was updated. @cmsbuild, @mandrenguyen, @clacaputo can you please check and sign again.

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-00138f/28269/summary.html
COMMIT: 76bd7f9
CMSSW: CMSSW_12_4_X_2022-10-14-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39740/28269/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 3675164
• DQMHistoTests: Total failures: 8
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3675134
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
• Checked 208 log files, 45 edm output root files, 50 DQM output files
• TriggerResults: no differences found

[clacaputo]

test parameters:

• enable_tests = gpu
• workflows_gpu = 11634.523

[clacaputo]

please test

[missirol]

@silviodonato @fwyzard

Is the plan to ask for a new (patch) release as soon as this bug-fix is merged?

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-00138f/28332/summary.html
COMMIT: 76bd7f9
CMSSW: CMSSW_12_4_X_2022-10-18-1100/el8_amd64_gcc10
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39740/28332/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 2 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 3675164
• DQMHistoTests: Total failures: 8
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3675134
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
• Checked 208 log files, 45 edm output root files, 50 DQM output files
• TriggerResults: no differences found

GPU Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-00138f/11634.523_TTbar_14TeV+2021_Patatrack_HCALOnlyGPU_Validation+TTbar_14TeV_TuneCP5_GenSim+Digi+RecoNano+HARVESTNano

Summary:

• No significant changes to the logs found
• Reco comparison results: 17 differences found in the comparisons
• DQMHistoTests: Total files compared: 5
• DQMHistoTests: Total histograms compared: 21147
• DQMHistoTests: Total failures: 223
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 20924
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 4 files compared)
• Checked 16 log files, 12 edm output root files, 5 DQM output files
• TriggerResults: found differences in 2 / 4 workflows

[silviodonato]

@silviodonato @fwyzard

Is the plan to ask for a new (patch) release as soon as this bug-fix is merged?

yes, we would like to ask another patch release (if possible) to include this bug-fix. This PR should solve the last remaining "common" crash that we see in the HLT farm (@rappoccio @perrotta)

[clacaputo]

+reconstruction

• backport of Skip invalid channels in the GPU HCAL RecHit producer #39738
• GPUvsCPU diffs for wf 11634.523 are in line with what is stated in Skip invalid channels in the GPU HCAL RecHit producer #39738 (comment)

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_4_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_6_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

[perrotta]

+1