Fix `SiPixelRawToClusterGPUKernel` for spurious ROCs [12.4.x] #39713

fwyzard · 2022-10-12T15:55:29Z

PR description:

Make the implementation of the pixel unpacker running on GPU skip spurious ROCs similar to the legacy version of the unpacker.

Fixes an HLT crash that has been happening more or less regularly for the past two months.

See #39045 for more details on the problem, investigation, and tests.

Also, changes the debug function parameter to a template parameter, to enable or disable the printouts at compile time.

PR validation:

Running the HLT over the error stream files does not crash any more.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Backport of #39711 to 12.4.x for data taking.

fwyzard · 2022-10-12T15:55:49Z

type bugfix

cmsbuild · 2022-10-12T15:55:54Z

A new Pull Request was created by @fwyzard (Andrea Bocci) for CMSSW_12_4_X.

It involves the following packages:

RecoLocalTracker/SiPixelClusterizer (reconstruction)

@cmsbuild, @mandrenguyen, @clacaputo can you please review it and eventually sign? Thanks.
@mtosi, @VourMa, @felicepantaleo, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @mroguljic, @missirol, @dkotlins, @ferencek, @gpetruc, @mmusich, @threus, @tvami this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

Backported from Fix SiPixelRawToClusterGPUKernel for spurious ROCs #39711

fwyzard · 2022-10-12T15:55:55Z

backport #39711

fwyzard · 2022-10-12T15:56:15Z

enable gpu

fwyzard · 2022-10-12T15:56:24Z

please test

VinInn · 2022-10-12T16:02:33Z

RecoLocalTracker/SiPixelClusterizer/plugins/SiPixelRawToClusterGPUKernel.cu

+      // check for spurious channels
+      if (roc > MAX_ROC or link > MAX_LINK) {
+        if constexpr (debug) {
+          printf("spurious roc %d found on link %d, detector %d (index %d)\n", roc, link, rawId, gIndex);


rawId is "of course" wrong at this point...
one could use

getRawId(cablingMap, fedId, link, 1).rawId;

under the assumption that all rocs in a link belong to the same det

cmsbuild · 2022-10-12T16:42:11Z

Pull request #39713 was updated. @cmsbuild, @mandrenguyen, @clacaputo can you please check and sign again.

cmsbuild · 2022-10-12T16:47:36Z

Pull request #39713 was updated. @cmsbuild, @mandrenguyen, @clacaputo can you please check and sign again.

fwyzard · 2022-10-12T16:49:23Z

please test

cmsbuild · 2022-10-12T20:44:21Z

-1

Failed Tests: RelVals-GPU
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c7a516/28205/summary.html
COMMIT: 3b29a91
CMSSW: CMSSW_12_4_X_2022-10-12-1100/el8_amd64_gcc10
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39713/28205/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-GPU

11634.52211634.522_TTbar_14TeV+2021_Patatrack_HCALOnlyGPU+TTbar_14TeV_TuneCP5_GenSim+Digi+RecoNano+HARVESTNano/step2_TTbar_14TeV+2021_Patatrack_HCALOnlyGPU+TTbar_14TeV_TuneCP5_GenSim+Digi+RecoNano+HARVESTNano.log
11634.50611634.506_TTbar_14TeV+2021_Patatrack_PixelOnlyTripletsGPU+TTbar_14TeV_TuneCP5_GenSim+Digi+RecoNano+HARVESTNano/step2_TTbar_14TeV+2021_Patatrack_PixelOnlyTripletsGPU+TTbar_14TeV_TuneCP5_GenSim+Digi+RecoNano+HARVESTNano.log
11634.51211634.512_TTbar_14TeV+2021_Patatrack_ECALOnlyGPU+TTbar_14TeV_TuneCP5_GenSim+Digi+RecoNano+HARVESTNano/step2_TTbar_14TeV+2021_Patatrack_ECALOnlyGPU+TTbar_14TeV_TuneCP5_GenSim+Digi+RecoNano+HARVESTNano.log

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3675164
DQMHistoTests: Total failures: 8
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3675134
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

fwyzard · 2022-10-13T10:35:19Z

NVML Error 18: Driver/library version mismatch

means that (some of) the test machines need to reload the NVIDIA drivers.
@smuzaffar is there a way to know on which machine the tests were run ? or ask to reload the drivers on all of them ?

smuzaffar · 2022-10-13T10:46:46Z

@fwyzard , we do not have any dedicated GPU resource so we use HTCondor GPU resources for PR/IB tests. I will ask CERN IT to reload the drivers for these resources.

[a]

Building remotely on [grid121390](https://cmssdt.cern.ch/jenkins/computer/grid121390) 
gridid-121390.0 is-ssse3 disk-189G-164G is-avx512vl eos schedulername-bigbird20.cern.ch slc7_amd64 is-sse2 cpu-medium x86_64 is-avx singularity scripts user-cmsbuild nvidia cpu-tiny is-avx512bw kernel-3.10.0-1160.76.1.el7.x86_64 is-avx2 cpu-small cpu-4 is-avx512dq nvidia-515.43.04 is-avx512_vnni is-avx512f memory-15507 real-memory-15507 GenuineIntel is-sse4_2 auto-delete is-sse is-sse4_1 hostname-b7g47n9785 real-cpu-4 slc7 condor

fwyzard · 2022-10-13T11:05:28Z

OK, thanks.

cmsbuild · 2022-10-13T17:20:58Z

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c7a516/28221/summary.html
COMMIT: 3b29a91
CMSSW: CMSSW_12_4_X_2022-10-13-1100/el8_amd64_gcc10
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39713/28221/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

The relvals timed out after 4 hours.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3675164
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3675134
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 49 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 19876
DQMHistoTests: Total failures: 8
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 19868
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
Checked 12 log files, 9 edm output root files, 4 DQM output files
TriggerResults: found differences in 1 / 3 workflows

missirol · 2022-10-13T20:12:06Z

urgent

TSG asked for a patch release including this PR, for HLT online operations.

@cms-sw/reconstruction-l2 , please prioritise the review of this PR (the 12_6_X PR needs to be merged first, etc.).

missirol · 2022-10-13T20:13:36Z

please test

It's clear that this PR passes the tests. We can anyway try to 'clean up the labels' overnight. I admit I don't know the cause behind these recurring "RelVals-INPUT" errors; I wonder whether it is not solvable, or if there is an issue about it.

cmsbuild · 2022-10-14T00:02:22Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c7a516/28245/summary.html
COMMIT: 3b29a91
CMSSW: CMSSW_12_4_X_2022-10-13-1100/el8_amd64_gcc10
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39713/28245/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3675164
DQMHistoTests: Total failures: 13
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3675128
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 49 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
Reco comparison had 3 failed jobs
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 19876
DQMHistoTests: Total failures: 8
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 19868
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
Checked 12 log files, 9 edm output root files, 4 DQM output files
TriggerResults: found differences in 1 / 3 workflows

rappoccio · 2022-10-14T15:29:02Z

Status described here: #39045 (comment)

mandrenguyen · 2022-10-14T15:45:28Z

+1

cmsbuild · 2022-10-14T15:45:49Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_4_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_6_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2022-10-15T16:36:08Z

+1

fwyzard added 2 commits October 12, 2022 17:40

Skip invalid or corrupted ROCs

82d67dd

Make if (debug) a compile-time check

208983d

cmsbuild added this to the CMSSW_12_4_X milestone Oct 12, 2022

cmsbuild added bug-fix orp-pending pending-signatures reconstruction-pending tests-pending labels Oct 12, 2022

cmsbuild added the backport label Oct 12, 2022

cmsbuild added tests-started and removed tests-pending labels Oct 12, 2022

fwyzard mentioned this pull request Oct 12, 2022

HLT crash caused by SiPixelDigisClustersFromSoA (run 357271) #39045

Closed

VinInn reviewed Oct 12, 2022

View reviewed changes

cmsbuild added tests-pending and removed tests-started labels Oct 12, 2022

Print the correct detector id for spurious ROCs

3b29a91

fwyzard force-pushed the fix_SiPixelRawToClusterGPUKernel_124x branch from 02f4852 to 3b29a91 Compare October 12, 2022 16:47

cmsbuild added tests-started and removed tests-pending labels Oct 12, 2022

cmsbuild added tests-rejected and removed tests-started labels Oct 12, 2022

This was referenced Oct 13, 2022

Fix SiPixelRawToClusterGPUKernel for spurious ROCs #39711

Merged

Fix SiPixelRawToClusterGPUKernel for spurious ROCs [12.5.x] #39712

Merged

cmsbuild added tests-rejected and removed tests-started labels Oct 13, 2022

cmsbuild added the urgent label Oct 13, 2022

cmsbuild added tests-started and removed tests-rejected labels Oct 13, 2022

cmsbuild added tests-approved and removed tests-started labels Oct 14, 2022

cmsbuild added backport-ok and removed backport labels Oct 14, 2022

cmsbuild added fully-signed reconstruction-approved and removed reconstruction-pending pending-signatures labels Oct 14, 2022

cmsbuild added orp-approved and removed orp-pending labels Oct 15, 2022

cmsbuild merged commit a89c6b6 into cms-sw:CMSSW_12_4_X Oct 15, 2022

fwyzard deleted the fix_SiPixelRawToClusterGPUKernel_124x branch October 29, 2022 08:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix `SiPixelRawToClusterGPUKernel` for spurious ROCs [12.4.x] #39713

Fix `SiPixelRawToClusterGPUKernel` for spurious ROCs [12.4.x] #39713

fwyzard commented Oct 12, 2022

fwyzard commented Oct 12, 2022

cmsbuild commented Oct 12, 2022 •

edited

fwyzard commented Oct 12, 2022

fwyzard commented Oct 12, 2022

fwyzard commented Oct 12, 2022

VinInn Oct 12, 2022 •

edited

fwyzard Oct 12, 2022

cmsbuild commented Oct 12, 2022

cmsbuild commented Oct 12, 2022

fwyzard commented Oct 12, 2022

cmsbuild commented Oct 12, 2022

fwyzard commented Oct 13, 2022

smuzaffar commented Oct 13, 2022

fwyzard commented Oct 13, 2022

cmsbuild commented Oct 13, 2022

missirol commented Oct 13, 2022

missirol commented Oct 13, 2022

cmsbuild commented Oct 14, 2022

rappoccio commented Oct 14, 2022

mandrenguyen commented Oct 14, 2022

cmsbuild commented Oct 14, 2022

perrotta commented Oct 15, 2022

Navigation Menu

Fix SiPixelRawToClusterGPUKernel for spurious ROCs [12.4.x] #39713

Fix SiPixelRawToClusterGPUKernel for spurious ROCs [12.4.x] #39713

Conversation

fwyzard commented Oct 12, 2022

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

fwyzard commented Oct 12, 2022

cmsbuild commented Oct 12, 2022 • edited

fwyzard commented Oct 12, 2022

fwyzard commented Oct 12, 2022

fwyzard commented Oct 12, 2022

VinInn Oct 12, 2022 • edited

Choose a reason for hiding this comment

fwyzard Oct 12, 2022

Choose a reason for hiding this comment

cmsbuild commented Oct 12, 2022

cmsbuild commented Oct 12, 2022

fwyzard commented Oct 12, 2022

cmsbuild commented Oct 12, 2022

RelVals-GPU

Comparison Summary

fwyzard commented Oct 13, 2022

smuzaffar commented Oct 13, 2022

fwyzard commented Oct 13, 2022

cmsbuild commented Oct 13, 2022

RelVals-INPUT

Comparison Summary

GPU Comparison Summary

missirol commented Oct 13, 2022

missirol commented Oct 13, 2022

cmsbuild commented Oct 14, 2022

Comparison Summary

GPU Comparison Summary

rappoccio commented Oct 14, 2022

mandrenguyen commented Oct 14, 2022

cmsbuild commented Oct 14, 2022

perrotta commented Oct 15, 2022

Fix `SiPixelRawToClusterGPUKernel` for spurious ROCs [12.4.x] #39713

Fix `SiPixelRawToClusterGPUKernel` for spurious ROCs [12.4.x] #39713

cmsbuild commented Oct 12, 2022 •

edited

VinInn Oct 12, 2022 •

edited