Clamp the number of pixel tracks to the capacity of the SoA [12.4.x] #38884

fwyzard · 2022-07-28T15:38:11Z

PR description:

The data structure used to store the pixel tracks currently has a fixed size of 32768, which was found to be adequate based on MC studies.
While running over collisions data a small fraction of events has been found to have a larger number of tracks, leading to assertion failures or memory access errors.

This PR clamps the number of pixel tracks to the capacity of the SoA, and adds a warning like

%MSG-w PixelTracks:  PixelTrackSoAFromCUDA:hltPixelTracksFromGPU  30-Jul-2022 09:16:12 CEST Run: 356071 Event: 46424286
Unsorted reconstructed pixel tracks truncated to 32767 candidates
%MSG

PR validation:

Running over an event with > 33k pixel tracks no longer causes assertion failures.

If this PR is a backport please specify the original PR and why you need to backport that PR.

Backport of #38883 for data taking.

fwyzard · 2022-07-28T15:38:21Z

type bugfix

fwyzard · 2022-07-28T15:38:25Z

backport #38883

fwyzard · 2022-07-28T15:38:31Z

enable gpu

cmsbuild · 2022-07-28T15:38:32Z

A new Pull Request was created by @fwyzard (Andrea Bocci) for CMSSW_12_4_X.

It involves the following packages:

RecoPixelVertexing/PixelTrackFitting (reconstruction)
RecoPixelVertexing/PixelTriplets (reconstruction)

@jpata, @cmsbuild, @clacaputo can you please review it and eventually sign? Thanks.
@VourMa, @felicepantaleo, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @mmusich, @mtosi, @dgulhan this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy, @rappoccio you are the release manager for this.

cms-bot commands are listed here

Backported from Clamp the number of pixel tracks to the capacity of the SoA #38883

fwyzard · 2022-07-28T15:38:39Z

please test

cmsbuild · 2022-07-28T19:51:09Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-51580b/26509/summary.html
COMMIT: 9c1a5dc
CMSSW: CMSSW_12_4_X_2022-07-28-1100/el8_amd64_gcc10
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38884/26509/install.sh to create a dev area with all the needed externals and cmssw changes.

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
Reco comparison had 3 failed jobs
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 19876
DQMHistoTests: Total failures: 8
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 19868
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
Checked 12 log files, 9 edm output root files, 4 DQM output files
TriggerResults: found differences in 1 / 3 workflows

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3676198
DQMHistoTests: Total failures: 2
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3676174
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

clacaputo · 2022-07-29T10:21:35Z

+reconstruction

cmsbuild · 2022-07-29T10:21:53Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_4_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_5_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

qliphy · 2022-07-30T01:18:02Z

hold
as in the master #38883 (comment)

cmsbuild · 2022-07-30T01:18:22Z

Pull request has been put on hold by @qliphy
They need to issue an unhold command to remove the hold state or L1 can unhold it for all

fwyzard · 2022-07-30T07:27:41Z

unhold
as in the master #38883 (comment)

fwyzard · 2022-07-30T07:27:51Z

please test

fwyzard · 2022-07-30T08:00:10Z

please test

cmsbuild · 2022-07-30T08:00:15Z

Pull request #38884 was updated. @jpata, @clacaputo can you please check and sign again.

cmsbuild · 2022-07-30T12:04:20Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-51580b/26542/summary.html
COMMIT: d04920f
CMSSW: CMSSW_12_4_X_2022-07-29-2300/el8_amd64_gcc10
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38884/26542/install.sh to create a dev area with all the needed externals and cmssw changes.

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
Reco comparison had 2 failed jobs
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 19876
DQMHistoTests: Total failures: 8
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 19868
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
Checked 12 log files, 9 edm output root files, 4 DQM output files
TriggerResults: found differences in 2 / 3 workflows

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3676198
DQMHistoTests: Total failures: 2
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3676174
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

clacaputo · 2022-07-31T06:54:07Z

+reconstruction

fwyzard · 2022-08-01T07:57:41Z

urgent

qliphy · 2022-08-01T08:30:49Z

unhold

cmsbuild · 2022-08-01T08:31:09Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_4_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_5_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

qliphy · 2022-08-02T00:51:32Z

+1

cmsbuild added this to the CMSSW_12_4_X milestone Jul 28, 2022

cmsbuild added backport bug-fix orp-pending pending-signatures reconstruction-pending tests-pending labels Jul 28, 2022

cmsbuild added tests-started and removed tests-pending labels Jul 28, 2022

fwyzard changed the title ~~Clamp the number of pixel tracks to the capacity of the SoA~~ Clamp the number of pixel tracks to the capacity of the SoA [12.4.x] Jul 28, 2022

cmsbuild added tests-approved and removed tests-started labels Jul 28, 2022

cmsbuild added fully-signed reconstruction-approved and removed reconstruction-pending pending-signatures labels Jul 29, 2022

cmsbuild added hold pending-signatures and removed fully-signed labels Jul 30, 2022

Clamp the number of tracks to the capacity of the SoA

a9cb462

cmsbuild added tests-started and removed tests-pending labels Jul 30, 2022

fwyzard added 2 commits July 30, 2022 09:59

Warn if the number of pixel track candidates was truncated

9ebe6f3

Simplify the final emplace

d04920f

fwyzard force-pushed the Clamp_the_number_of_pixel_tracks_to_the_capacity_of_the_SoA branch from f3c7494 to d04920f Compare July 30, 2022 07:59

cmsbuild added tests-approved and removed tests-started labels Jul 30, 2022

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Jul 31, 2022

cmsbuild added the urgent label Aug 1, 2022

cmsbuild added fully-signed backport-ok and removed pending-signatures hold backport labels Aug 1, 2022

fwyzard mentioned this pull request Aug 1, 2022

assertion failed in RecoLocalTracker/SiPixelClusterizer/plugins/gpuClustering.h #38922

Open

cmsbuild added orp-approved and removed orp-pending labels Aug 2, 2022

cmsbuild merged commit 71da596 into cms-sw:CMSSW_12_4_X Aug 2, 2022

fwyzard deleted the Clamp_the_number_of_pixel_tracks_to_the_capacity_of_the_SoA branch October 29, 2022 08:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clamp the number of pixel tracks to the capacity of the SoA [12.4.x] #38884

Clamp the number of pixel tracks to the capacity of the SoA [12.4.x] #38884

fwyzard commented Jul 28, 2022 •

edited

fwyzard commented Jul 28, 2022

fwyzard commented Jul 28, 2022

fwyzard commented Jul 28, 2022

cmsbuild commented Jul 28, 2022 •

edited

fwyzard commented Jul 28, 2022

cmsbuild commented Jul 28, 2022

clacaputo commented Jul 29, 2022

cmsbuild commented Jul 29, 2022

qliphy commented Jul 30, 2022

cmsbuild commented Jul 30, 2022

fwyzard commented Jul 30, 2022

fwyzard commented Jul 30, 2022

fwyzard commented Jul 30, 2022

cmsbuild commented Jul 30, 2022

cmsbuild commented Jul 30, 2022

clacaputo commented Jul 31, 2022

fwyzard commented Aug 1, 2022

qliphy commented Aug 1, 2022

cmsbuild commented Aug 1, 2022

qliphy commented Aug 2, 2022

Clamp the number of pixel tracks to the capacity of the SoA [12.4.x] #38884

Clamp the number of pixel tracks to the capacity of the SoA [12.4.x] #38884

Conversation

fwyzard commented Jul 28, 2022 • edited

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR.

fwyzard commented Jul 28, 2022

fwyzard commented Jul 28, 2022

fwyzard commented Jul 28, 2022

cmsbuild commented Jul 28, 2022 • edited

fwyzard commented Jul 28, 2022

cmsbuild commented Jul 28, 2022

GPU Comparison Summary

Comparison Summary

clacaputo commented Jul 29, 2022

cmsbuild commented Jul 29, 2022

qliphy commented Jul 30, 2022

cmsbuild commented Jul 30, 2022

fwyzard commented Jul 30, 2022

fwyzard commented Jul 30, 2022

fwyzard commented Jul 30, 2022

cmsbuild commented Jul 30, 2022

cmsbuild commented Jul 30, 2022

GPU Comparison Summary

Comparison Summary

clacaputo commented Jul 31, 2022

fwyzard commented Aug 1, 2022

qliphy commented Aug 1, 2022

cmsbuild commented Aug 1, 2022

qliphy commented Aug 2, 2022

fwyzard commented Jul 28, 2022 •

edited

cmsbuild commented Jul 28, 2022 •

edited