Detect and remove all duplicate pixels [12.4.x]

[fwyzard]

PR description:

Detect and remove all duplicate pixels, after unpacking each pixel module but before running the clustering.
Use shared memory for inter-thread communication and to speed up marking and detecting the duplicates.

PR validation:

Running the online pixel reconstruction and the full HLT menu on GPU over non-problematic events shows only a moderate slow down:

reconstruction	pixel tracking	full HLT
no duplicate removal	1566 ± 16 ev/s (--)	873 ± 4 ev/s (--)
duplicate removal with atomicOR (ce8a57b)	1530 ± 17 ev/s (-2.3%)	872 ± 4 ev/s (-0.2%)
duplicate removal with atomicCAS (2522012)	1519 ± 14 ev/s (-3.0%)	869 ± 2 ev/s (-0.4%)

If this PR is a backport please specify the original PR and why you need to backport that PR.

Backport of #38946 for data taking.

[fwyzard]

type bugfix

[cmsbuild]

A new Pull Request was created by @fwyzard (Andrea Bocci) for CMSSW_12_4_X.

It involves the following packages:

• DataFormats/SiPixelDigi (simulation)
• HeterogeneousCore/CUDAUtilities (heterogeneous)
• RecoLocalTracker/SiPixelClusterizer (reconstruction)

@civanch, @makortel, @clacaputo, @cmsbuild, @mdhildreth, @fwyzard, @jpata can you please review it and eventually sign? Thanks.
@mtosi, @VourMa, @makortel, @felicepantaleo, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @OzAmram, @ferencek, @dkotlins, @gpetruc, @mmusich, @threus, @tvami this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy, @rappoccio you are the release manager for this.

cms-bot commands are listed here

• Backported from Detect and remove all duplicate pixels #38946

[fwyzard]

enable gpu

[fwyzard]

please test

[fwyzard]

backport #38946

[fwyzard]

urgent

[fwyzard]

+heterogeneous

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-af153f/26604/summary.html
COMMIT: ce8a57b
CMSSW: CMSSW_12_4_X_2022-08-02-1100/el8_amd64_gcc10
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/38947/26604/install.sh to create a dev area with all the needed externals and cmssw changes.

GPU Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 4
• DQMHistoTests: Total histograms compared: 19876
• DQMHistoTests: Total failures: 8
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 19868
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
• Checked 12 log files, 9 edm output root files, 4 DQM output files
• TriggerResults: found differences in 1 / 3 workflows

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 3676198
• DQMHistoTests: Total failures: 2
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3676174
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
• Checked 208 log files, 45 edm output root files, 50 DQM output files
• TriggerResults: no differences found

[civanch]

+1

[cmsbuild]

Pull request #38947 was updated. @civanch, @makortel, @clacaputo, @cmsbuild, @mdhildreth, @fwyzard, @jpata can you please check and sign again.

[fwyzard]

enable gpu

[fwyzard]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-af153f/26626/summary.html
COMMIT: 2522012
CMSSW: CMSSW_12_4_X_2022-08-03-1100/el8_amd64_gcc10
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/38947/26626/install.sh to create a dev area with all the needed externals and cmssw changes.

GPU Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• Reco comparison had 3 failed jobs
• DQMHistoTests: Total files compared: 4
• DQMHistoTests: Total histograms compared: 19876
• DQMHistoTests: Total failures: 36
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 19840
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
• Checked 12 log files, 9 edm output root files, 4 DQM output files
• TriggerResults: no differences found

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 3676198
• DQMHistoTests: Total failures: 2
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3676174
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
• Checked 208 log files, 45 edm output root files, 50 DQM output files
• TriggerResults: no differences found

[civanch]

+1

[fwyzard]

+heterogeneous

[clacaputo]

+reconstruction

• backport Detect and remove all duplicate pixels #38946

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_4_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_5_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

[qliphy]

+1

[fabiocos]

I see that this PR updates a DataFormat, although just adding a method to retrieve information, there is no change in data members or interfaces, so probably it is still ok for a patch release (which is already built)