Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initialise the errors_ data member of SiPixelDigiErrorsSoAFromCUDA #38476

Merged

Conversation

fwyzard
Copy link
Contributor

@fwyzard fwyzard commented Jun 22, 2022

PR description:

Initialise the errors_ data member of the SiPixelDigiErrorsSoAFromCUDA producer.

Prevents reading from uninitialised memory when the first event of a job has zero error words, and fixes a crash observed at HLT during data taking.

PR validation:

Without this fix, it is possible to reproduce the error observed online running over the "error stream" events that triggered it.
With this fix, no crash is observed after running 30+ times on the same input.

Prevents reading from uninitialised memory when the first event of a
job has zero error words.

Fixes a crash observed at HLT during data taking.
@fwyzard
Copy link
Contributor Author

fwyzard commented Jun 22, 2022

type bug-fix

@fwyzard
Copy link
Contributor Author

fwyzard commented Jun 22, 2022

urgent

@fwyzard
Copy link
Contributor Author

fwyzard commented Jun 22, 2022

enable gpu

@fwyzard
Copy link
Contributor Author

fwyzard commented Jun 22, 2022

please test

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-38476/30692

  • This PR adds an extra 12KB to repository

  • There are other open Pull requests which might conflict with changes you have proposed:

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @fwyzard (Andrea Bocci) for master.

It involves the following packages:

  • EventFilter/SiPixelRawToDigi (reconstruction)

@jpata, @clacaputo can you please review it and eventually sign? Thanks.
@OzAmram, @VinInn, @Martin-Grunewald, @missirol, @dkotlins, @ferencek, @tvami this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3206f6/25711/summary.html
COMMIT: 532c257
CMSSW: CMSSW_12_5_X_2022-06-22-1100/el8_amd64_gcc10
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/38476/25711/install.sh to create a dev area with all the needed externals and cmssw changes.

GPU Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 4
  • DQMHistoTests: Total histograms compared: 19864
  • DQMHistoTests: Total failures: 9
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 19855
  • DQMHistoTests: Total skipped: 0
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
  • Checked 12 log files, 9 edm output root files, 4 DQM output files
  • TriggerResults: no differences found

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 6 differences found in the comparisons
  • DQMHistoTests: Total files compared: 50
  • DQMHistoTests: Total histograms compared: 3659099
  • DQMHistoTests: Total failures: 13
  • DQMHistoTests: Total nulls: 1
  • DQMHistoTests: Total successes: 3659063
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.004 KiB( 49 files compared)
  • DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
  • Checked 208 log files, 45 edm output root files, 50 DQM output files
  • TriggerResults: no differences found

@VinInn
Copy link
Contributor

VinInn commented Jun 23, 2022

and why valgrind, ubsan, asan etc did not catch it?

@fwyzard
Copy link
Contributor Author

fwyzard commented Jun 23, 2022

I think, because the problem only happens if the first event has gpuDigiErrors.nErrorWords() == 0.
If the first event has some errors, the SimpleVector is initialised when reading the data from the gpu, and for all events after the SimpleVector is reset inside produce() after putting the produce in the event.

@clacaputo
Copy link
Contributor

+reconstruction

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

@perrotta
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit a2029cb into cms-sw:master Jun 24, 2022
@fwyzard fwyzard deleted the fix_SiPixelDigiErrorsSoAFromCUDA_12.5.x branch July 31, 2022 13:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants