Speedup strip digitization #40618

VinInn · 2023-01-26T12:46:08Z

Speed up of Strip digitization

a)
ZeroSuppressor done in two loops: one compares adc with thresholds, the second performs the algo logic.
(it fixes also a defect in the previous implementation (fix done ONLY for the digitizer version))
speed up x2
b) remove maps and various copies in PreMixingSiStripWorker
c) optimize and make consistent math in single precision

igprof
original
http://innocent.web.cern.ch/innocent/perfResults/igprof-navigator/digiDebug1T_ori_CMSSW_13_0_X_2023-01-24-1100_slc7_amd64_gcc11/24

this PR

http://innocent.web.cern.ch/innocent/perfResults/igprof-navigator/digiDebug1T_speedStrip5_CMSSW_13_0_X_2023-01-24-1100_slc7_amd64_gcc11/26

purely technical, regression expected due to the bug fix.

I'm under the impression that using a single-precision random number generator may speed up all those gaussian noise generators by a large amount.
I think that this would require a dedicated "project".

VinInn · 2023-01-26T12:50:10Z

RecoLocalTracker/SiStripZeroSuppression/src/SiStripFedZeroSuppression.cc

-        adcPrev2 = in[i - 2].adc();
-        thePrev2FEDlowThresh = static_cast<int16_t>(
-            threshold.getData(strip - 2, detThRange).getLth() * noise.getNoiseFast(strip - 2, detNoiseRange) + 0.5);
+        ldata.statPrev2 = stat[i - 2];
      }
    }



the old logic below was wrong in case say adcNext <= adcPrev but prev is below and next is above

cmsbuild · 2023-01-26T12:53:40Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40618/33891

This PR adds an extra 28KB to repository

cmsbuild · 2023-01-26T12:54:07Z

A new Pull Request was created by @VinInn (Vincenzo Innocente) for master.

It involves the following packages:

RecoLocalTracker/SiStripZeroSuppression (reconstruction)
SimTracker/SiStripDigitizer (simulation)

@cmsbuild, @civanch, @mandrenguyen, @clacaputo, @mdhildreth can you please review it and eventually sign? Thanks.
@echabert, @VourMa, @swertz, @missirol, @GiacomoSguazzoni, @yduhm, @robervalwalsh, @JanFSchulte, @rovere, @VinInn, @alesaggio, @prolay, @gpetruc, @felicepantaleo, @mtosi, @gbenelli, @mmusich, @threus, @dgulhan this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

civanch · 2023-01-26T13:17:31Z

please test

mandrenguyen · 2023-01-26T14:48:32Z

@civanch Should we enable the profiling?

civanch · 2023-01-26T15:21:22Z

@mandrenguyen , we establish CPU/memory/profiling of top of some IB, we are doing this for pre-releases. Vincenzo made CPU check in unit test with the factor 2 for this method, what will be summary effect we do may try to measure privately or if we ask Natascha for her standard analysis of performance.

cmsbuild · 2023-01-26T15:54:06Z

-1

Failed Tests: RelVals-INPUT AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-267794/30191/summary.html
COMMIT: 51a04f3
CMSSW: CMSSW_13_0_X_2023-01-25-2300/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40618/30191/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

10424.010424.0_TTbar_13+2017Design/step2_TTbar_13+2017Design.log
11224.011224.0_TTbar_13+2018Design/step2_TTbar_13+2018Design.log
11224.611224.6_TTbar_13+2018Design_heCollapse/step2_TTbar_13+2018Design_heCollapse.log

Expand to see more relval errors ...

AddOn Tests

hlt_mc_GRun

[hlt_mc_GRun:1] cmsDriver.py TTbar_13TeV_TuneCUETP8M1_cfi -s GEN,SIM,DIGI,L1,DIGI2RAW --mc --scenario=pp -n 10 --conditions auto:run3_mc_GRun --relval 9000,50 --datatier "GEN-SIM-RAW" --eventcontent RAWSIM --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run3 --fileout file:RelVal_Raw_GRun_MC.root : FAILED - elapsed time: 267 sec (ended on Thu Jan 26 14:39:05 2023) - exit: 34304

hlt_mc_GRun

----- Begin Fatal Exception 26-Jan-2023 14:39:53 CET-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   Additional Info:
      [a] Input file file:RelVal_Raw_GRun_MC.root could not be opened.
      [b] Fatal Root Error: @SUB=TStorageFactoryFile::Init
file RelVal_Raw_GRun_MC.root probably not closed, trying to recover

----- End Fatal Exception -------------------------------------------------

hlt_mc_GRun

----- Begin Fatal Exception 26-Jan-2023 14:41:25 CET-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   Additional Info:
      [a] Input file file:RelVal_Raw_GRun_MC.root could not be opened.
      [b] Fatal Root Error: @SUB=TStorageFactoryFile::Init
file RelVal_Raw_GRun_MC.root probably not closed, trying to recover

----- End Fatal Exception -------------------------------------------------

Expand to see more addon errors ...

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 87 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3555495
DQMHistoTests: Total failures: 7304
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3548169
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.309 KiB( 48 files compared)
DQMHistoSizes: changed ( 7.3 ): 0.309 KiB SiStrip/MechanicalView
Checked 211 log files, 162 edm output root files, 49 DQM output files
TriggerResults: no differences found

VinInn · 2023-02-14T12:07:26Z

Is there anything blocking integration (and/or further review)?

cmsbuild · 2023-02-14T12:30:24Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40618/34179

This PR adds an extra 16KB to repository

cmsbuild · 2023-02-14T12:30:48Z

Pull request #40618 was updated. @civanch, @mandrenguyen, @clacaputo, @mdhildreth can you please check and sign again.

cmsbuild · 2023-02-14T16:53:55Z

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-267794/30624/summary.html
COMMIT: d5c93fc
CMSSW: CMSSW_13_1_X_2023-02-14-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/40618/30624/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

The relvals timed out after 4 hours.

Comparison Summary

Summary:

You potentially removed 12 lines from the logs
Reco comparison results: 248 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3555972
DQMHistoTests: Total failures: 9256
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3546694
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.309 KiB( 48 files compared)
DQMHistoSizes: changed ( 7.3 ): 0.309 KiB SiStrip/MechanicalView
Checked 213 log files, 164 edm output root files, 49 DQM output files
TriggerResults: no differences found

VinInn · 2023-02-15T10:11:35Z

"The relvals timed out after 4 hours."
Shall I assume this is independent of this PR?

mmusich · 2023-02-15T10:16:08Z

Shall I assume this is independent of this PR?

I've seen it plenty of times in other PRs, so I think at this stage it can be ignored.
@civanch feel free to review. Thanks

civanch · 2023-02-15T10:20:07Z

please test

civanch · 2023-02-15T10:21:09Z

Such timeouts happens and I do not know other solution than resubmit the test.

cmsbuild · 2023-02-15T13:41:22Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-267794/30643/summary.html
COMMIT: d5c93fc
CMSSW: CMSSW_13_1_X_2023-02-14-2300/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/40618/30643/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 32 lines to the logs
Reco comparison results: 254 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3556272
DQMHistoTests: Total failures: 9256
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3546994
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.309 KiB( 48 files compared)
DQMHistoSizes: changed ( 7.3 ): 0.309 KiB SiStrip/MechanicalView
Checked 213 log files, 164 edm output root files, 49 DQM output files
TriggerResults: no differences found

civanch · 2023-02-15T18:00:25Z

+1

some differences in comparisons are expected.

mandrenguyen · 2023-02-15T18:11:09Z

+1

cmsbuild · 2023-02-15T18:11:36Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

mmusich · 2023-02-15T18:18:05Z

release-notes: expected minor regressions from fix of ZeroSuppression algorithm

perrotta · 2023-02-15T18:20:50Z

+1

VinInn added 6 commits January 25, 2023 10:16

fast return for large adc

d5e2c61

make math consistent in float

4fe57b7

remove useless maps, avoid copies

7e0f725

fix stupid bug and remove one more copy

3152ed7

add useless shortcuts

2fe3c19

avoid multiple loading of noise

51a04f3

cmsbuild added this to the CMSSW_13_0_X milestone Jan 26, 2023

cmsbuild added code-checks-pending orp-pending pending-signatures reconstruction-pending simulation-pending tests-pending trk labels Jan 26, 2023

VinInn commented Jan 26, 2023

View reviewed changes

cmsbuild added code-checks-approved and removed code-checks-pending labels Jan 26, 2023

cmsbuild added tests-started and removed tests-pending labels Jan 26, 2023

cmsbuild added tests-rejected and removed tests-started labels Jan 26, 2023

smuzaffar modified the milestones: CMSSW_13_0_X, CMSSW_13_1_X Feb 11, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Feb 14, 2023

cmsbuild added tests-rejected and removed tests-started labels Feb 14, 2023

cmsbuild added tests-started and removed tests-rejected labels Feb 15, 2023

cmsbuild added tests-approved and removed tests-started labels Feb 15, 2023

cmsbuild added simulation-approved and removed simulation-pending labels Feb 15, 2023

cmsbuild added reconstruction-approved fully-signed and removed reconstruction-pending pending-signatures labels Feb 15, 2023

cmsbuild added orp-approved and removed orp-pending labels Feb 15, 2023

cmsbuild merged commit 3aa77af into cms-sw:master Feb 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Speedup strip digitization #40618

Speedup strip digitization #40618

VinInn commented Jan 26, 2023

VinInn Jan 26, 2023

cmsbuild commented Jan 26, 2023

cmsbuild commented Jan 26, 2023

civanch commented Jan 26, 2023

mandrenguyen commented Jan 26, 2023

civanch commented Jan 26, 2023

cmsbuild commented Jan 26, 2023

VinInn commented Feb 14, 2023

cmsbuild commented Feb 14, 2023

cmsbuild commented Feb 14, 2023

cmsbuild commented Feb 14, 2023

VinInn commented Feb 15, 2023

mmusich commented Feb 15, 2023

civanch commented Feb 15, 2023

civanch commented Feb 15, 2023

cmsbuild commented Feb 15, 2023

civanch commented Feb 15, 2023

mandrenguyen commented Feb 15, 2023

cmsbuild commented Feb 15, 2023

mmusich commented Feb 15, 2023

perrotta commented Feb 15, 2023

Speedup strip digitization #40618

Speedup strip digitization #40618

Conversation

VinInn commented Jan 26, 2023

VinInn Jan 26, 2023

Choose a reason for hiding this comment

cmsbuild commented Jan 26, 2023

cmsbuild commented Jan 26, 2023

civanch commented Jan 26, 2023

mandrenguyen commented Jan 26, 2023

civanch commented Jan 26, 2023

cmsbuild commented Jan 26, 2023

RelVals-INPUT

AddOn Tests

Comparison Summary

VinInn commented Feb 14, 2023

cmsbuild commented Feb 14, 2023

cmsbuild commented Feb 14, 2023

cmsbuild commented Feb 14, 2023

RelVals-INPUT

Comparison Summary

VinInn commented Feb 15, 2023

mmusich commented Feb 15, 2023

civanch commented Feb 15, 2023

civanch commented Feb 15, 2023

cmsbuild commented Feb 15, 2023

Comparison Summary

civanch commented Feb 15, 2023

mandrenguyen commented Feb 15, 2023

cmsbuild commented Feb 15, 2023

mmusich commented Feb 15, 2023

perrotta commented Feb 15, 2023