speedup of trackValidatorJetCore and TrackAssociatorByChi2

[slava77]

trackValidatorJetCore is currently enabled only with seedingDeepCore; which is not in production yet.
Compared to other tracking validation cases, this one is using association by chi2.
As clarified in #33531 (comment)
The trackAssociatorByChi2 is needed to have a "fair" performance estimation with DeepCore. DeepCore uses pixel-less seed [1], thus the associator by hits is not optimal in this case.

A problem in timing was observed while testing #33531 , as described in #33531 (comment) (and related messages). There are two components to resolve the problem:

• reduce the set of tracking particles to only the in-time ones (BX = 0)
• improve computational implementation of TrackAssociatorByChi2 by reducing unnecessary computations done in a nested double-loop (nTrack * nTrackingParticles )

In the test setup based on 11834.0 with seedingDeepCore modifier, the time in trackValidatorJetCore is reduced from around 5400 s/event (averaged over the first 9 events) to 3.4 s/evt (averaged over 200 events).

Updates were tested on wf 11651 with seedingDeepCore (only in-time particles) to confirm that there are no changes in the DQM plots after the speedup implemented in TrackAssociatorByChi2

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-33731/22670

• This PR adds an extra 28KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Validation/RecoTrack/python/TrackValidation_cff.py modified in PR(s): update deepCore for Run3 #33531, EXO regional tracking iteration #32806

[slava77]

@cmsbuild please test workflow 11723.17

[cmsbuild]

A new Pull Request was created by @slava77 (Slava Krutelyov) for master.

It involves the following packages:

SimTracker/TrackAssociation
SimTracker/TrackAssociatorProducers
Validation/RecoTrack

@civanch, @kmaeshima, @andrius-k, @mdhildreth, @ErnestaP, @jfernan2, @ahmad3213, @rvenditti can you please review it and eventually sign? Thanks.
@makortel, @felicepantaleo, @abbiendi, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @jhgoh, @wmtford, @mtosi, @ebrondol, @threus, @dgulhan this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[mtosi]

thanks A LOT @slava77 !!!!!!!!!

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3dcf7d/15084/summary.html
COMMIT: ac99d68
CMSSW: CMSSW_12_0_X_2021-05-13-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/33731/15084/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-3dcf7d/11723.17_QCD_Pt_1800_2400_14+2021_seedingDeepCore+QCD_Pt_1800_2400_14TeV_TuneCP5_GenSim+Digi+Reco+HARVEST

Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 37
• DQMHistoTests: Total histograms compared: 2648242
• DQMHistoTests: Total failures: 7
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2648213
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 36 files compared)
• Checked 155 log files, 37 edm output root files, 37 DQM output files
• TriggerResults: no differences found

[slava77]

In the test setup based on 11834.0 with seedingDeepCore modifier, the time in trackValidatorJetCore is reduced from around 5400 s/event (averaged over the first 9 events) to 3.4 s/evt (averaged over 200 events).

sadly, while running this on top of #33531 trackValidatorJetCore is still extremely slow, investigating.

[slava77]

sadly, while running this on top of #33531 trackValidatorJetCore is still extremely slow, investigating.

that was a false alarm, I didn't run the right thing.
I confirm that the timing speedup is present on top of #33531 as well

[civanch]

+1

[jfernan2]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[silviodonato]

+1