[HGCAL] Pattern Recognition as Plugins, with CLUE3D addition. #33645

rovere · 2021-05-06T11:48:48Z

PR description:

This PR will move the current Pattern Recognition algorithms in HGCAL to become plugins.
In addition to that, a new Pattern Recognition algorithm, CLUE3D, has been added, as a plugin. For the time being, it's not run in production, but it has been tested.
Pattern Recognition parameters have been moved directly into the corresponding plugins.

Expect no regression.

PR validation:

Typical single-particle workflows related to HGCAL. Tested enabling and disabling CLUE3D.

cmsbuild · 2021-05-06T11:53:46Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-33645/22521

This PR adds an extra 104KB to repository
Found files with invalid states:
- RecoHGCal/TICL/python/CLUE3DStep_cff.py:
  - Added: 48a9960
  - Modified: b7cc62d, aaf817d
  - Deleted: 9c37f1f
There are other open Pull requests which might conflict with changes you have proposed:
- File RecoHGCal/TICL/python/iterativeTICL_cff.py modified in PR(s): [DD4hep] start on geometry XML payload producer #33548

cmsbuild · 2021-05-06T11:54:08Z

A new Pull Request was created by @rovere (Marco Rovere) for master.

It involves the following packages:

RecoHGCal/TICL

@perrotta, @kpedro88, @cmsbuild, @srimanob, @slava77, @jpata can you please review it and eventually sign? Thanks.
@felicepantaleo, @apsallid, @sobhatta, @lecriste, @hatakeyamak, @ebrondol, @clelange this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

rovere · 2021-05-06T11:56:10Z

@cmsbuild please test

cmsbuild · 2021-05-06T14:29:14Z

-1

Failed Tests: HeaderConsistency
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b59406/14910/summary.html
COMMIT: 2e3959e
CMSSW: CMSSW_12_0_X_2021-05-05-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/33645/14910/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

The workflows 140.53 have different files in step1_dasquery.log than the ones found in the baseline. You may want to check and retrigger the tests if necessary. You can check it in the "files" directory in the results of the comparisons

Summary:

No significant changes to the logs found
Reco comparison results: 1267 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2662646
DQMHistoTests: Total failures: 3673
DQMHistoTests: Total nulls: 19
DQMHistoTests: Total successes: 2658932
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -45.703 KiB( 36 files compared)
DQMHistoSizes: changed ( 140.53 ): -44.531 KiB Hcal/DigiRunHarvesting
DQMHistoSizes: changed ( 140.53 ): -1.172 KiB RPC/DCSInfo
Checked 155 log files, 37 edm output root files, 37 DQM output files
TriggerResults: no differences found

lecriste · 2021-05-06T14:40:15Z

RecoHGCal/TICL/plugins/TrackstersProducer.cc

@@ -92,9 +90,17 @@ TrackstersProducer::TrackstersProducer(const edm::ParameterSet& ps, const Tracks
          consumes<std::vector<TICLSeedingRegion>>(ps.getParameter<edm::InputTag>("seeding_regions"))),
      itername_(ps.getParameter<std::string>("itername")) {
  if (doNose_) {
+    auto plugin = ps.getParameter<std::string>("patternRecognitionBy");


This and next line can be moved out of the if, removing their repetiion in the else.

lecriste · 2021-05-06T14:41:28Z

RecoHGCal/TICL/python/CLUE3DHighStep_cff.py

+
+# MULTICLUSTERS
+
+ticlMultiClustersFromTrackstersCLUE3DHigh = _multiClustersFromTrackstersProducer.clone(


Do we still want to produce these MultiClusters?

lecriste · 2021-05-06T14:42:24Z

RecoHGCal/TICL/python/CLUE3DLowStep_cff.py

+
+# MULTICLUSTERS
+
+ticlMultiClustersFromTrackstersCLUE3DLow = _multiClustersFromTrackstersProducer.clone(


As above, do we still want to produce these MultiClusters?

cmsbuild · 2021-05-06T15:06:01Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-33645/22530

This PR adds an extra 20KB to repository
Found files with invalid states:
- RecoHGCal/TICL/python/CLUE3DStep_cff.py:
  - Added: 48a9960
  - Modified: b7cc62d, aaf817d
  - Deleted: 9c37f1f
There are other open Pull requests which might conflict with changes you have proposed:
- File RecoHGCal/TICL/python/iterativeTICL_cff.py modified in PR(s): [DD4hep] start on geometry XML payload producer #33548

cmsbuild · 2021-05-06T15:06:20Z

Pull request #33645 was updated. @perrotta, @kpedro88, @cmsbuild, @srimanob, @slava77, @jpata can you please check and sign again.

cmsbuild · 2021-05-21T19:33:01Z

Pull request #33645 was updated. @perrotta, @silviodonato, @franzoni, @kpedro88, @cmsbuild, @srimanob, @slava77, @jpata, @qliphy, @fabiocos, @davidlange6 can you please check and sign again.

rovere · 2021-05-21T19:33:14Z

@cmsbuild please test

slava77 · 2021-05-21T19:35:55Z

Pull request #33645 was updated.

I was just starting with the follow-up review.

@rovere
please clarify if you are done with updates or if something is still pending.
Thank you.

cmsbuild · 2021-05-21T22:23:05Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b59406/15248/summary.html
COMMIT: d501c9e
CMSSW: CMSSW_12_0_X_2021-05-21-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/33645/15248/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 7 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2650486
DQMHistoTests: Total failures: 12
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2650451
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 36 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 155 log files, 37 edm output root files, 37 DQM output files
TriggerResults: no differences found

rovere · 2021-05-21T22:42:57Z

@slava77
Done, thanks.

rovere · 2021-05-26T12:59:20Z

Ciao,
is there any hope this PR would enter pre2?

slava77 · 2021-05-27T03:45:46Z

+reconstruction

for #33645 d501c9e

code changes are in line with the PR description and reasonably follow the code review
jenkins tests pass and comparisons with the baseline show no (relevant) differences as expected for the part of the PR where the currently running code is refactored/reorganized in a mostly technical way
a local test with wf 34834.999 (PU50 phase-2) and clue3D process modifier (also with relevant keep for the output tracksters) show that the High and Low variants of CLUE3D run to completion with somewhat reasonable but perhaps still to be tuned/reduced time of 220-230 ms/evt each (vs e.g. 107 ms/evt in ticlTrackstersEM)

srimanob · 2021-05-27T12:46:42Z

+Upgrade

This PR is to organize the PR algorithms to become plugin, and introduce new CLUE3D PR. No change is expected on wf as of now.

silviodonato · 2021-05-27T15:16:11Z

+1

cmsbuild · 2021-05-27T15:16:33Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.

cmsbuild added this to the CMSSW_12_0_X milestone May 6, 2021

cmsbuild added code-checks-pending orp-pending pending-signatures reconstruction-pending tests-pending upgrade-pending labels May 6, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels May 6, 2021

cmsbuild added tests-started and removed tests-pending labels May 6, 2021

cmsbuild added tests-rejected and removed tests-started labels May 6, 2021

lecriste reviewed May 6, 2021

View reviewed changes

cmsbuild added code-checks-pending tests-pending and removed code-checks-approved tests-rejected labels May 6, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels May 6, 2021

cmsbuild added code-checks-pending and removed code-checks-approved labels May 6, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels May 21, 2021

cmsbuild added tests-started and removed tests-pending labels May 21, 2021

cmsbuild added tests-approved and removed tests-started labels May 21, 2021

cmsbuild mentioned this pull request May 26, 2021

HGCal/TICL: Examples of clang-tidy replacement of getByToken with getHandle #33853

Closed

cmsbuild added reconstruction-approved and removed reconstruction-pending labels May 27, 2021

cmsbuild added upgrade-approved and removed upgrade-pending labels May 27, 2021

cmsbuild added fully-signed operations-approved orp-approved and removed operations-pending orp-pending pending-signatures labels May 27, 2021

cmsbuild merged commit 4162d66 into cms-sw:master May 27, 2021

This was referenced May 27, 2021

Another fix for static analyzer script. cms-sw/cms-bot#1558

Merged

Clean up Utilities/StaticAnalyers scripts to avoid errors because of formatting. #33870

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[HGCAL] Pattern Recognition as Plugins, with CLUE3D addition. #33645

[HGCAL] Pattern Recognition as Plugins, with CLUE3D addition. #33645

rovere commented May 6, 2021

cmsbuild commented May 6, 2021

cmsbuild commented May 6, 2021

rovere commented May 6, 2021

cmsbuild commented May 6, 2021

lecriste May 6, 2021

lecriste May 6, 2021

lecriste May 6, 2021

cmsbuild commented May 6, 2021

cmsbuild commented May 6, 2021

cmsbuild commented May 21, 2021

rovere commented May 21, 2021

slava77 commented May 21, 2021

cmsbuild commented May 21, 2021

rovere commented May 21, 2021

rovere commented May 26, 2021

slava77 commented May 27, 2021

srimanob commented May 27, 2021

silviodonato commented May 27, 2021

cmsbuild commented May 27, 2021


		# MULTICLUSTERS

		ticlMultiClustersFromTrackstersCLUE3DHigh = _multiClustersFromTrackstersProducer.clone(


		# MULTICLUSTERS

		ticlMultiClustersFromTrackstersCLUE3DLow = _multiClustersFromTrackstersProducer.clone(

[HGCAL] Pattern Recognition as Plugins, with CLUE3D addition. #33645

[HGCAL] Pattern Recognition as Plugins, with CLUE3D addition. #33645

Conversation

rovere commented May 6, 2021

PR description:

PR validation:

cmsbuild commented May 6, 2021

cmsbuild commented May 6, 2021

rovere commented May 6, 2021

cmsbuild commented May 6, 2021

Comparison Summary

lecriste May 6, 2021

Choose a reason for hiding this comment

lecriste May 6, 2021

Choose a reason for hiding this comment

lecriste May 6, 2021

Choose a reason for hiding this comment

cmsbuild commented May 6, 2021

cmsbuild commented May 6, 2021

cmsbuild commented May 21, 2021

rovere commented May 21, 2021

slava77 commented May 21, 2021

cmsbuild commented May 21, 2021

Comparison Summary

rovere commented May 21, 2021

rovere commented May 26, 2021

slava77 commented May 27, 2021

srimanob commented May 27, 2021

silviodonato commented May 27, 2021

cmsbuild commented May 27, 2021