Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TICL Reorganization, reloaded. #27160

Merged
merged 17 commits into from Jun 26, 2019

Conversation

rovere
Copy link
Contributor

@rovere rovere commented Jun 11, 2019

PR description:

TICL Reorganization

Several changes entered this PR.

  • The data structures used by TICL have been moved into
    DataFormats/TICL, that has to be added as a CMSSW package.
    Tracksters and Tile are currently present. The encapsulation of
    std::array as a transient data member of a Tile is due to the fact
    that ROOT natively supports those and refuses to create ad-hoc
    dictionaries. Moreover, at present, an std::array cannot sit at the
    top of a ROOT branch requiring, once again, its encapsulation.
  • The "master Tile" has been added as a single producer. A unique Tile
    containing all the 2D reconstructed clusters is created once and put
    into the Event. All iterations will fetch it from the event and mask
    the 2D clusters on the fly. This required also some changes to the
    clusters filters logic. Now the filters produce also a vector
    that represents the result of merging their input mask with their
    own selection. TICL iterations will make use of the filter's
    vector to dynamically mask clusters from the "master Tile".
  • The fractions have been added to the Tracksters. The sharing of the
    energy of 2D clusters is flat and based on their multiplicity, i.e,
    a fraction of energy/multiplicity is assigned to all Tracksters that
    use the 2D clusters. Better splitting algorithms can be studies or
    Machine Learning techniques could be put at work here.
  • The hits and fraction of the Trackster are now correctly populated.
    This required an additional loop over the components of each 2D
    cluster. In the future the CaloCluster, PFCluster and hits and
    fractions data structures could be reviewed or, at least, TICL could
    use some other, more efficient, internal data representation and
    collapse into the legacy ones only at the very last stage.

PR validation:

runTheMatrix.py -l limited -i all -j 4

and other samples.

Note to the reviewers: all changes in RecoLocalCalo/HGCalRecAlgos are purely due to the code-format. No other changes have been committed there. I did so so that PR #26991 could be closed (if and when this PR gets merged)

@cmsbuild
Copy link
Contributor

The code-checks are being triggered in jenkins.

@rovere
Copy link
Contributor Author

rovere commented Jun 11, 2019

@slava77 for some weird reason PR #27026 got a little confused after my last rebase. I decided to close that and open a fresh new PR, hoping that this time the review process will go smoother.
I'll read the comments you made on #27026 and implement some of the suggestions.

As for moving stuff out of DataFormats/TICL into DataFormats/HGCRecHits, I disagree. That package has a very specific name and I'd not mix stuff together. I see the value of not adding too many packages, yet the proposed solution is not optimal, from my point of view. Either we change the name of the current DataFormats/HGCRecHit into something more generic, like HGCRecoCommonn, or any other name you could suggest, or I'd prefer to keep things logically separated into different folders.

I'll go through the other comments in the coming days.

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-27160/10308

  • This PR adds an extra 212KB to repository

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @rovere (Marco Rovere) for master.

It involves the following packages:

DataFormats/TICL
RecoHGCal/TICL
RecoLocalCalo/HGCalRecAlgos

The following packages do not have a category, yet:

DataFormats/TICL
Please create a PR for https://github.com/cms-sw/cms-bot/blob/master/categories_map.py to assign category

@perrotta, @cmsbuild, @kpedro88, @slava77 can you please review it and eventually sign? Thanks.
@edjtscott, @vandreev11, @sethzenz, @felicepantaleo, @kpedro88, @argiro, @cseez, @pfs, @lgray this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

@rovere
Copy link
Contributor Author

rovere commented Jun 11, 2019

@slava77 Also for the move of TICL algorithms into RecoLocalCalo, I believe this is misleading. TICL, or more in general HGCAL reconstruction, will have internally links and connections with other detectors, like tracker/muons. I'd prefer to have it in a separate package that explicitly declares its purpose.
RecoLocalCalo/HGCalRecAlgos, instead, contains correctly algorithms to perform real local (i.e. 2d, eventually 3D) clustering.

@perrotta
Copy link
Contributor

please test

@slava77
Copy link
Contributor

slava77 commented Jun 11, 2019

@slava77 Also for the move of TICL algorithms into RecoLocalCalo, I believe this is misleading. TICL, or more in general HGCAL reconstruction, will have internally links and connections with other detectors, like tracker/muons. I'd prefer to have it in a separate package that explicitly declares its purpose.
RecoLocalCalo/HGCalRecAlgos, instead, contains correctly algorithms to perform real local (i.e. 2d, eventually 3D) clustering.

I was apparently incorrect that the current status of this package represents the target as a skeleton.
From the comment above it sounds like the data formats are incomplete; that there will be objects linking HGCAL, tracker, and muon. So, it's not really a reco of HGCAL clusters.
If that's the case, RecoParticleFlow subsystem is perhaps more appropriate subsystem than RecoHGCAL.

About the DataFormats. Please remind me the motivation to move the transient-only products to the DataFormats subsystem.
It seems like even the Trackster is actually transient (even though it's the only one not marked to be as such).
https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideCreatingNewProducts#Restrictions_on_a_package_defini
There is much more freedom for transient products definitions and developments.

@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 11, 2019

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/889/console Started: 2019/06/11 23:48

@cmsbuild
Copy link
Contributor

@cmsbuild
Copy link
Contributor

Comparison job queued.

@cmsbuild
Copy link
Contributor

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b389fa/889/summary.html

Comparison Summary:

  • No significant changes to the logs found
  • Reco comparison results: 4 differences found in the comparisons
  • DQMHistoTests: Total files compared: 33
  • DQMHistoTests: Total histograms compared: 3215458
  • DQMHistoTests: Total failures: 2
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3215122
  • DQMHistoTests: Total skipped: 334
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 32 files compared)
  • Checked 137 log files, 14 edm output root files, 33 DQM output files

@cmsbuild
Copy link
Contributor

@cmsbuild
Copy link
Contributor

Comparison job queued.

@slava77
Copy link
Contributor

slava77 commented Jun 26, 2019

+1

for #27160 fba76c7

  • code changes are in line with the PR description and the follow up review
  • jenkins tests pass and comparisons with the baseline show no differences (as seen in a slightly earlier test of somewhat the same code; the TICL code is not running in production anyways).

@kpedro88 please check this PR at your earliest so that we can then proceed with #27194

Copy link
Contributor

@kpedro88 kpedro88 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These few minor comments can be addressed in a followup PR.

ticl::constants::nPhiBins,
layerClusters,
mask,
2,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could these magic numbers be named constants?

std::copy(std::begin(effective_cluster_idx), std::end(effective_cluster_idx),
std::back_inserter(tmp.vertices));
tmp.vertex_multiplicity.resize(effective_cluster_idx.size(), 0);
std::copy(std::begin(effective_cluster_idx), std::end(effective_cluster_idx), std::back_inserter(tmp.vertices));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this would be simpler (and maybe better performing) as:

tmp.vertices.insert(tmp.vertices.end(),effective_cluster_idx.begin(),effective_cluster_idx.end());

sort(idx.begin(), idx.end(),
[&v](size_t i1, size_t i2) {return v.clusters()[i1]->z() < v.clusters()[i2]->z();});
std::iota(std::begin(idx), std::end(idx), 0);
sort(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

std::sort

@kpedro88
Copy link
Contributor

+upgrade

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

@cmsbuild
Copy link
Contributor

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b389fa/1165/summary.html

Comparison Summary:

  • No significant changes to the logs found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 33
  • DQMHistoTests: Total histograms compared: 3254598
  • DQMHistoTests: Total failures: 1
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3254263
  • DQMHistoTests: Total skipped: 334
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 32 files compared)
  • Checked 137 log files, 14 edm output root files, 33 DQM output files

@slava77
Copy link
Contributor

slava77 commented Jun 26, 2019

@fabiocos
now that this is fully signed,
perhaps you can check this some time today or soon after.
Thank you.

@fabiocos
Copy link
Contributor

@makortel there is an overlap with you #27194, but this is not creating conflicts

@fabiocos
Copy link
Contributor

@slava77 @kpedro88 I understand that you are satisfied enough with the present status of the code, to integrate it closing a long-lasting saga, and further improvements may come later. I will integrate this PR in next IB.

@slava77
Copy link
Contributor

slava77 commented Jun 26, 2019 via email

@fabiocos
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit c1ac6be into cms-sw:master Jun 26, 2019
@slava77 slava77 mentioned this pull request Jun 28, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants