Extend thinning to support slimming #31100

makortel · 2020-08-08T01:07:52Z

PR description:

This PR extends the framework's thinning feature to support slimming as outlined in #30544 (comment):

If there are both thinned and thinned+slimmed collections present, a de-reference of Ref-to-parent goes always only to the thinned collections
- Even if none of the thinned collections contain the requested element (in which case nullptr is returned) and the thinned+slimmed collection would contain the requested element
- I.e. Ref de-reference means
  - if parent collection is available, return element in it
  - else if any thinned collection (branch) exists
    - if a thinned collection containing the element is available, return the element
    - else return nullptr
  - else if a slimmed collection (branch) exists, take the first slimmed collection searching downward through the parentage tree of thinned/slimmed collections, and apply this algorithm recursively with the slimmed collection as the parent collection
  - else return nullptr
The parentage tree of thinned and thinned+slimmed collections may have at most one path from the root (parent) collection to any leaf collection that has slimmed collections visible in that job
- i.e. one can do e.g. parent -> slimmed1 -> thinned2 -> slimmed3, but not any of
```
parent --> slimmed11
       \-> slimmed21

parent --> thinned11 -> slimmed12
       \-> thinned21 -> slimmed22

parent --> thinned11 -> slimmed12
       \-> slimmed21 -> thinned22
```
- Notably having slimmed1X and slimmed2Y modules in separate jobs is allowed, assuming the other slimmed products are not visible there.
  - Trying to read both slimmed1X and slimmed2Y from different files (via primary and secondary inputs) results an error

In addition, this PR

adds an implementation of thinnedRefFrom() (suggested in [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544 (comment)) in a way that does not lead to asking asking products from ProductResolvers ([RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544 (comment))
adds support for thinning edmNew::DetSetVector (needed for Fix the MadGraph bug for multithreading (backport #30444) #30445)
adds ThinnedRefSet to help with Selector to keep track of keys to input collection that may come from any of the parent collections in the parentage tree of thinned collections
adds reset() function to be required on the thinning Selector classes. It will be called after the object loop to allow clearing any intermediate data structures.

PR validation:

Framework unit tests pass.

cmsbuild · 2020-08-08T01:08:17Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-08-08T01:16:01Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31100/17650

This PR adds an extra 472KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Common/interface/DetSetVectorNew.h modified in PR(s): remove ifdefs in DataFormats/Common #30885
- File DataFormats/Common/interface/EDProductGetter.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/Common/interface/RefItemGet.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/Common/test/DetSetVector_t.cpp modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/Common/test/SimpleEDProductGetter.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/Common/test/ptr_t.cppunit.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/Common/test/ptrvector_t.cppunit.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/Common/test/ref_t.cppunit.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/Common/test/reftobaseprod_t.cppunit.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/FWLite/interface/ChainEvent.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/FWLite/interface/DataGetterHelper.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/FWLite/interface/Event.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/FWLite/interface/MultiChainEvent.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/FWLite/src/ChainEvent.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/FWLite/src/DataGetterHelper.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/FWLite/src/Event.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/FWLite/src/MultiChainEvent.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/Provenance/interface/ThinnedAssociationsHelper.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/Provenance/src/ThinnedAssociationsHelper.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File FWCore/FWLite/src/BareRootProductGetter.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File FWCore/FWLite/src/BareRootProductGetter.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File FWCore/Framework/interface/EventPrincipal.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File FWCore/Framework/interface/Principal.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File FWCore/Framework/interface/stream/ThinningProducer.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File FWCore/Framework/src/EventPrincipal.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File FWCore/Framework/src/Principal.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File FWCore/Framework/src/Worker.cc modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553, Use TransitionInfo in more interfaces #30969
- File FWCore/Framework/src/Worker.h modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553, Use TransitionInfo in more interfaces #30969
- File FWCore/Integration/test/ThinningThingProducer.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File IOPool/Streamer/interface/StreamerInputSource.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File IOPool/Streamer/src/StreamerInputSource.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-checks:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31100/17650/code-checks.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31100/17650/code-checks.patch | patch -p1
You can also run scram build code-checks to apply code checks directly

…thinned collection

Add ThinnedAssociationsHelper::slimmed_

cmsbuild · 2020-08-08T01:18:02Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-08-08T01:26:01Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31100/17651

This PR adds an extra 472KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Common/interface/DetSetVectorNew.h modified in PR(s): remove ifdefs in DataFormats/Common #30885
- File DataFormats/Common/interface/EDProductGetter.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/Common/interface/RefItemGet.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/Common/test/DetSetVector_t.cpp modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/Common/test/SimpleEDProductGetter.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/Common/test/ptr_t.cppunit.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/Common/test/ptrvector_t.cppunit.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/Common/test/ref_t.cppunit.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/Common/test/reftobaseprod_t.cppunit.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/FWLite/interface/ChainEvent.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/FWLite/interface/DataGetterHelper.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/FWLite/interface/Event.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/FWLite/interface/MultiChainEvent.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/FWLite/src/ChainEvent.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/FWLite/src/DataGetterHelper.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/FWLite/src/Event.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/FWLite/src/MultiChainEvent.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/Provenance/interface/ThinnedAssociationsHelper.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File DataFormats/Provenance/src/ThinnedAssociationsHelper.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File FWCore/FWLite/src/BareRootProductGetter.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File FWCore/FWLite/src/BareRootProductGetter.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File FWCore/Framework/interface/EventPrincipal.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File FWCore/Framework/interface/Principal.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File FWCore/Framework/interface/stream/ThinningProducer.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File FWCore/Framework/src/EventPrincipal.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File FWCore/Framework/src/Principal.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File FWCore/Framework/src/Worker.cc modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553, Use TransitionInfo in more interfaces #30969
- File FWCore/Framework/src/Worker.h modified in PR(s): Delete unscheduled modules that are not consumed by any scheduled module #29553, Use TransitionInfo in more interfaces #30969
- File FWCore/Integration/test/ThinningThingProducer.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File IOPool/Streamer/interface/StreamerInputSource.h modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544
- File IOPool/Streamer/src/StreamerInputSource.cc modified in PR(s): [RFC] Store TrackExtra, TrackingRecHits, and SiPixel/StripClusters for muons in AOD+MINIAOD #30544

cmsbuild · 2020-08-08T01:26:24Z

A new Pull Request was created by @makortel (Matti Kortelainen) for master.

It involves the following packages:

DataFormats/Common
DataFormats/FWLite
DataFormats/Provenance
DataFormats/TestObjects
FWCore/FWLite
FWCore/Framework
FWCore/Integration
IOPool/Streamer

@makortel, @smuzaffar, @cmsbuild, @Dr15Jones can you please review it and eventually sign? Thanks.
@rovere, @wddgit this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

makortel · 2020-08-08T01:29:30Z

@cmsbuild, please test

makortel · 2020-08-08T01:29:45Z

@Dr15Jones @wddgit Please review

makortel · 2020-08-08T01:29:50Z

FYI @bendavid

cmsbuild · 2020-08-08T01:29:53Z

The tests are being triggered in jenkins.

CMSSW_11_2_X_2020-08-07-1100/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/8667/console Started: 2020/08/08 03:32

cmsbuild · 2020-09-01T18:42:55Z

The tests are being triggered in jenkins.

CMSSW_11_2_X_2020-09-01-1100/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/9041/console Started: 2020/09/01 20:44

cmsbuild · 2020-09-01T19:24:42Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-58f66a/9036/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 35
DQMHistoTests: Total histograms compared: 2609667
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2609644
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 34 files compared)
Checked 149 log files, 22 edm output root files, 35 DQM output files

cmsbuild · 2020-09-01T22:31:06Z

+1
Tested at: b1c8f44
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-58f66a/9041/summary.html
CMSSW: CMSSW_11_2_X_2020-09-01-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-09-01T22:31:08Z

Comparison job queued.

cmsbuild · 2020-09-02T00:34:55Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-58f66a/9041/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 35
DQMHistoTests: Total histograms compared: 2609667
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2609638
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 34 files compared)
Checked 149 log files, 22 edm output root files, 35 DQM output files

makortel · 2020-09-02T01:35:02Z

+1

I'll collect the remaining items into an issue.

cmsbuild · 2020-09-02T01:35:29Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

makortel · 2020-09-02T01:37:38Z

@slava77 @jpata @bendavid Just to make sure, following the discussion in ORP I'm not going to backport this PR (and #30906) to 10_6_X. If the plans change and a backport of these developments would be used in 10_6_X, I can look into a backport (technically should be straightforward).

makortel · 2020-09-02T01:50:27Z

I'll collect the remaining items into an issue.

Done in #31321.

silviodonato · 2020-09-02T08:31:48Z

+1

cmsbuild added this to the CMSSW_11_2_X milestone Aug 8, 2020

cmsbuild added code-checks-pending comparison-pending core-pending orp-pending pending-signatures tests-pending labels Aug 8, 2020

cmsbuild added code-checks-rejected and removed code-checks-pending labels Aug 8, 2020

makortel added 5 commits August 7, 2020 20:17

Add thinnedRefFrom() function to get a Ref from parent to a specific …

baeb812

…thinned collection

Add support for slimming thinned collections

9c73fa6

Add ThinnedAssociationsHelper::slimmed_

Add support for thinning edmNew::DetSetVector

16f9c54

Add ThinnedRefSet

5c40cd2

Add reset() to thinning selectors

ef39b71

makortel force-pushed the thinningSlimming branch from ec0d907 to ef39b71 Compare August 8, 2020 01:17

cmsbuild added code-checks-pending and removed code-checks-rejected labels Aug 8, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Aug 8, 2020

cmsbuild added tests-started and removed tests-pending labels Aug 8, 2020

cmsbuild added tests-started and removed tests-pending labels Sep 1, 2020

cmsbuild added comparison-available and removed comparison-pending labels Sep 1, 2020

cmsbuild added comparison-pending tests-approved and removed comparison-available tests-started labels Sep 1, 2020

cmsbuild added comparison-available and removed comparison-pending labels Sep 2, 2020

cmsbuild added core-approved fully-signed and removed core-pending pending-signatures labels Sep 2, 2020

makortel mentioned this pull request Sep 2, 2020

Possible improvements in thinning #31321

Open

cmsbuild added orp-approved and removed orp-pending labels Sep 2, 2020

cmsbuild merged commit d4d452e into cms-sw:master Sep 2, 2020

makortel deleted the thinningSlimming branch September 2, 2020 13:12

cmsbuild mentioned this pull request Sep 2, 2020

DD4hep: CmsTrackerLevelBuilder Fix Tracker Geometry Navigation #31337

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extend thinning to support slimming #31100

Extend thinning to support slimming #31100

makortel commented Aug 8, 2020 •

edited

cmsbuild commented Aug 8, 2020

cmsbuild commented Aug 8, 2020

cmsbuild commented Aug 8, 2020

cmsbuild commented Aug 8, 2020

cmsbuild commented Aug 8, 2020

makortel commented Aug 8, 2020

makortel commented Aug 8, 2020

makortel commented Aug 8, 2020

cmsbuild commented Aug 8, 2020 •

edited

cmsbuild commented Sep 1, 2020 •

edited

cmsbuild commented Sep 1, 2020

cmsbuild commented Sep 1, 2020

cmsbuild commented Sep 1, 2020

cmsbuild commented Sep 2, 2020

makortel commented Sep 2, 2020

cmsbuild commented Sep 2, 2020

makortel commented Sep 2, 2020

makortel commented Sep 2, 2020

silviodonato commented Sep 2, 2020

Extend thinning to support slimming #31100

Extend thinning to support slimming #31100

Conversation

makortel commented Aug 8, 2020 • edited

PR description:

PR validation:

cmsbuild commented Aug 8, 2020

cmsbuild commented Aug 8, 2020

cmsbuild commented Aug 8, 2020

cmsbuild commented Aug 8, 2020

cmsbuild commented Aug 8, 2020

makortel commented Aug 8, 2020

makortel commented Aug 8, 2020

makortel commented Aug 8, 2020

cmsbuild commented Aug 8, 2020 • edited

cmsbuild commented Sep 1, 2020 • edited

cmsbuild commented Sep 1, 2020

cmsbuild commented Sep 1, 2020

cmsbuild commented Sep 1, 2020

cmsbuild commented Sep 2, 2020

makortel commented Sep 2, 2020

cmsbuild commented Sep 2, 2020

makortel commented Sep 2, 2020

makortel commented Sep 2, 2020

silviodonato commented Sep 2, 2020

makortel commented Aug 8, 2020 •

edited

cmsbuild commented Aug 8, 2020 •

edited

cmsbuild commented Sep 1, 2020 •

edited