[DD4hep] DD Filtered View: Add Debug Print, RegExp Speed Up and Clean Up #29208

ianna · 2020-03-15T17:41:36Z

PR description:

Remove the need to an obscure mergeSpecifics() method because it does not merge anything, but filters the specifics registry. Note, the method is still accessible, but will be cleaned up in a later PR.
Instead of mergeSpecifics(), now DDFilteredView can be constructed from a DDCompactView and a DDFilter:

  const cms::DDFilter filter("OnlyForHcalSimNumbering", "HCAL");
  cms::DDFilteredView fv(cpv, filter);

Use regexp as keys in a Filter to speed it up a bit
~~Add a print filter method to allow easier debugging~~
Remove methods that are not needed any more
Use Hcal parameters builder as a user story study

@bsunanda - you may want to remove the BenchmarkGrd and disable the MessageLogger for the Geometry.

It's slightly slower to merge the filter registry:

< Benchmark 'DTGeometryESProducer Filter Registry' took 0.057576 millis
---
> Benchmark 'DTGeometryESProducer Filter Registry' took 0.052404 millis

but, it's ~8% faster to build the DT geometry:

< Benchmark 'DTGeometryESProducer' took 8082.02 millis
---
> Benchmark 'DTGeometryESProducer' took 8713.86 millis

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

Before submitting your pull requests, make sure you followed this checklist:

verify that the PR is really intended for the chosen branch
verify that changes follow CMS Naming, Coding, And Style Rules
verify that the PR passes the basic test procedure suggested in the CMSSW PR instructions

cmsbuild · 2020-03-15T17:42:02Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-03-15T17:49:28Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-29208/14197

This PR adds an extra 24KB to repository

cmsbuild · 2020-03-15T17:49:51Z

A new Pull Request was created by @ianna (Ianna Osborne) for master.

It involves the following packages:

DetectorDescription/DDCMS
Geometry/DTGeometryBuilder

@civanch, @Dr15Jones, @makortel, @cvuosalo, @ianna, @mdhildreth, @cmsbuild can you please review it and eventually sign? Thanks.
@vargasa this is something you requested to watch as well.
@davidlange6, @silviodonato, @fabiocos you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2020-03-16T15:17:08Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-03-16T15:23:20Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-29208/14209

This PR adds an extra 40KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-29208/14209/code-format.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-29208/14209/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

ianna

@bsunanda - time retrieving material names has been reduced from 70996.3 millis to 49157.4 millis

SpecPar names OnlyForHcalSimNumbering, HCAL:
hcal
Benchmark 'HcalSimParametersFromDD HCal materials OnlyForHcalSimNumbering, HCAL' took 70996.3 mill-is

due to the change in the selection from //.*HEScintillator.* to //HEScintillator.*:

hcal
   //HBS.*
   //HTS.*
   //HES.*
   //HVQF
   //HCal
   //HBS.*
   //HTS.*
   //.*HEScintillator.*
   //HVQX
   //VcalElecCathode
   //FibreBundle.*
   //VcalFibreBundleL.*
   //VcalFibreBundleR.*
   //CALO
   //MBAT
   //MBBT.*
   //VCAL

hcal
   //HBS.*
   //HTS.*
   //HES.*
   //HVQF
   //HCal
   //HBS.*
   //HTS.*
   //HEScintillator.*
   //HVQX
   //VcalElecCathode
   //FibreBundle.*
   //VcalFibreBundleL.*
   //VcalFibreBundleR.*
   //CALO
   //MBAT
   //MBBT.*
   //VCAL
Benchmark 'HcalSimParametersFromDD HCal materials OnlyForHcalSimNumbering, HCAL' took 48402.1 millis

The paths seem to be fine to use this type of selection:

world_volume/OCMS_1/CMSE_1/OQUA_1/VcalElectronics_1/VcalEleBox_1/VcalElecRBX_1/VcalElecPMT_1/VcalElecCathode_1
    world_volume/OCMS_1/CMSE_1/OQUA_1/FibreContainer_1/FibreBoxTrap_1/FibreBoxL_1/FibreBundle12_2
    world_volume/OCMS_1/CMSE_1/OQUA_1/VcalFibreBundleContainer_1/VcalFibreBundleSector_1/VcalFibreBundleHalfSectorL_1/VcalFibreBundleL11_2

    world_volume/OCMS_1/CMSE_1/VCAL_5002/HVQF_1/HVQX_18

Here is the summary before the changes:

TimeModule> 1 1 hpa HcalParametersAnalyzer 183.178
TimeModule> 1 1 p1 PathStatusInserter 0.00127196
TimeModule> 1 1 TriggerResults TriggerResultInserter 3.60012e-05
TimeEvent> 1 1 183.183
MemoryReport> Peak virtual size 536.102 Mbytes
 Key events increasing vsize: 
[0] run: 0 lumi: 0 event: 0  vsize = 0 deltaVsize = 0 rss = 0 delta = 0
[0] run: 0 lumi: 0 event: 0  vsize = 0 deltaVsize = 0 rss = 0 delta = 0
[1] run: 1 lumi: 1 event: 1  vsize = 536.102 deltaVsize = 0 rss = 96.9062 delta = 0
[0] run: 0 lumi: 0 event: 0  vsize = 0 deltaVsize = 0 rss = 0 delta = 0
[0] run: 0 lumi: 0 event: 0  vsize = 0 deltaVsize = 0 rss = 0 delta = 0
[0] run: 0 lumi: 0 event: 0  vsize = 0 deltaVsize = 0 rss = 0 delta = 0
[0] run: 0 lumi: 0 event: 0  vsize = 0 deltaVsize = 0 rss = 0 delta = 0
[1] run: 1 lumi: 1 event: 1  vsize = 536.102 deltaVsize = 0 rss = 96.9062 delta = 0
TimeReport> Time report complete in 186.166 seconds
 Time Summary: 
 - Min event:   183.183
 - Max event:   183.183
 - Avg event:   183.183
 - Total loop:  183.186
 - Total init:  2.9769
 - Total job:   186.166
 - EventSetup Lock:   6.19888e-06
 - EventSetup Get:   191.965
 Event Throughput: 0.00545894 ev/s
 CPU Summary: 
 - Total loop:  183.018
 - Total init:  1.43986
 - Total extra: 0
 - Total job:   184.461
 Processing Summary: 
 - Number of Events:  1
 - Number of Global Begin Lumi Calls:  1
 - Number of Global Begin Run Calls: 1

and after:

TimeModule> 1 1 hpa HcalParametersAnalyzer 140.614
TimeModule> 1 1 p1 PathStatusInserter 0.00015378
TimeModule> 1 1 TriggerResults TriggerResultInserter 2.40803e-05
TimeEvent> 1 1 140.614
MemoryReport> Peak virtual size 542.105 Mbytes
 Key events increasing vsize: 
[0] run: 0 lumi: 0 event: 0  vsize = 0 deltaVsize = 0 rss = 0 delta = 0
[0] run: 0 lumi: 0 event: 0  vsize = 0 deltaVsize = 0 rss = 0 delta = 0
[1] run: 1 lumi: 1 event: 1  vsize = 542.105 deltaVsize = 0 rss = 137.961 delta = 0
[0] run: 0 lumi: 0 event: 0  vsize = 0 deltaVsize = 0 rss = 0 delta = 0
[0] run: 0 lumi: 0 event: 0  vsize = 0 deltaVsize = 0 rss = 0 delta = 0
[0] run: 0 lumi: 0 event: 0  vsize = 0 deltaVsize = 0 rss = 0 delta = 0
[0] run: 0 lumi: 0 event: 0  vsize = 0 deltaVsize = 0 rss = 0 delta = 0
[1] run: 1 lumi: 1 event: 1  vsize = 542.105 deltaVsize = 0 rss = 137.961 delta = 0
TimeReport> Time report complete in 143.36 seconds
 Time Summary: 
 - Min event:   140.614
 - Max event:   140.614
 - Avg event:   140.614
 - Total loop:  140.616
 - Total init:  2.74258
 - Total job:   143.36
 - EventSetup Lock:   1.38283e-05
 - EventSetup Get:   149.858
 Event Throughput: 0.00711159 ev/s
 CPU Summary: 
 - Total loop:  140.488
 - Total init:  1.4459
 - Total extra: 0
 - Total job:   141.936
 Processing Summary: 
 - Number of Events:  1
 - Number of Global Begin Lumi Calls:  1
 - Number of Global Begin Run Calls: 1

DetectorDescription/DDCMS/interface/Filter.h

cmsbuild · 2020-03-16T15:48:39Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-03-16T15:54:13Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-29208/14210

This PR adds an extra 36KB to repository

cmsbuild · 2020-03-17T11:13:07Z

The code-checks are being triggered in jenkins.

ianna · 2020-03-17T11:16:56Z

please test

cmsbuild · 2020-03-17T11:18:21Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-29208/14227

This PR adds an extra 96KB to repository

cmsbuild · 2020-03-17T11:18:47Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/5218/console Started: 2020/03/17 12:23

cmsbuild · 2020-03-17T11:18:49Z

Pull request #29208 was updated. @civanch, @Dr15Jones, @makortel, @cvuosalo, @ianna, @mdhildreth, @cmsbuild can you please check and sign again.

cmsbuild · 2020-03-17T12:38:34Z

+1
Tested at: 0bb2c0c
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-36cfbe/5218/summary.html
CMSSW: CMSSW_11_1_X_2020-03-16-2300
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-03-17T12:38:37Z

Comparison job queued.

cmsbuild · 2020-03-17T14:09:25Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-36cfbe/5218/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2680593
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2680273
DQMHistoTests: Total skipped: 319
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
Checked 147 log files, 16 edm output root files, 34 DQM output files

civanch · 2020-03-18T08:37:02Z

+1

Carl, I think now PR is fine.

cmsbuild · 2020-03-18T08:37:23Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @silviodonato, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

silviodonato · 2020-03-18T09:00:53Z

+1

ianna added 2 commits March 15, 2020 18:29

Add more debug info and speed it up a bit

ca74d9d

code format

ed97a4f

cmsbuild added this to the CMSSW_11_1_X milestone Mar 15, 2020

cmsbuild added code-checks-pending comparison-pending geometry-pending orp-pending pending-signatures tests-pending labels Mar 15, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 15, 2020

restructure to match an Hcal user story

ab4972e

cmsbuild added code-checks-pending simulation-pending and removed code-checks-approved labels Mar 16, 2020

cmsbuild added code-checks-rejected and removed code-checks-pending labels Mar 16, 2020

ianna commented Mar 16, 2020

View reviewed changes

DetectorDescription/DDCMS/interface/Filter.h Outdated Show resolved Hide resolved

clean up and code format

8058951

cmsbuild added code-checks-pending and removed code-checks-rejected labels Mar 16, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 16, 2020

cmsbuild added the tests-pending label Mar 17, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 17, 2020

cmsbuild added tests-started and removed tests-pending labels Mar 17, 2020

cmsbuild added tests-approved and removed tests-started labels Mar 17, 2020

cmsbuild added comparison-available and removed comparison-pending labels Mar 17, 2020

cmsbuild added fully-signed geometry-approved simulation-approved and removed geometry-pending pending-signatures simulation-pending labels Mar 18, 2020

cmsbuild added orp-approved and removed orp-pending labels Mar 18, 2020

cmsbuild merged commit cab9f3b into cms-sw:master Mar 18, 2020

cmsbuild mentioned this pull request Mar 18, 2020

Clean residual dependency of SimG4Core/Forward on DetectorDescription #29223

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DD4hep] DD Filtered View: Add Debug Print, RegExp Speed Up and Clean Up #29208

[DD4hep] DD Filtered View: Add Debug Print, RegExp Speed Up and Clean Up #29208

ianna commented Mar 15, 2020 •

edited

cmsbuild commented Mar 15, 2020

cmsbuild commented Mar 15, 2020

cmsbuild commented Mar 15, 2020

cmsbuild commented Mar 16, 2020

cmsbuild commented Mar 16, 2020

ianna left a comment •

edited

cmsbuild commented Mar 16, 2020

cmsbuild commented Mar 16, 2020

cmsbuild commented Mar 17, 2020

ianna commented Mar 17, 2020

cmsbuild commented Mar 17, 2020

cmsbuild commented Mar 17, 2020 •

edited

cmsbuild commented Mar 17, 2020

cmsbuild commented Mar 17, 2020

cmsbuild commented Mar 17, 2020

cmsbuild commented Mar 17, 2020

civanch commented Mar 18, 2020

cmsbuild commented Mar 18, 2020

silviodonato commented Mar 18, 2020

[DD4hep] DD Filtered View: Add Debug Print, RegExp Speed Up and Clean Up #29208

[DD4hep] DD Filtered View: Add Debug Print, RegExp Speed Up and Clean Up #29208

Conversation

ianna commented Mar 15, 2020 • edited

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented Mar 15, 2020

cmsbuild commented Mar 15, 2020

cmsbuild commented Mar 15, 2020

cmsbuild commented Mar 16, 2020

cmsbuild commented Mar 16, 2020

ianna left a comment • edited

Choose a reason for hiding this comment

cmsbuild commented Mar 16, 2020

cmsbuild commented Mar 16, 2020

cmsbuild commented Mar 17, 2020

ianna commented Mar 17, 2020

cmsbuild commented Mar 17, 2020

cmsbuild commented Mar 17, 2020 • edited

cmsbuild commented Mar 17, 2020

cmsbuild commented Mar 17, 2020

cmsbuild commented Mar 17, 2020

cmsbuild commented Mar 17, 2020

civanch commented Mar 18, 2020

cmsbuild commented Mar 18, 2020

silviodonato commented Mar 18, 2020

ianna commented Mar 15, 2020 •

edited

ianna left a comment •

edited

cmsbuild commented Mar 17, 2020 •

edited