Adding HGCAL simhit validation histograms

[indra-ehep]

PR description:

• Few validation histograms will be added in the HGCal SimHitValidation folder of DQM root file
• It does not depend upon other PR
• This is following the presentation at https://indico.cern.ch/event/1085122/contributions/4563005/attachments/2327395/3965157/2021-10-13_HGCal_in_CMSSW.pdf
  and plan discussed in HGCAL GEANT activity at https://indico.cern.ch/event/1099648/contributions/4627667/attachments/2354197/4016902/2021-11-26_Plan_for_Documentation_and_PR.pdf

PR validation:

All the suggested tests are perfomed as mentioned in https://cms-sw.github.io/PRWorkflow.html including 'scram build code-checks' and 'scram build code-format'

if this PR is a backport please specify the original PR and why you need to backport that PR:

Not applicable.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36484/27356

• This PR adds an extra 20KB to repository

[cmsbuild]

A new Pull Request was created by @indra-ehep for master.

It involves the following packages:

• Validation/HGCalValidation (dqm)

@emanueleusai, @ahmad3213, @cmsbuild, @jfernan2, @pmandrik, @pbo0, @rvenditti can you please review it and eventually sign? Thanks.
@vandreev11, @sethzenz, @bsunanda, @rovere, @lgray, @cseez, @apsallid, @pfs, @lecriste, @hatakeyamak, @ebrondol this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[jfernan2]

@indra-ehep can you please add yourself along with your github username in the comments filed to the HGCAL Validation developers e-group to keep track of the people working on it?
https://twiki.cern.ch/twiki/bin/viewauth/CMS/DQMContacts#HGCAL
Thanks

[indra-ehep]

@indra-ehep can you please add yourself along with your github username in the comments filed to the HGCAL Validation developers e-group to keep track of the people working on it? https://twiki.cern.ch/twiki/bin/viewauth/CMS/DQMContacts#HGCAL Thanks

Dear Javier,
Thank you for prompt response. I have added myself to the list that you have mentioned. Now waiting for approval.
regards
Indra

[jfernan2]

please test

[indra-ehep]

please test

Thank you. I see that I am a member now.

[cmsbuild]

-1

Failed Tests: Build HeaderConsistency ClangBuild
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-44816f/21251/summary.html
COMMIT: 67b311f
CMSSW: CMSSW_12_3_X_2021-12-13-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/36484/21251/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @alja Transition from sigcpp-2 to sigcpp-3 #36343
• @dinyar Use correct accessors for RegionalMuonShower data type #36455
• @iarspider Remove test of h5py_cache #36447
• @francescobrivio Apply Naming Rules to BeamSpotObjects methods #36419
• @tvami Modernize / DB output migration of GBRWrapperMaker.cc #36366
• @missirol ensure HLTProcess adds all (End)Paths to the schedule #36422

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-44816f/21251/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-44816f/21251/git-merge-result

Build

I found compilation error when building:

/cvmfs/cms-ib.cern.ch/nweek-02711/slc7_amd64_gcc900/external/sigcpp/2.6.2-cms/include/sigc++-2.0/sigc++/functors/slot.h:1655:7:   required from 'class sigc::slot'
/cvmfs/cms-ib.cern.ch/nweek-02711/slc7_amd64_gcc900/external/sigcpp/2.6.2-cms/include/sigc++-2.0/sigc++/signal.h:658:41:   required from 'struct sigc::internal::signal_emit0'
/cvmfs/cms-ib.cern.ch/nweek-02711/slc7_amd64_gcc900/external/sigcpp/2.6.2-cms/include/sigc++-2.0/sigc++/signal.h:2635:54:   required from 'class sigc::signal0'
/cvmfs/cms-ib.cern.ch/nweek-02711/slc7_amd64_gcc900/external/sigcpp/2.6.2-cms/include/sigc++-2.0/sigc++/signal.h:3839:7:   required from 'class sigc::signal'
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_3_X_2021-12-13-1100/src/Fireworks/Core/interface/FWColorManager.h:101:32:   required from here
/cvmfs/cms-ib.cern.ch/nweek-02711/slc7_amd64_gcc900/external/sigcpp/2.6.2-cms/include/sigc++-2.0/sigc++/functors/slot.h:431:22: error: function returning a function
  431 |   typedef T_return (*call_type)(rep_type*);
      |                      ^~~~~~~~~
/cvmfs/cms-ib.cern.ch/nweek-02711/slc7_amd64_gcc900/external/sigcpp/2.6.2-cms/include/sigc++-2.0/sigc++/functors/slot.h:437:19: error: function returning a function
  437 |   inline T_return operator()() const
      |                   ^~~~~~~~


Clang Build

I found compilation error while trying to compile with clang. Command used:

USER_CUDA_FLAGS='--expt-relaxed-constexpr' USER_CXXFLAGS='-Wno-register -fsyntax-only' scram build -k -j 32 COMPILER='llvm compile'

>> Entering Package RecoEgamma/EgammaTools
>> Entering Package RecoVertex/BeamSpotProducer
>> Entering Package SimTransport/PPSProtonTransport
>> Entering Package Validation/HGCalValidation
>> Compile sequence completed for CMSSW CMSSW_12_3_X_2021-12-13-1100
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 1
+ eval scram build outputlog '&&' '(python3' /data/cmsbld/jenkins/workspace/ib-run-pr-tests/cms-bot/buildLogAnalyzer.py --ignoreWarning=Wdeprecated-declarations --logDir /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_3_X_2021-12-13-1100/tmp/slc7_amd64_gcc900/cache/log/src '||' 'true)'
++ scram build outputlog
>> Entering Package Alignment/OfflineValidation
>> Compiling  /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_12_3_X_2021-12-13-1100/src/Alignment/OfflineValidation/bin/DMRtrends.cc
Entering library rule at src/Alignment/OfflineValidation/plugins


[perrotta]

please test
(CMSSW_12_3_X_2021-12-13-2300, now available, contains the needed external)

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-44816f/21259/summary.html
COMMIT: 67b311f
CMSSW: CMSSW_12_3_X_2021-12-13-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/36484/21259/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 42
• DQMHistoTests: Total histograms compared: 3250704
• DQMHistoTests: Total failures: 0
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3250682
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 2203098.4 KiB( 41 files compared)
• DQMHistoSizes: changed ( 23234.0,... ): 445730.602 KiB HGCAL/HGCalSimHitsV
• DQMHistoSizes: changed ( 38634.0 ): 420175.992 KiB HGCAL/HGCalSimHitsV
• Checked 177 log files, 37 edm output root files, 42 DQM output files
• TriggerResults: no differences found

[jfernan2]

I am sorry @indra-ehep but your PR is adding more than 400MB of new MEs to the DQM root file.
You should reduce the memory footprint, I suggest reducing the number of MEs and/or the binning (1k bins in 1D and 600x600 in 2D plots is too much)
Please inspect the results and reduce accordingly:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_3_X_2021-12-13-2300+44816f/47470/dqm-histo-comparison-summary.html
Thanks

[indra-ehep]

I am sorry @indra-ehep but your PR is adding more than 400MB of new MEs to the DQM root file. You should reduce the memory footprint, I suggest reducing the number of MEs and/or the binning (1k bins in 1D and 600x600 in 2D plots is too much) Please inspect the results and reduce accordingly: https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_3_X_2021-12-13-2300+44816f/47470/dqm-histo-comparison-summary.html Thanks

Dear Javier,

The main issue of large memory footprint is due to TH2. Earlier we have used TGraph instead of TH2 which contains the same information but with a significantly smaller size. However, I did not find any handler for TGraph in DQMStore.h. May be it is mentioned in other files that I missed. So how do we proceed now ?

Plan A : I can switch off the TH2 for now, re-optimize binning of 1D histograms. Then commit the changes. Later we add the TGraph features when available in DQMStore.h after contacting DQM experts.

Plan B : I add TGraph feature in DQMStore.h and then modify HGCalSimHitValidation.cc for TGraph and 1D histograms.

Or if you have other plan. Please let me know your feedback.

regards
Indra

[indra-ehep]

I am sorry @indra-ehep but your PR is adding more than 400MB of new MEs to the DQM root file. You should reduce the memory footprint, I suggest reducing the number of MEs and/or the binning (1k bins in 1D and 600x600 in 2D plots is too much) Please inspect the results and reduce accordingly: https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_3_X_2021-12-13-2300+44816f/47470/dqm-histo-comparison-summary.html Thanks

Dear Javier,
Sorry for multiple posts. Is there any tool in CMSSW to check how much extra storage requirement is added in DQM.root file due to the new changes ? Then I can run such tests locally before making the git push.
regards
Indra

[indra-ehep]

@jfernan2 I have pushed the file that has been agreed. Thanks.

[cmsbuild]

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36484/27446

• This PR adds an extra 20KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

• code-checks:
  https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36484/27446/code-checks.patch
  e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36484/27446/code-checks.patch | patch -p1
  You can also run scram build code-checks to apply code checks directly

• code-format:
  https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36484/27446/code-format.patch
  e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36484/27446/code-format.patch | patch -p1
  You can also run scram build code-format to apply code format directly

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36484/27454

• This PR adds an extra 20KB to repository

[cmsbuild]

Pull request #36484 was updated. @emanueleusai, @ahmad3213, @cmsbuild, @jfernan2, @pmandrik, @pbo0, @rvenditti can you please check and sign again.

[jfernan2]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-44816f/21365/summary.html
COMMIT: 5993f50
CMSSW: CMSSW_12_3_X_2021-12-17-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/36484/21365/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 42
• DQMHistoTests: Total histograms compared: 3250719
• DQMHistoTests: Total failures: 6
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3250691
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 33718.237 KiB( 41 files compared)
• DQMHistoSizes: changed ( 23234.0,... ): 6818.627 KiB HGCAL/HGCalSimHitsV
• DQMHistoSizes: changed ( 38634.0 ): 6443.729 KiB HGCAL/HGCalSimHitsV
• Checked 177 log files, 37 edm output root files, 42 DQM output files
• TriggerResults: no differences found

[jfernan2]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[perrotta]

+1

• The size increase of the DQM root files, all below 7 MB, is deemed acceptable by DQM