Slimming Strip Calibration Trees output #23045

mmusich · 2018-04-24T08:46:47Z

Greetings,
this PR packages several updates to the CalibTracker code in order to optimize the storage disk consumption of the Strip Calibration ntuples on eos.
Main features added:

possibilty to prescale events entering any of the shallow tree producers;
added compression of TFileService for the ShallowTree class;
moved where unnecessary double precision in favor of float;
unused variables masked behind a preprocessor flag, ExtendedCALIBTree;
chargeoverpath (single largest offender) variable masked, in favor of building it via ratio of existing charge and path ;
updated unit tests

Changes have been proposed and revised by @mdelcourt,@clacaputo,@echabert and @jlagram.

Testing this branch with a O(1k) events from a 2017C ALCARECO file, we get a net reduction of ~ 28% in size.
Going tree by tree:

`gainCalibrationTree` tree:

`anEff` tree:

`EventInfo` tree:

As the code touched here, is used also for the SiStripGains PCL algorithm a dedicated test with O(100k) events has been carried out by using the following commands:

cmsDriver.py step3 --datatier ALCARECO --conditions auto:run2_data -s ALCA:PromptCalibProdSiStripGains --eventcontent ALCARECO -n -1 --dasquery='file dataset=/ZeroBias/Run2016C-SiStripCalMinBias-18Apr2017-v1/ALCARECO run=276097'

followed by:

cmsDriver.py stepMultiHarvest --data --conditions auto:run2_data --scenario pp -s ALCAHARVEST:SiStripGains --filein file:PromptCalibProdSiStripGains.root -n -1 --fileout file:calib.root --customise_command "process.DQMStore.collateHistograms = cms.untracked.bool(True)\nprocess.dqmSaver.saveByRun=cms.untracked.int32(-1)\n process.dqmSaver.saveAtJobEnd=cms.untracked.bool(True)\nprocess.dqmSaver.forceRunNumber=cms.untracked.int32(999999)" --no_exec

to emulate the Multi-Run Harvesting.
No difference is found in the output (the complete histograms comparison is available here)
Just to show two examples:

…rom the config gile

modified: CalibTracker/SiStripCommon/plugins/ShallowTracksProducer.cc - double to float modified: CalibTracker/SiStripCommon/plugins/ShallowTracksProducer.cc - std::bitset introduced - mods encapsulated in a preprocessore flag, CALIBTreeDEV Note: std::bitset type still not accepted from ShallowTree

modified: CalibTracker/SiStripCommon/plugins/ShallowEventDataProducer.cc modified: CalibTracker/SiStripCommon/plugins/ShallowGainCalibration.cc modified: CalibTracker/SiStripCommon/plugins/ShallowTracksProducer.cc modified: CalibTracker/SiStripHitEfficiency/interface/HitEff.h modified: CalibTracker/SiStripHitEfficiency/src/HitEff.cc - Unused variables masked behind a preprocessor flag, CALIBTreeDEV - chargeoverpath varable masked modified: CalibTracker/SiStripChannelGain/src/SiStripGainsPCLWorker.cc - chargeoverpath dependencies removed runTheMatrix.py -l 1001.0 successfully passed

- Preprocessor flag changed to ExtendedCALIBTree - Minor indentation mods

…on algo concept in the Hit Efficiency tree

cmsbuild · 2018-04-24T08:47:07Z

The code-checks are being triggered in jenkins.

cmsbuild · 2018-04-24T08:50:13Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-23045/4443

Code check has found code style and quality issues which could be resolved by applying a patch in https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-23045/4443/git-diff.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-23045/4443/git-diff.patch | patch -p1

You can run scram build code-checks to apply code checks directly

cmsbuild · 2018-04-24T08:58:26Z

The code-checks are being triggered in jenkins.

cmsbuild · 2018-04-24T09:00:03Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-23045/4445

cmsbuild · 2018-04-24T09:00:23Z

A new Pull Request was created by @mmusich (Marco Musich) for master.

It involves the following packages:

CalibTracker/Configuration
CalibTracker/SiStripChannelGain
CalibTracker/SiStripCommon
CalibTracker/SiStripHitEfficiency

@cmsbuild, @franzoni, @arunhep, @cerminar, @lpernie can you please review it and eventually sign? Thanks.
@echabert, @gbenelli, @tocheng, @mverzett, @OlivierBondu, @mmusich this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

lpernie · 2018-04-24T14:05:19Z

please test

cmsbuild · 2018-04-24T14:05:37Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/27634/console Started: 2018/04/24 16:11

cmsbuild · 2018-04-24T16:32:25Z

+1
Tested at: e8d44a1
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-23045/27634/summary.html

cmsbuild · 2018-04-24T16:32:27Z

Comparison job queued.

cmsbuild · 2018-04-24T17:32:19Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-23045/27634/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 29
DQMHistoTests: Total histograms compared: 2492830
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2492653
DQMHistoTests: Total skipped: 176
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 28 files compared)
Checked 119 log files, 9 edm output root files, 29 DQM output files

mmusich · 2018-04-26T12:36:51Z

@arunhep @lpernie
When looking for changes in the Comparison Summary of this PR, I realized that the only relevant workflow comparison for the changes proposed here (i.e. 1001.0) actually got empty relmon:

https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_10_2_X_2018-04-23-2300+23045/26271/1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5+ALCAHARVDSIPIXELCALRUN1/

The log file claims there is a division by 0:

https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_10_2_X_2018-04-23-2300+23045/26271/1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5+ALCAHARVDSIPIXELCALRUN1RelMonComp-1001.0.log

Traceback (most recent call last):
  File "/cvmfs/cms-ib.cern.ch/nweek-02521/slc6_amd64_gcc630/cms/cmssw/CMSSW_10_2_X_2018-04-22-1100/bin/slc6_amd64_gcc630/compare_using_files.py", line 345, in <module>
    directory2html(directory, options.hash_name, options.standalone)
  File "/cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_2_X_2018-04-23-2300/python/Utilities/RelMon/directories2html.py", line 464, in directory2html
    page_html+=get_rank_section(directory)
  File "/cvmfs/cms-ib.cern.ch/week1/slc6_amd64_gcc630/cms/cmssw-patch/CMSSW_10_2_X_2018-04-23-2300/python/Utilities/RelMon/directories2html.py", line 390, in get_rank_section
    scale = gPad.GetUymax()/rightmax
ZeroDivisionError: float division by zero

N.B. This feature is common to any other recent PR and it is not due to the changes proposed here.
Upon manual inspection I realized that for some reason, when the last harvesting step of 1001.0 is run (i.e. ALCAHARVDSIPIXELCALRUN1) the harvested DQM files becomes empty.
This makes the comparison with the baseline useless in PRs as this one.
I took the liberty to "fix" this in #23063.

lpernie · 2018-04-26T22:12:45Z

Very nice

lpernie · 2018-04-26T22:12:48Z

+1

cmsbuild · 2018-04-26T22:13:05Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

fabiocos · 2018-05-02T09:32:46Z

+1

Martin Delcourt and others added 17 commits April 20, 2018 16:04

Adding prescale

2a07c50

Added run number filter

c0a2f4d

Updated to dasgoclient

be3b646

Adding voms proxy file to run on data outside of CERN

8f64b60

Change user id of the voms proxy key

20b3f06

Updated debug mode

91c91e4

Removed temporary test values and updated for 2018 dataset - castor dir

65d8dd4

update configuration of the test

f48fb36

Updated "testTree" name

d24b9a8

add compression of TFileService for the ShallowTree class - control f…

1eb36d5

…rom the config gile

changes move upstream in ntuple_cff - Add documentation

cc466ff

Changes requested implemented

0314eb5

- Preprocessor flag changed to ExtendedCALIBTree - Minor indentation mods

restore mistakenly removed vars

7ddd1d4

restore the local error variables as well and introduce the compressi…

0a9b195

…on algo concept in the Hit Efficiency tree

cleanup

8652858

cmsbuild added this to the CMSSW_10_2_X milestone Apr 24, 2018

cmsbuild added alca-pending code-checks-pending comparison-pending orp-pending pending-signatures tests-pending labels Apr 24, 2018

cmsbuild added code-checks-rejected code-checks-pending and removed code-checks-pending code-checks-rejected labels Apr 24, 2018

cmsbuild added code-checks-pending and removed code-checks-rejected labels Apr 24, 2018

cmsbuild added code-checks-approved and removed code-checks-pending labels Apr 24, 2018

cmsbuild added tests-started and removed tests-pending labels Apr 24, 2018

cmsbuild added tests-approved and removed tests-started labels Apr 24, 2018

cmsbuild added comparison-available and removed comparison-pending labels Apr 24, 2018

cmsbuild added alca-approved fully-signed and removed alca-pending pending-signatures labels Apr 26, 2018

cmsbuild added orp-approved and removed orp-pending labels May 2, 2018

cmsbuild merged commit 7cba396 into cms-sw:master May 2, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Slimming Strip Calibration Trees output #23045

Slimming Strip Calibration Trees output #23045

mmusich commented Apr 24, 2018

cmsbuild commented Apr 24, 2018

cmsbuild commented Apr 24, 2018

cmsbuild commented Apr 24, 2018

cmsbuild commented Apr 24, 2018

cmsbuild commented Apr 24, 2018

lpernie commented Apr 24, 2018

cmsbuild commented Apr 24, 2018 •

edited

cmsbuild commented Apr 24, 2018

cmsbuild commented Apr 24, 2018

cmsbuild commented Apr 24, 2018

mmusich commented Apr 26, 2018

lpernie commented Apr 26, 2018

lpernie commented Apr 26, 2018

cmsbuild commented Apr 26, 2018

fabiocos commented May 2, 2018

Slimming Strip Calibration Trees output #23045

Slimming Strip Calibration Trees output #23045

Conversation

mmusich commented Apr 24, 2018

gainCalibrationTree tree:

anEff tree:

EventInfo tree:

cmsbuild commented Apr 24, 2018

cmsbuild commented Apr 24, 2018

cmsbuild commented Apr 24, 2018

cmsbuild commented Apr 24, 2018

cmsbuild commented Apr 24, 2018

lpernie commented Apr 24, 2018

cmsbuild commented Apr 24, 2018 • edited

cmsbuild commented Apr 24, 2018

cmsbuild commented Apr 24, 2018

cmsbuild commented Apr 24, 2018

mmusich commented Apr 26, 2018

lpernie commented Apr 26, 2018

lpernie commented Apr 26, 2018

cmsbuild commented Apr 26, 2018

fabiocos commented May 2, 2018

`gainCalibrationTree` tree:

`anEff` tree:

`EventInfo` tree:

cmsbuild commented Apr 24, 2018 •

edited