HCAL DQM Update #12651

vkhristenko · 2015-12-02T20:54:24Z

This is the update for HCAL DQM in preparation for 2016.

Packages such as:
DQM/HcalCommon
DQM/HcalTasks
are new. 2 new configuration files, with exactly the same naming conventions as before:
DQM/Integration/python/clients/hcal_dqm_sourceclient-live_cfg.py
DQM/Integration/python/clients/hcalcalib_dqm_sourceclient-live_cfg.py

These new config files use only newly added packages.
Also, the DQMOffline/Configuration/python/DQMOffline_cff.py has been updated to stop using old packages for Offline (I mean modules coming from DQM/ only, none of modules from DQMOffline/ were touched).

runTheMatrix passed successfully.

To note, none of the packages sitting in DQM/, which are:
HcalMonitorTasks
HcalMonitorModule
HcalMonitorClient
will not be running any longer(at least that is the goal/plan!), with a single exception of 2 things:
DQM/HcalMonitorClient/python/HcalDataCertification_cfi.py
DQM/HcalMonitorClient/python/HcalDAQInfo_cfi.py
being used in Offline. That will be replaced at some point before the actual datataking

Submitting only for 80X as this is intended only for 2016.
VK

cmsbuild · 2015-12-02T20:54:49Z

A new Pull Request was created by @vkhristenko (Viktor Khristenko) for CMSSW_8_0_X.

It involves the following packages:

DQM/HcalCommon
DQM/HcalTasks
DQM/Integration
DQMOffline/Configuration

@cmsbuild, @vanbesien, @deguio, @davidlange6 can you please review it and eventually sign? Thanks.
@threus, @batinkov, @rociovilar this is something you requested to watch as well.
@slava77, @Degano, @smuzaffar you are the release manager for this.

Following commands in first line of a comment are recognized

+1|approve[d]|sign[ed]: L1/L2's to approve it
-1|reject[ed]: L1/L2's to reject it
assign <category>[,<category>[,...]]: L1/L2's to request signatures from other categories
unassign <category>[,<category>[,...]]: L1/L2's to remove signatures from other categories
hold: L1/all L2's/release manager to mark it as on hold
unhold: L1/user who put this PR on hold
merge: L1/release managers to merge this request
[@cmsbuild,] please test: L1/L2 and selected users to start jenkins tests
[@cmsbuild,] please test with cms-sw/cmsdist#<PR>: L1/L2 and selected users to start jenkins tests using externals from cmsdist

vkhristenko · 2015-12-02T21:01:44Z

@abdoulline

vanbesien · 2015-12-03T08:41:31Z

please test

cmsbuild · 2015-12-03T08:41:52Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/10106/console

cmsbuild · 2015-12-03T10:21:59Z

-1
Tested at: 9e346e8
When I ran the RelVals I found an error in the following worklfows:
9.0 step3

runTheMatrix-results/9.0_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST/step3_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST.log

25.0 step3

runTheMatrix-results/25.0_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT/step3_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT.log

1306.0 step3

runTheMatrix-results/1306.0_SingleMuPt1_UP15+SingleMuPt1_UP15+DIGIUP15+RECOUP15+HARVESTUP15/step3_SingleMuPt1_UP15+SingleMuPt1_UP15+DIGIUP15+RECOUP15+HARVESTUP15.log

1330.0 step3

runTheMatrix-results/1330.0_ZMM_13+ZMM_13+DIGIUP15+RECOUP15+HARVESTUP15/step3_ZMM_13+ZMM_13+DIGIUP15+RECOUP15+HARVESTUP15.log

50202.0 step3

runTheMatrix-results/50202.0_TTbar_13+TTbar_13+DIGIUP15_PU50+RECOUP15_PU50+HARVESTUP15_PU50/step3_TTbar_13+TTbar_13+DIGIUP15_PU50+RECOUP15_PU50+HARVESTUP15_PU50.log

25202.0 step3

runTheMatrix-results/25202.0_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25/step3_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25.log

you can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12651/10106/summary.html

vkhristenko · 2015-12-03T12:00:32Z

Checked the main log:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12651/10106/runTheMatrix-results/matrixTests.log

Found problems here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12651/10106/runTheMatrix-results/25.0_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT/step3_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT.log
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12651/10106/runTheMatrix-results/9.0_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST/step3_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST.log
and other places.

I ran the following runTheMatrix sequence:
runTheMatrix.py -l 4.53 -j 0

I see now that DQMOfflineMC imports modules from DQM/HcalMonitor*

VK

cmsbuild · 2015-12-03T22:43:40Z

Pull request #12651 was updated. @cmsbuild, @vanbesien, @deguio, @davidlange6 can you please check and sign again.

vkhristenko · 2015-12-03T22:44:53Z

I've updated the imports in the rest of the DQMOffline... files
Only parts that import Online (DQM/HcalMonitor...) modules are replaced. None from DQMOffline/Hcal has been touched.

Tested against 25.0 that was causing issues

VK

vanbesien · 2015-12-04T10:14:43Z

please test

cmsbuild · 2015-12-04T10:15:07Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/10136/console

cmsbuild · 2016-02-15T09:43:12Z

Pull request #12651 was updated. @cmsbuild, @vanbesien, @deguio, @davidlange6 can you please check and sign again.

deguio · 2016-02-15T09:43:35Z

please test

cmsbuild · 2016-02-15T09:43:55Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/11212/console

deguio · 2016-02-15T10:54:23Z

+1
still waiting for the tests, but should be ok.

cmsbuild · 2016-02-15T10:54:48Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_8_0_X IBs after it passes the integration tests. This pull request requires discussion in the ORP meeting before it's merged. @slava77, @davidlange6, @Degano, @smuzaffar

cmsbuild · 2016-02-15T11:07:20Z

+1
Tested at: 3a46253
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12651/11211/summary.html

cmsbuild · 2016-02-15T11:25:51Z

+1
Tested at: 0e412ce
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12651/11212/summary.html

cmsbuild · 2016-02-15T12:15:44Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12651/11211/summary.html

cmsbuild · 2016-02-15T12:25:04Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12651/11212/summary.html

davidlange6 · 2016-02-15T17:23:18Z

Finally catching up on this full PR. On the technical aspects of this code base I have a few comments below.

Most important (for this PR), we need to remove the try-catch blocks - at least in the offline workflow.
Second (needed for 8_0_1), there are a lot of detector properties defined in Constants.h. These are mostly (all?) defined elsewhere and should simply be taken from there. Eg, fed ids for hcal, sub detector enums, etc. Similarly, the validDetId function must already be implemented in hcal code - use it.
In the utilites functions, please pass the complex objects as const refs. I happened to notice that the 'ped' argument to maxTS has no influence on the outcome of the function.
- Longer term, much more of this should be templated to remove duplicated code - I think that would make a large part of this completely independent of HCAL and thus could be moved up stream.

davidlange6 · 2016-02-15T17:23:39Z

@vkhristenko @deguio - sorry should have cc-ed you on the previous message in case you miss it.

vkhristenko · 2016-02-15T17:30:58Z

@davidlange6 ,

try - catch - ok - possible
if there was any documentation - I would have done that - there is none. Previously all of that would be just hardcoded... Fed ids I know where to find and will update that, but the rest is not documented anywhere, at least there is no documentation that hcaldqm has....
Sounds good
Reagrding the templating - there is a reason why I didn't do that. I have had problems compiling the code when I use templating, would run out of memory - although that was done on some older machines..... but point taken!

David, what would be the timeline to fix (try-catch for instance) - a couple of days? say by thursday - or too late???

davidlange6 · 2016-02-15T17:44:03Z

On Feb 15, 2016, at 6:31 PM, Viktor Khristenko notifications@github.com wrote:

@davidlange6 ,

try - catch - ok - possible

if there was any documentation - I would have done that - there is none. Previously all of that would be just hardcoded... Fed ids I know where to find and will update that, but the rest is not documented anywhere, at least there is no documentation that hcaldqm has….

I guess the HCAL group can help with more expertise..but here are some examples.

https://github.com/cms-sw/cmssw/blob/CMSSW_8_0_X/DataFormats/FEDRawData/interface/FEDNumbering.h
https://github.com/cms-sw/cmssw/blob/CMSSW_8_0_X/DataFormats/HcalDetId/interface/HcalSubdetector.h

Sounds good

Reagrding the templating - there is a reason why I didn't do that. I have had problems compiling the code when I use templating, would run out of memory - although that was done on some older machines..... but point taken!

David, what would be the timeline to fix (try-catch for instance) - a couple of days? say by thursday - or too late???

thursday is good - I can merge this in if thats a doable timescale - then we can make a patch.

—
Reply to this email directly or view it on GitHub.

vkhristenko · 2016-02-15T17:47:08Z

LOL - your 2 refs are the ones that I have used already lol... ok sounds good... I understood...
Yes, then by thursday morning I will provide an update with resolving try-catch issue in there!

VK

HCAL DQM Update

vkhristenko added 4 commits December 2, 2015 21:05

Preparing for a PR

7204c16

fixing Offline Sequence

20d41bd

deleting unused

dafaa64

proper comments

9e346e8

cmsbuild added this to the Next CMSSW_8_0_X milestone Dec 2, 2015

cmsbuild added dqm-pending pending-signatures tests-pending orp-pending comparison-pending labels Dec 2, 2015

cmsbuild added tests-started and removed tests-pending labels Dec 3, 2015

cmsbuild added tests-rejected and removed tests-started labels Dec 3, 2015

vkhristenko added 2 commits December 3, 2015 23:02

fixing problems observed during the Test of the PR

72e5260

deleting

1c96873

cmsbuild added tests-pending and removed tests-rejected labels Dec 3, 2015

cmsbuild added tests-started and removed tests-pending labels Dec 4, 2015

removing tpTask

0e412ce

cmsbuild added tests-pending and removed tests-started labels Feb 15, 2016

cmsbuild added tests-started and removed tests-pending labels Feb 15, 2016

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Feb 15, 2016

cmsbuild added tests-approved and removed tests-started labels Feb 15, 2016

cmsbuild added comparison-available and removed comparison-pending labels Feb 15, 2016

davidlange6 added a commit that referenced this pull request Feb 16, 2016

Merge pull request #12651 from vkhristenko/CMSSW_8_0_X_HCALDQM

e73481b

HCAL DQM Update

davidlange6 merged commit e73481b into cms-sw:CMSSW_8_0_X Feb 16, 2016

diguida mentioned this pull request Feb 16, 2016

Further cleanup of Condition access in AlCaDB, online DQM, Geometry, HLT, Simulation, and standard configuration; Further fixes for online DQM #13170

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HCAL DQM Update #12651

HCAL DQM Update #12651

vkhristenko commented Dec 2, 2015

cmsbuild commented Dec 2, 2015

vkhristenko commented Dec 2, 2015

vanbesien commented Dec 3, 2015

cmsbuild commented Dec 3, 2015

cmsbuild commented Dec 3, 2015

vkhristenko commented Dec 3, 2015

cmsbuild commented Dec 3, 2015

vkhristenko commented Dec 3, 2015

vanbesien commented Dec 4, 2015

cmsbuild commented Dec 4, 2015

cmsbuild commented Feb 15, 2016

deguio commented Feb 15, 2016

cmsbuild commented Feb 15, 2016

deguio commented Feb 15, 2016

cmsbuild commented Feb 15, 2016

cmsbuild commented Feb 15, 2016

cmsbuild commented Feb 15, 2016

cmsbuild commented Feb 15, 2016

cmsbuild commented Feb 15, 2016

davidlange6 commented Feb 15, 2016

davidlange6 commented Feb 15, 2016

vkhristenko commented Feb 15, 2016

davidlange6 commented Feb 15, 2016

vkhristenko commented Feb 15, 2016

HCAL DQM Update #12651

HCAL DQM Update #12651

Conversation

vkhristenko commented Dec 2, 2015

cmsbuild commented Dec 2, 2015

vkhristenko commented Dec 2, 2015

vanbesien commented Dec 3, 2015

cmsbuild commented Dec 3, 2015

cmsbuild commented Dec 3, 2015

vkhristenko commented Dec 3, 2015

cmsbuild commented Dec 3, 2015

vkhristenko commented Dec 3, 2015

vanbesien commented Dec 4, 2015

cmsbuild commented Dec 4, 2015

cmsbuild commented Feb 15, 2016

deguio commented Feb 15, 2016

cmsbuild commented Feb 15, 2016

deguio commented Feb 15, 2016

cmsbuild commented Feb 15, 2016

cmsbuild commented Feb 15, 2016

cmsbuild commented Feb 15, 2016

cmsbuild commented Feb 15, 2016

cmsbuild commented Feb 15, 2016

davidlange6 commented Feb 15, 2016

davidlange6 commented Feb 15, 2016

vkhristenko commented Feb 15, 2016

davidlange6 commented Feb 15, 2016

vkhristenko commented Feb 15, 2016