Online HCAL DQM adding a new application #13805

vkhristenko · 2016-03-22T21:21:58Z

This is a test PR based on #13565
to test the addition of a new application to the Online HCAL DQM Workflow

VK

…lities

…nd hcal2_source....

cmsbuild · 2016-03-22T21:23:47Z

A new Pull Request was created by @vkhristenko (Viktor Khristenko) for CMSSW_8_1_X.

It involves the following packages:

DQM/HcalCommon
DQM/HcalTasks
DQM/Integration
DQMOffline/Configuration

@cmsbuild, @vanbesien, @deguio, @davidlange6 can you please review it and eventually sign? Thanks.
@threus, @batinkov, @rociovilar this is something you requested to watch as well.
@slava77, @Degano, @smuzaffar you are the release manager for this.

cms-bot commands are list here #13028

deguio · 2016-03-23T10:30:55Z

please test

cmsbuild · 2016-03-23T10:31:16Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/12015/console

deguio · 2016-03-23T10:31:44Z

please test

deguio · 2016-03-23T10:37:07Z

please submit to 80x as well

cmsbuild · 2016-03-23T11:39:44Z

Pull request #13805 was updated. @cmsbuild, @vanbesien, @deguio, @davidlange6 can you please check and sign again.

cmsbuild · 2016-03-23T12:54:23Z

-1
Tested at: 91913d5
When I ran the RelVals I found an error in the following worklfows:
134.911 step1

DAS Error

1000.0 step1

DAS Error

you can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-13805/12015/summary.html

deguio · 2016-03-23T12:58:58Z

please test

cmsbuild · 2016-03-23T12:59:19Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/12022/console

deguio · 2016-04-26T14:32:13Z

please test

deguio · 2016-04-26T14:32:16Z

+1

cmsbuild · 2016-04-26T14:32:43Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/12648/console

cmsbuild · 2016-04-26T14:32:44Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_8_1_X IBs after it passes the integration tests. This pull request requires discussion in the ORP meeting before it's merged. @slava77, @davidlange6, @Degano, @smuzaffar

cmsbuild · 2016-04-26T18:24:12Z

+1
Tested at: 2efc201
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-13805/12648/summary.html

cmsbuild · 2016-04-26T19:29:19Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-13805/12648/summary.html

deguio · 2016-05-04T10:53:30Z

@davidlange6
could we have this one merged please?
thanks,
F.

@dmitrijus @vanbesien

abdoulline · 2016-05-09T08:58:07Z

I second Federico's request to David @davidlange6
Then we could hopefully proceed with (back-ported to 80X) #13813

davidlange6 · 2016-05-11T13:23:22Z

+1

Dr15Jones · 2016-05-12T14:10:55Z

It looks like this pull request is causing the crashes in the IB Release Validation Jobs: https://cms-sw.github.io/relvalLogDetail.html#slc6_amd64_gcc493;CMSSW_8_1_X_2016-05-11-2300

The traceback from the crash is

#5  0x00007f93a367bf7d in hcaldqm::RawRunSummary::endJob(DQMStore::IBooker&, DQMStore::IGetter&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_1_X_2016-05-11-2300/lib/slc6_amd64_gcc493/libDQMHcalTasks.so
#6  0x00007f93a36f6aff in HcalOfflineHarvesting::_dqmEndJob(DQMStore::IBooker&, DQMStore::IGetter&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_1_X_2016-05-11-2300/lib/slc6_amd64_gcc493/pluginDQMHcalTasksAuto.so
#7  0x00007f93f97ac0d7 in DQMEDHarvester::endJob() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_1_X_2016-05-11-2300/lib/slc6_amd64_gcc493/libDQMServicesCore.so
#8  0x00007f9400904df8 in edm::Worker::endJob() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_1_X_2016-05-11-2300/lib/slc6_amd64_gcc493/libFWCoreFramework.so
#9  0x00007f9400906790 in edm::WorkerManager::endJob(edm::ExceptionCollector&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_1_X_2016-05-11-2300/lib/slc6_amd64_gcc493/libFWCoreFramework.so
#10 0x00007f94008bf755 in edm::Schedule::endJob(edm::ExceptionCollector&) () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_1_X_2016-05-11-2300/lib/slc6_amd64_gcc493/libFWCoreFramework.so
#11 0x00007f9400836f91 in edm::EventProcessor::endJob() () from /cvmfs/cms-ib.cern.ch/week0/slc6_amd64_gcc493/cms/cmssw/CMSSW_8_1_X_2016-05-11-2300/lib/slc6_amd64_gcc493/libFWCoreFramework.so
#12 0x000000000040ceea in main::{lambda()#1}::operator()() const ()
#13 0x000000000040b406 in main ()

dmitrijus · 2016-05-12T14:20:43Z

@vkhristenko can you please take care of this?

vkhristenko · 2016-05-12T18:56:59Z

@deguio , @dmitrijus , @Dr15Jones

There are 2 separate issues in here.

Workflows 134.7** and 134.7** Already fixed. pushed. 1 statement should've been included...
Workflow 4.77
This workflow contains Harvesting over 2 runs - you can see that from logs of step3 for instance where the DQMEDAnalyzers are run for step3, but harvesting will get applied to that guy in a similar manner. Even though I misread https://twiki.cern.ch/twiki/bin/view/CMSPublic/FWMultithreadedFrameworkOneModuleInterface slightly.... it was guaranteed that HARVESTING step is done only over 1 run always for the following time period [Construction of Harvestor, Destruction of Harvestor]. I'm not sure how that is imposed... Because from the description of edm::one you can have (if you have WatchRuns) harvesting done over many runs just not in parallel...

Since this was a guaranty, all of my Harvestors are designed to have harvesting being done only over 1 Run from Construction of the object until the destruction! I've pushed an update which would be a temporary solution and it does pass 4.77 - basically just clear vectors per each beginRun transition...

my apology

VK

dmitrijus · 2016-05-13T10:17:00Z

I saw the fix for 8_0_X - but create another PR for 8_1_X, on the IB containing this.
I will accept it - and hopefully make IBs valid again.

deguio · 2016-05-13T13:06:49Z

let me try to clarify:

in prod at the moment there are no WFs that run the harvesting step on multiple runs (neither in relvals nor in prompt)
in the future it would be nice to have this possibility supported (multiRun harvesting) so I would propose to extend the use case to cover multi run processing as well
the skims are a special case due to the way in which they are processed in the RECO-DQM step:
- each file could contain multiple runs
- in prod at the harvesting N separate jobs are potentially run on the same file where N is the number of runs contained in that file
- the LS from a given run are selected at the source where a filter is configured by WMAgent
- even if only single runs are processed in each harvesting job, endRun - beginRun transitions could be triggered anyway during the processing.

In summary a design which assumes that the beginRun transition is called only once per harvesting job conclusion it is not safe.

I include few experts who might want to comment/correct me if I am wrong.
@cerminar @franzoni @fabozzi

Dr15Jones · 2016-05-13T13:09:52Z

From the framework perspective, modules should always be able to handle multiple Runs or LuminosityBlocks.

vkhristenko added 8 commits March 2, 2016 21:39

preping for PR

09ef20b

adding, modifying Offline Harvester for HCAL

2c6139a

use const refs for templated complex objects in Utilities.h

90b6d0b

adding calibration sequence harvestor and some cleaning of unused uti…

3740dbd

…lities

moving harvestors into tasks folder

07ac025

fixing imports

9922e66

adding a new application for online DQM. now we have hcal_source... a…

b2d9378

…nd hcal2_source....

fixing applications

91913d5

cmsbuild added this to the Next CMSSW_8_1_X milestone Mar 22, 2016

cmsbuild added dqm-pending pending-signatures tests-pending orp-pending comparison-pending labels Mar 22, 2016

cmsbuild added tests-started and removed tests-pending labels Mar 23, 2016

deguio mentioned this pull request Mar 23, 2016

HCAL DQM Updates in Preparation for 2016 Collisions #13565

Closed

fixing file naming conventions

bb6b83c

cmsbuild added tests-pending and removed tests-started labels Mar 23, 2016

cmsbuild removed the tests-pending label Mar 23, 2016

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Apr 26, 2016

cmsbuild added tests-started and removed tests-pending labels Apr 26, 2016

cmsbuild added tests-approved and removed tests-started labels Apr 26, 2016

cmsbuild added comparison-available and removed comparison-pending labels Apr 26, 2016

cmsbuild added orp-approved and removed orp-pending labels May 11, 2016

cmsbuild merged commit d89f2c1 into cms-sw:CMSSW_8_1_X May 11, 2016

dmitrijus mentioned this pull request May 12, 2016

Online HCAL DQM adding a 3rd application #13813

Merged

dmitrijus mentioned this pull request May 13, 2016

HCAL DQM Updates. On top of 13805 #14491

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Online HCAL DQM adding a new application #13805

Online HCAL DQM adding a new application #13805

vkhristenko commented Mar 22, 2016

cmsbuild commented Mar 22, 2016

deguio commented Mar 23, 2016

cmsbuild commented Mar 23, 2016

deguio commented Mar 23, 2016

deguio commented Mar 23, 2016

cmsbuild commented Mar 23, 2016

cmsbuild commented Mar 23, 2016

deguio commented Mar 23, 2016

cmsbuild commented Mar 23, 2016

deguio commented Apr 26, 2016

deguio commented Apr 26, 2016

cmsbuild commented Apr 26, 2016 •

edited

cmsbuild commented Apr 26, 2016

cmsbuild commented Apr 26, 2016

cmsbuild commented Apr 26, 2016

deguio commented May 4, 2016

abdoulline commented May 9, 2016 •

edited

davidlange6 commented May 11, 2016

Dr15Jones commented May 12, 2016

dmitrijus commented May 12, 2016

vkhristenko commented May 12, 2016 •

edited

dmitrijus commented May 13, 2016

deguio commented May 13, 2016

Dr15Jones commented May 13, 2016

Online HCAL DQM adding a new application #13805

Online HCAL DQM adding a new application #13805

Conversation

vkhristenko commented Mar 22, 2016

cmsbuild commented Mar 22, 2016

deguio commented Mar 23, 2016

cmsbuild commented Mar 23, 2016

deguio commented Mar 23, 2016

deguio commented Mar 23, 2016

cmsbuild commented Mar 23, 2016

cmsbuild commented Mar 23, 2016

deguio commented Mar 23, 2016

cmsbuild commented Mar 23, 2016

deguio commented Apr 26, 2016

deguio commented Apr 26, 2016

cmsbuild commented Apr 26, 2016 • edited

cmsbuild commented Apr 26, 2016

cmsbuild commented Apr 26, 2016

cmsbuild commented Apr 26, 2016

deguio commented May 4, 2016

abdoulline commented May 9, 2016 • edited

davidlange6 commented May 11, 2016

Dr15Jones commented May 12, 2016

dmitrijus commented May 12, 2016

vkhristenko commented May 12, 2016 • edited

dmitrijus commented May 13, 2016

deguio commented May 13, 2016

Dr15Jones commented May 13, 2016

cmsbuild commented Apr 26, 2016 •

edited

abdoulline commented May 9, 2016 •

edited

vkhristenko commented May 12, 2016 •

edited