Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Major DQM update needed for the online #4369

Merged
merged 14 commits into from Jun 26, 2014
Merged

Major DQM update needed for the online #4369

merged 14 commits into from Jun 26, 2014

Conversation

dmitrijus
Copy link
Contributor

This update touches:

  • DQMFileSaver (now is Global::EDAnalyzer)
  • DQMStore (supports proper loading of protocol buffer files)
  • DQMProtobufSource (new)
  • DQMStreamerSource (minor updates)

Salvatore Di Guida and others added 5 commits June 19, 2014 14:35
Modify DQMFileSaver to support new online use-cases.
b27d6a3 
Fixed wrong include.
Removed unused dependency from EventFiter package in DQMServices/StreamerIO/plugins BuidFile.
Use ak5 jets.
Make DQMFileSaver create also json files.
Use FilterUnit in tests.
Temporary fix: reset MEs if stored in PB format.
Temporary workaround: run step1 with PB on.
Provide json and data filenames with one single function.
save all the needed fields in the json file.
Remove unneeded pointers to MEs.
Include EvF services in DQMFileSaver.
Use the new location of the EvF services.
Remove not needed include in DQMFileSaver.cc
Use EvF services to get the path for files in Filter Unit.
Avoid the usage of string literals in the stream naming convention.
Use only one function for saving files in Filter Unit: the format is determined by enum passed as function parameter. This removes code duplication.
Fix definition field in the json file. CAVEAT: the slashes are escaped with backslash.
Fix source field in the json file.
@deguio
Modify DQMStore to support new online use-cases.
4218b9d 
Squashes onto 7_1_X:

new option for DQMStore to make all the MEs LS-based
add forceResetOnBeginLumi option to DQMStore
remove transitions which are not triggered anymore by the threaded framework
make DQMFileSaver create also json files.
reset lumi object instead of deleting it. needed for the online when saving per LS
simplify the logic of the indexes reset when running with enableMultiThread
correctly set flags default
bound forceResetOnBeginLumi and enableMultiThread together
percolate the enableMultiThread flag to the DQMFileSaver, MEtoEDMConverter and DQMRootOutputModule classes
force reset at the end of the LS for all the histograms when in FilterUnit mode and new tester now available
enabling the fuctionalities of the DQMFileSaver one by one
added offline use cases
remove useless ME booked in DQMFileSaver
Implements an input source for PB files.
4ef1f48 
Online DQM needs to load the Protobuf files produced by the FU.
This source also creates the required structure (run/lumi transitions)
to support the harvesting step.

Also this modifed the fastHadd to support primitive monitor elements.
However, unlike TH1 derived MEs, such elements are just copied around,
but are never merged.
@borrello
Adding trigger selection for the DQMStreamerReader.
4dce54b
Various minor bug fixes to DQMStore and DQMFileSaver.
7f4f8f2 
Also add a few test cases to test the protobuf chain.
@cmsbuild
Copy link
Contributor

A new Pull Request was created by @dmitrijus (Dmitrijus) for CMSSW_7_1_X.

Major DQM update needed for the online

It involves the following packages:

DQMServices/Components
DQMServices/Core
DQMServices/Examples
DQMServices/FwkIO
DQMServices/StreamerIO

@ojeda, @danduggan, @rovere, @cmsbuild, @nclopezo, @deguio, @Degano can you please review it and eventually sign? Thanks.
@barvic this is something you requested to watch as well.
You can sign-off by replying to this message having '+1' in the first line of your reply.
You can reject by replying to this message having '-1' in the first line of your reply.
@nclopezo, @ktf you are the release manager for this.
You can merge this pull request by typing 'merge' in the first line of your comment.

@cmsbuild
Copy link
Contributor

Pull request #4369 was updated. @ojeda, @danduggan, @rovere, @cmsbuild, @nclopezo, @deguio, @Degano can you please check and sign again.

@deguio
Copy link
Contributor

deguio commented Jun 25, 2014

hello @nclopezo
is it possible to have the tests run on this as soon as possible?
both the normal tests and the comparison against a recent IB are needed.
thanks,
F.

@cmsbuild
Copy link
Contributor

Pull request #4369 was updated. @ojeda, @danduggan, @rovere, @cmsbuild, @nclopezo, @deguio, @Degano can you please check and sign again.

@cmsbuild
Copy link
Contributor

Pull request #4369 was updated. @ojeda, @danduggan, @rovere, @cmsbuild, @nclopezo, @deguio, @Degano can you please check and sign again.

@Martin-Grunewald
Copy link
Contributor

Presumably this must also go into 72X.

@cmsbuild
Copy link
Contributor

Pull request #4369 was updated. @ojeda, @danduggan, @rovere, @cmsbuild, @nclopezo, @deguio, @Degano can you please check and sign again.

Fix a compilation error in the previous commit and make DQMProtobufRe…
3826165 
…ader to generate an empty lumi transitions instead of skipping them.
@cmsbuild
Copy link
Contributor

Pull request #4369 was updated. @ojeda, @danduggan, @rovere, @cmsbuild, @nclopezo, @deguio, @Degano can you please check and sign again.

@deguio
Copy link
Contributor

deguio commented Jun 25, 2014

+1
approving before it gets lost. needed for the GR
still would like to see the outcome of the tests..

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next CMSSW_7_1_X IBs unless changes or unless it breaks tests. @nclopezo, @ktf can you please take care of it?

@cmsbuild
Copy link
Contributor

+1
Tested at: 3826165
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4369/1753/summary.html

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next CMSSW_7_1_X IBs unless changes (tests are also fine). @nclopezo, @ktf can you please take care of it?

@cmsbuild
Copy link
Contributor

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4369/1753/summary.html

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next CMSSW_7_1_X IBs unless changes (tests are also fine). @nclopezo, @ktf can you please take care of it?

davidlange6 added a commit that referenced this pull request Jun 26, 2014
@davidlange6
Merge pull request #4369 from dmitrijus/OnlineDQMMergeAttempt2
ccad0ef 
Major DQM update needed for the online
@davidlange6 davidlange6 merged commit ccad0ef into cms-sw:CMSSW_7_1_X Jun 26, 2014
@ktf ktf mentioned this pull request Jun 26, 2014
Merged
@deguio deguio mentioned this pull request Jun 27, 2014
Merged
@dmitrijus dmitrijus mentioned this pull request Jun 30, 2014
Merged
ktf added a commit to ktf/cmssw that referenced this pull request Jul 3, 2014
@ktf
Ignore cms-sw#4369 into 72X
2eae858
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
comparison-available dqm-approved fully-signed orp-pending tests-approved
Projects
None yet
Milestone
CMSSW_7_1_X
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@dmitrijus @cmsbuild @deguio @Martin-Grunewald @davidlange6 @borrello