DQM: new DQMFileSaver #28588

schneiml · 2019-12-09T19:01:05Z

PR description:

In offline, the legacy TDirectory files for the DQMGUI are written by the DQMFileSaver module. While reading the code and its configurations, it became apparent that this module is not actually used in online or HLT (any more?), even though most of the code is dedicated to these cases. (Online and HLT use DQMServices/FileIO now.)

So, I stripped down this module to prove the hypothesis.

In addition, this PR contains the new TDirectory writing code by @andrius-k, as a new independent class. It is still linked into the DQMStore and can be called as edm::Service<DQMStore>()->save(...). I also updated the Online output modules to use this helper (DQMFileSaverOnline) or contain the writing code (DQMFileSaverPB). Consequently, DQMStore itself does not contain file writing code any more. There is still a huge amount of file reading code, which is largely useless and to be removed next.

PR validation:

This PR needs testing in online DQM (not done yet) and ideally also in HLT (or a solid confirmation that the module is never loaded in HLT). The changes to the ProtoBuf writer need to be tested in the HLT/fastHadd/online DQM chain. None of this is done so far. I'll try to provide a test setup in another PR.

This PR needs testing in Tier0 Express processing, where the runIsComplete logic is used.

The main point of the PR is to test

That only a single mode of this module is used in offline, to be confirmed by the IB tests; (as far as it seems confirmed)
That the new TDirectory writing code works; (confirmed)
Which unit tests depend on behavior removed in this PR. (none, it seems)

[I suspect some tests to fail (assuming these tests are run in the PR tests, not sure about that), and some flaws in the TDirectory writing code (it does not support QTests yet, for example).] Edit: all fixed.

Using commit 70fc23c from andrius-k:dqm-new-dqmstore-on-CMSSW_11_0_0_pre5. Includes changes to make the coe compile and run with the old DQMStore.

…dule.

It looks like this was never actually used in online DQM or HLT in run2. There are plenty of threading-related code paths, but DQMServices/FileIO seems to have taken over at some point.

cmsbuild · 2019-12-09T19:01:37Z

The code-checks are being triggered in jenkins.

schneiml · 2019-12-09T19:02:41Z

please test

I did remember to run code-format, this time!

cmsbuild · 2019-12-09T19:06:43Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28588/13085

This PR adds an extra 24KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DQMServices/Components/plugins/SealModule.cc modified in PR(s): DQM: Remove ClientConfig package #28380

cmsbuild · 2019-12-09T19:07:12Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/3885/console Started: 2019/12/09 20:08

cmsbuild · 2019-12-09T19:07:14Z

A new Pull Request was created by @schneiml (Marcel Schneider) for master.

It involves the following packages:

DQMServices/Components

@andrius-k, @kmaeshima, @schneiml, @cmsbuild, @jfernan2, @fioriNTU can you please review it and eventually sign? Thanks.
@barvic this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2019-12-09T20:52:55Z

+1
Tested at: eddafb9
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-071e70/3885/summary.html

cmsbuild · 2019-12-09T20:52:58Z

Comparison job queued.

cmsbuild · 2019-12-09T22:25:45Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-071e70/3885/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2800084
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2799742
DQMHistoTests: Total skipped: 341
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 119367.177 KiB( 33 files compared)
DQMHistoSizes: changed ( 1000.0,... ): 524.656 KiB Tracking/TrackParameters
DQMHistoSizes: changed ( 1000.0,... ): -81.534 KiB RPC/AllHits
DQMHistoSizes: changed ( 1000.0,... ): -78.862 KiB RPC/Muon
DQMHistoSizes: changed ( 1000.0,... ): 72.000 KiB Muons/globalMuons
DQMHistoSizes: changed ( 1000.0,... ): 54.984 KiB OfflinePV/Resolution
DQMHistoSizes: changed ( 1000.0,... ): 42.240 KiB Btag/TagCorrelation_pfCombinedCvsBJetTags_vs_pfCombinedCvsLJetTags_GLOBAL
DQMHistoSizes: changed ( 1000.0,... ): 42.240 KiB Btag/TagCorrelation_pfCombinedCvsBJetTags_vs_pfCombinedCvsLJetTags_ETA_1v4-2v4
DQMHistoSizes: changed ( 1000.0,... ): 42.240 KiB Btag/TagCorrelation_pfCombinedCvsBJetTags_vs_pfCombinedCvsLJetTags_ETA_0-1v4
DQMHistoSizes: changed ( 1000.0,... ): 27.686 KiB Egamma/gedPhotonAnalyzer
DQMHistoSizes: changed ( 1000.0,... ): 27.686 KiB Egamma/stdPhotonAnalyzer
DQMHistoSizes: changed ( 1000.0 ): ...
Checked 147 log files, 16 edm output root files, 34 DQM output files

schneiml · 2019-12-10T09:11:21Z

I am surprised that no tests failed, even though #28407 (comment) indicated that the old whiteRabbit DQM tests run in the PR tests.

The comparison looks much more messy than expected though, we'll need to fix a bunch of things it seems.

Edit: I see

Superfluous quotes on strings
Rounding differences with floats (not sure if we'll be able to resolve these)
The missing QTests (I assume these are the 1000's of missing items)
Edit²: actually the majority of missing items are efficiency flag markers.

I don't see what the DQMHistoSizes comparison is doing here though, I assume it is just confused by something. I don't see 100's of KB of new histograms in the bin-to-bin comparison.

cmsbuild · 2019-12-13T18:22:54Z

Pull request #28588 was updated. @andrius-k, @kmaeshima, @schneiml, @cmsbuild, @jfernan2, @fioriNTU can you please check and sign again.

Sam-Harper · 2019-12-16T09:06:50Z

@Martin-Grunewald , thanks, I have alerted FOG and we can see if anybody is around to do a quick hilton test

schneiml · 2019-12-16T09:35:37Z

@Sam-Harper if you get to it, can you provide a sample of the streamDQMHistograms output stream? This is what we need to be able to read on online DQM side, and while I can't reproduce this currently (the files seem to be missing a "streamer" header -- is that because I used the fakeFilterUnitMode or does some DAQ mechanism add that later?), looking at the files themselves would be useful...

fabiocos · 2019-12-16T17:12:54Z

please test

cmsbuild · 2019-12-16T17:13:26Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/3998/console Started: 2019/12/16 18:14

cmsbuild · 2019-12-16T19:50:19Z

+1
Tested at: b3259d5
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-071e70/3998/summary.html
CMSSW: CMSSW_11_1_X_2019-12-16-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2019-12-16T19:50:24Z

Comparison job queued.

cmsbuild · 2019-12-16T21:36:30Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-071e70/3998/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2798405
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2798063
DQMHistoTests: Total skipped: 341
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
Checked 147 log files, 16 edm output root files, 34 DQM output files

missirol · 2019-12-17T09:02:28Z

@Sam-Harper if you get to it, can you provide a sample of the streamDQMHistograms output stream? This is what we need to be able to read on online DQM side, and while I can't reproduce this currently (the files seem to be missing a "streamer" header -- is that because I used the fakeFilterUnitMode or does some DAQ mechanism add that later?), looking at the files themselves would be useful...

as suggested by @Sam-Harper , we are trying to test this PR on a Hilton machine (in my understanding, we would then be able to provide a sample of the streamDQMHistograms output stream), but at the moment we are having issues installing a recent CMSSW release there. I'm still trying, but given the limited availability of people this week, this might get delayed to the new year.

schneiml · 2019-12-17T09:24:11Z

@missirol thanks for the update! I'll try to pull some information out of the DQM online playback system in parallel.

schneiml · 2019-12-20T09:40:34Z

For the record: waiting for changes&validation at HLT.

schneiml · 2020-01-06T09:44:06Z

@missirol ran some tests but there were unrelated problems. Still waiting for HLT validation.

schneiml · 2020-01-13T10:16:12Z

The test by @missirol last week looks ok, so this is good to go. (Sorry, forgot to update this PR in time)

schneiml · 2020-01-13T10:16:54Z

+1

cmsbuild · 2020-01-13T10:18:25Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @silviodonato, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

silviodonato · 2020-01-13T21:26:08Z

+1

schneiml added 4 commits December 9, 2019 17:45

Import Legacy IO code from @andrius-k.

7b15a4b

Using commit 70fc23c from andrius-k:dqm-new-dqmstore-on-CMSSW_11_0_0_pre5. Includes changes to make the coe compile and run with the old DQMStore.

WIP: make DQMFileSaver a single file, and save() a function in the mo…

94a8b09

…dule.

Revert to old logic in DQMFileSaver

9c425b0

Strip out old code from DQMFileSaver

eddafb9

It looks like this was never actually used in online DQM or HLT in run2. There are plenty of threading-related code paths, but DQMServices/FileIO seems to have taken over at some point.

cmsbuild added this to the CMSSW_11_1_X milestone Dec 9, 2019

cmsbuild added code-checks-pending comparison-pending dqm-pending orp-pending pending-signatures tests-pending labels Dec 9, 2019

cmsbuild added code-checks-approved and removed code-checks-pending labels Dec 9, 2019

cmsbuild added tests-started and removed tests-pending labels Dec 9, 2019

cmsbuild added tests-approved and removed tests-started labels Dec 9, 2019

cmsbuild added comparison-available and removed comparison-pending labels Dec 9, 2019

cmsbuild modified the milestones: CMSSW_11_1_X, CMSSW_11_0_X Dec 10, 2019

cmsbuild added the code-checks-approved label Dec 13, 2019

cmsbuild added tests-started and removed tests-pending labels Dec 16, 2019

cmsbuild added tests-approved and removed tests-started labels Dec 16, 2019

cmsbuild added comparison-available and removed comparison-pending labels Dec 16, 2019

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Jan 13, 2020

cmsbuild added orp-approved and removed orp-pending labels Jan 13, 2020

cmsbuild merged commit e3e4a3c into cms-sw:master Jan 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DQM: new DQMFileSaver #28588

DQM: new DQMFileSaver #28588

schneiml commented Dec 9, 2019 •

edited

cmsbuild commented Dec 9, 2019

schneiml commented Dec 9, 2019

cmsbuild commented Dec 9, 2019

cmsbuild commented Dec 9, 2019 •

edited

cmsbuild commented Dec 9, 2019

cmsbuild commented Dec 9, 2019

cmsbuild commented Dec 9, 2019

cmsbuild commented Dec 9, 2019

schneiml commented Dec 10, 2019 •

edited

cmsbuild commented Dec 13, 2019

Sam-Harper commented Dec 16, 2019

schneiml commented Dec 16, 2019

fabiocos commented Dec 16, 2019

cmsbuild commented Dec 16, 2019 •

edited

cmsbuild commented Dec 16, 2019

cmsbuild commented Dec 16, 2019

cmsbuild commented Dec 16, 2019

missirol commented Dec 17, 2019

schneiml commented Dec 17, 2019

schneiml commented Dec 20, 2019

schneiml commented Jan 6, 2020

schneiml commented Jan 13, 2020

schneiml commented Jan 13, 2020

cmsbuild commented Jan 13, 2020

silviodonato commented Jan 13, 2020

DQM: new DQMFileSaver #28588

DQM: new DQMFileSaver #28588

Conversation

schneiml commented Dec 9, 2019 • edited

PR description:

PR validation:

cmsbuild commented Dec 9, 2019

schneiml commented Dec 9, 2019

cmsbuild commented Dec 9, 2019

cmsbuild commented Dec 9, 2019 • edited

cmsbuild commented Dec 9, 2019

cmsbuild commented Dec 9, 2019

cmsbuild commented Dec 9, 2019

cmsbuild commented Dec 9, 2019

schneiml commented Dec 10, 2019 • edited

cmsbuild commented Dec 13, 2019

Sam-Harper commented Dec 16, 2019

schneiml commented Dec 16, 2019

fabiocos commented Dec 16, 2019

cmsbuild commented Dec 16, 2019 • edited

cmsbuild commented Dec 16, 2019

cmsbuild commented Dec 16, 2019

cmsbuild commented Dec 16, 2019

missirol commented Dec 17, 2019

schneiml commented Dec 17, 2019

schneiml commented Dec 20, 2019

schneiml commented Jan 6, 2020

schneiml commented Jan 13, 2020

schneiml commented Jan 13, 2020

cmsbuild commented Jan 13, 2020

silviodonato commented Jan 13, 2020

schneiml commented Dec 9, 2019 •

edited

cmsbuild commented Dec 9, 2019 •

edited

schneiml commented Dec 10, 2019 •

edited

cmsbuild commented Dec 16, 2019 •

edited