[DQMOffline/Trigger] improved DQM plugin for monitoring of HLT path/filter efficiencies #29095

missirol · 2020-03-04T12:59:32Z

PR description:

This PR is meant as an improvement of the DQM plugin used to monitor the efficiencies of all HLT paths in a given HLT-Menu, as well as the efficiencies of the individual filters making up a given (HLT) path.

Currently, this monitoring is done by the plugin GeneralHLTOffline, which builds a set of histograms (TH1Fs) counting how many times a given trigger fired, and how many times a given filter was executed.

The PR replaces GeneralHLTOffline with a new plugin named HLTFiltersDQMonitor, which serves the same purpose, but produces DQM outputs in a slightly different format.
In HLTFiltersDQMonitor, TProfiles are used to obtain efficiencies, of either a trigger path or an individual filter; this (i.e. using efficiencies via TProfiles instead of counts via TH1Fs) allows to make comparisons straightforward between runs/samples with different number of events; using TProfiles also allows to have the efficiency estimates directly, i.e. without any dedicated harvesting of these DQM outputs (thanks to @schneiml for the suggestion to use TProfiles). More specifically, the new DQM outputs (under HLT/Filters/) include

one plot with the efficiencies of every path in the HLT menu
for every primary dataset (PD) associated to physics-related streams, a directory which contains (1) a plot with the efficiencies of the paths associated to that PD, and (2) a plot for every path, showing the efficiency of each individual filter making up the path.

Since the PR changes the format of these DQM outputs (from counts to efficiencies), the comparisons will not be very meaningful; we will instead have to check the new outputs and verify that they are in agreement with expectations (an apples-to-apples comparison to the outputs of GeneralHLTOffline would not be possible anyway, because of a couple of fixes
included in the new plugin, with respect to the old one).

The new plugin was validated with a number of local tests with MC, but it will be useful to check what happens when testing it on a larger scale during the PR review.

The changes in test/ are mostly just renaming, for consistency, to accommodate for the new plugin (most of the configs in test/ seem fairly old anyway).

FYI @Sam-Harper @gennai

PR validation:

Successfully ran a few standard workflows, e.g. runTheMatrix.py -l 11634.0.

…/ and test/

cmsbuild · 2020-03-04T13:00:00Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-03-04T13:06:14Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-29095/13994

This PR adds an extra 36KB to repository

cmsbuild · 2020-03-04T13:06:39Z

A new Pull Request was created by @missirol (Marino Missiroli) for master.

It involves the following packages:

DQMOffline/Trigger

@andrius-k, @kmaeshima, @schneiml, @cmsbuild, @jfernan2, @fioriNTU can you please review it and eventually sign? Thanks.
@battibass, @mtosi, @Fedespring, @calderona, @HuguesBrun, @jhgoh, @cericeci, @trocino, @folguera, @rociovilar this is something you requested to watch as well.
@davidlange6, @silviodonato, @fabiocos you are the release manager for this.

cms-bot commands are listed here

jfernan2 · 2020-03-04T21:05:51Z

please test

cmsbuild · 2020-03-04T21:06:12Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/4999/console Started: 2020/03/04 22:48

cmsbuild · 2020-03-04T23:04:30Z

-1

Tested at: 42a848b

CMSSW: CMSSW_11_1_X_2020-03-04-1100
SCRAM_ARCH: slc7_amd64_gcc820
You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0a3cd6/4999/summary.html

I found follow errors while testing this PR

Failed tests: RelVals

RelVals:

When I ran the RelVals I found an error in the following workflows:
140.53 step2

runTheMatrix-results/140.53_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI/step2_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI.log

cmsbuild · 2020-03-04T23:04:33Z

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

…avyIons_cff

missirol · 2020-03-05T06:25:33Z

@jfernan2

I think the 2 lines removed in the last commit were previously having no effect (hltResults was an instance of GeneralHLTOffline, which did not have members named RecHitsEBTag and RecHitsEETag).

cmsbuild · 2020-03-05T06:32:45Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-29095/14005

This PR adds an extra 40KB to repository

cmsbuild · 2020-03-05T06:33:09Z

Pull request #29095 was updated. @andrius-k, @kmaeshima, @schneiml, @cmsbuild, @jfernan2, @fioriNTU can you please check and sign again.

jfernan2 · 2020-03-05T07:52:16Z

please test

cmsbuild · 2020-03-05T07:52:41Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/5002/console Started: 2020/03/05 10:34

Sam-Harper · 2020-03-05T08:27:37Z

thank you @missirol!

cmsbuild · 2020-03-05T10:51:43Z

+1
Tested at: d984d97
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0a3cd6/5002/summary.html
CMSSW: CMSSW_11_1_X_2020-03-04-2300
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-03-05T10:51:46Z

Comparison job queued.

cmsbuild · 2020-03-05T12:22:38Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-0a3cd6/5002/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 14 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2678823
DQMHistoTests: Total failures: 44
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2678460
DQMHistoTests: Total skipped: 319
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 100.388 KiB( 33 files compared)
DQMHistoSizes: changed ( 1000.0 ): 49.341 KiB HLT/Filters
DQMHistoSizes: changed ( 1000.0 ): -34.514 KiB HLT/GeneralHLTOffline
DQMHistoSizes: changed ( 10024.0,... ): 0.314 KiB HLT/Filters
DQMHistoSizes: changed ( 10024.0,... ): -0.298 KiB HLT/GeneralHLTOffline
DQMHistoSizes: changed ( 11634.0,... ): 123.676 KiB HLT/Filters
DQMHistoSizes: changed ( 11634.0,... ): -81.656 KiB HLT/GeneralHLTOffline
DQMHistoSizes: changed ( 4.22 ): 3.613 KiB HLT/Filters
DQMHistoSizes: changed ( 4.22 ): -2.195 KiB HLT/GeneralHLTOffline
DQMHistoSizes: changed ( 8.0 ): 0.313 KiB HLT/Filters
DQMHistoSizes: changed ( 8.0 ): -0.296 KiB HLT/GeneralHLTOffline
Checked 147 log files, 16 edm output root files, 34 DQM output files

missirol · 2020-03-05T13:33:42Z

fwiw, I didn't catch something obviously incorrect in the new outputs; the efficiency values themselves are harder to judge because of the limited stats, but things look as expected.

jfernan2 · 2020-03-05T21:07:16Z

+1

cmsbuild · 2020-03-05T21:07:42Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @silviodonato, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

silviodonato · 2020-03-09T15:38:54Z

+1

Marino Missiroli added 6 commits March 3, 2020 14:59

[DQMOffline/Trigger] removed plugin GeneralHLTOffline

e08241e

[DQMOffline/Trigger] added plugin HLTFiltersDQMonitor

7c217f1

[DQMOffline/Trigger] updates related to HLTFiltersDQMonitor in python…

5ae9ba7

…/ and test/

[DQMOffline/Trigger] fixed calls to bookProfile in HLTFiltersDQMonitor

95701af

[DQMOffline/Trigger] minor renaming in HLTFiltersDQMonitor

5adc020

[DQMOffline/Trigger] small update to HLTFiltersDQMonitor

42a848b

cmsbuild added this to the CMSSW_11_1_X milestone Mar 4, 2020

cmsbuild added code-checks-pending comparison-pending dqm-pending orp-pending pending-signatures tests-pending labels Mar 4, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 4, 2020

cmsbuild added tests-started and removed tests-pending labels Mar 4, 2020

cmsbuild added comparison-notrun tests-rejected and removed comparison-pending tests-started labels Mar 4, 2020

[DQMOffline/Configuration] removed hltResults remnant in DQMOfflineHe…

d984d97

…avyIons_cff

cmsbuild removed the code-checks-approved label Mar 5, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 5, 2020

cmsbuild added tests-started and removed tests-pending labels Mar 5, 2020

cmsbuild added tests-approved and removed tests-started labels Mar 5, 2020

cmsbuild added comparison-available and removed comparison-pending labels Mar 5, 2020

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Mar 5, 2020

cmsbuild added orp-approved and removed orp-pending labels Mar 9, 2020

cmsbuild merged commit 7c4303c into cms-sw:master Mar 9, 2020

cmsbuild mentioned this pull request Mar 9, 2020

Reduce boost dependence in EcalDigi data formats #29147

Merged

missirol deleted the devel_hltFiltersDQMonitor branch March 12, 2020 19:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DQMOffline/Trigger] improved DQM plugin for monitoring of HLT path/filter efficiencies #29095

[DQMOffline/Trigger] improved DQM plugin for monitoring of HLT path/filter efficiencies #29095

missirol commented Mar 4, 2020

cmsbuild commented Mar 4, 2020

cmsbuild commented Mar 4, 2020

cmsbuild commented Mar 4, 2020

jfernan2 commented Mar 4, 2020

cmsbuild commented Mar 4, 2020 •

edited

cmsbuild commented Mar 4, 2020

cmsbuild commented Mar 4, 2020

missirol commented Mar 5, 2020 •

edited

cmsbuild commented Mar 5, 2020

cmsbuild commented Mar 5, 2020

jfernan2 commented Mar 5, 2020

cmsbuild commented Mar 5, 2020 •

edited

Sam-Harper commented Mar 5, 2020

cmsbuild commented Mar 5, 2020

cmsbuild commented Mar 5, 2020

cmsbuild commented Mar 5, 2020

missirol commented Mar 5, 2020

jfernan2 commented Mar 5, 2020

cmsbuild commented Mar 5, 2020

silviodonato commented Mar 9, 2020

[DQMOffline/Trigger] improved DQM plugin for monitoring of HLT path/filter efficiencies #29095

[DQMOffline/Trigger] improved DQM plugin for monitoring of HLT path/filter efficiencies #29095

Conversation

missirol commented Mar 4, 2020

PR description:

PR validation:

cmsbuild commented Mar 4, 2020

cmsbuild commented Mar 4, 2020

cmsbuild commented Mar 4, 2020

jfernan2 commented Mar 4, 2020

cmsbuild commented Mar 4, 2020 • edited

cmsbuild commented Mar 4, 2020

cmsbuild commented Mar 4, 2020

missirol commented Mar 5, 2020 • edited

cmsbuild commented Mar 5, 2020

cmsbuild commented Mar 5, 2020

jfernan2 commented Mar 5, 2020

cmsbuild commented Mar 5, 2020 • edited

Sam-Harper commented Mar 5, 2020

cmsbuild commented Mar 5, 2020

cmsbuild commented Mar 5, 2020

cmsbuild commented Mar 5, 2020

missirol commented Mar 5, 2020

jfernan2 commented Mar 5, 2020

cmsbuild commented Mar 5, 2020

silviodonato commented Mar 9, 2020

cmsbuild commented Mar 4, 2020 •

edited

missirol commented Mar 5, 2020 •

edited

cmsbuild commented Mar 5, 2020 •

edited