Fix EcalDQMStatusWriter to watch runs and load the input file #35466

thomreis · 2021-09-29T08:30:45Z

PR description:

After the esConsumes migration of the EcalDQMStatusWriter in #34963 the beginRun() function was not called anymore by the framework because the module has been made an edm::one::EDAnalyzer<>. This means that the input file was not loaded and the module would have added only empty objects to the records. This PR changes the module type to a edm::one::EDAnalyzer<edm::one::WatchRuns> in order to call the beginRun() function. The configuration file, which had been outdated, was updated to send the correct parameters to the module.
In addition, the config file for readEcalDQMStatus was removed since the EcalDQMStatusReader module it is supposed to configure does not exist in CMSSW.

PR validation:

This module is run manually only and it has been tested by running the DQM/EcalCommon/data/writeEcalDQMStatus.py configuration, which is configured to produce an sqlite file with the two records.

… to date. Remove config file for non-existing EcalDQMStatusReader plugin.

cmsbuild · 2021-09-29T08:39:00Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35466/25621

This PR adds an extra 16KB to repository

cmsbuild · 2021-09-29T08:39:25Z

A new Pull Request was created by @thomreis (Thomas Reis) for master.

It involves the following packages:

DQM/EcalCommon (dqm)
DQM/EcalMonitorDbModule (dqm)

@emanueleusai, @ahmad3213, @cmsbuild, @jfernan2, @pmandrik, @pbo0, @rvenditti can you please review it and eventually sign? Thanks.
@rchatter, @simonepigazzini, @thomreis, @argiro this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

thomreis · 2021-09-29T08:41:07Z

@abhih1 @tanmaymudholkar FYI

jfernan2 · 2021-09-29T09:15:06Z

please test

cmsbuild · 2021-09-29T12:27:55Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d1676c/19230/summary.html
COMMIT: 366bfbf
CMSSW: CMSSW_12_1_X_2021-09-28-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/35466/19230/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 40
DQMHistoTests: Total histograms compared: 3211080
DQMHistoTests: Total failures: 11
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3211046
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 39 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 169 log files, 37 edm output root files, 40 DQM output files
TriggerResults: no differences found

jfernan2 · 2021-09-29T12:30:57Z

+1

cmsbuild · 2021-09-29T12:31:16Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

thomreis · 2021-09-29T12:40:51Z

hold

Do not merge this yet please. I need to make one more check. Probably unrelated to this PR but I want to be sure.

cmsbuild · 2021-09-29T12:41:13Z

Pull request has been put on hold by @thomreis
They need to issue an unhold command to remove the hold state or L1 can unhold it for all

thomreis · 2021-09-29T14:10:59Z

unhold

jfernan2 · 2021-09-29T14:15:18Z

please test

cmsbuild · 2021-09-29T14:16:31Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35466/25632

This PR adds an extra 16KB to repository

cmsbuild · 2021-09-29T14:16:55Z

Pull request #35466 was updated. @emanueleusai, @ahmad3213, @jfernan2, @rvenditti, @pbo0, @pmandrik can you please check and sign again.

cmsbuild · 2021-09-29T17:42:53Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d1676c/19244/summary.html
COMMIT: b0771e7
CMSSW: CMSSW_12_1_X_2021-09-28-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/35466/19244/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 1 differences found in the comparisons
DQMHistoTests: Total files compared: 40
DQMHistoTests: Total histograms compared: 3211080
DQMHistoTests: Total failures: 3
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3211055
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 39 files compared)
Checked 169 log files, 37 edm output root files, 40 DQM output files
TriggerResults: no differences found

jfernan2 · 2021-09-29T17:48:25Z

@thomreis apparently the PR is fine, but since the real validation comes from you in the form of the db check, I will wait for your green light before approving.
Thanks

thomreis · 2021-09-29T21:16:37Z

Hi @jfernan2 in the current state the xml dump of the created payloads looks reasonable. From my side I can give the green light. Thanks for approving.

jfernan2 · 2021-09-30T06:57:38Z

+1

cmsbuild · 2021-09-30T06:58:02Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2021-09-30T09:44:26Z

+1

Fix

ggovi · 2021-10-05T09:43:37Z

DQM/EcalCommon/data/writeEcalDQMStatus.py

-#process.CondDBCommon.connect = 'oracle://cms_orcon_prod/CMS_COND_34X_ECAL'
+process.load("CondCore.CondDB.CondDB_cfi")
+process.CondDB.DBParameters.authenticationPath = '/nfshome0/popcondev/conddb'
+process.CondDB.connect = 'sqlite_file:mask-ECAL.db'


@thomreis This is the change producing a valid sqlite... nothing to do with conddb dump

Without changing the writeOne() functions in DQM/EcalMonitorDbModule/plugins/EcalDQMStatusWriter.cc from pointer based to reference based the produced sqlite file was not valid. Or at least the payload in it could not be printed.

thomreis · 2021-10-05T09:51:29Z

DQM/EcalMonitorDbModule/plugins/EcalDQMStatusWriter.cc

 void EcalDQMStatusWriter::analyze(edm::Event const &, edm::EventSetup const &_es) {
  cond::service::PoolDBOutputService &dbOutput(*edm::Service<cond::service::PoolDBOutputService>());
  if (firstRun_ == dbOutput.endOfTime())
    return;

-  dbOutput.writeOne(&channelStatus_, firstRun_, "EcalDQMChannelStatusRcd");
-  dbOutput.writeOne(&towerStatus_, firstRun_, "EcalDQMTowerStatusRcd");
+  dbOutput.writeOne(channelStatus_, firstRun_, "EcalDQMChannelStatusRcd");


Hi @ggovi
The changes that I needed to make in order to get conddb dump to work are these two lines.

@thomreis Can you please dump the error that you get with conddb dump?

Sure. I have also sent it in an email to you last week already.

$ conddb --db mask-ECAL.db dump a68ed854e09eb04addd68ec350b820a4e171ca66 [2021-09-29 16:03:38,098] INFO: Connecting to mask-ECAL.db [sqlite:///mask-ECAL.db] [2021-09-29 16:03:38,150] INFO: Found payload of type EcalCondTowerObjectContainer<EcalDQMStatusCode>* [2021-09-29 16:03:39,303] WARNING: No XML converter for payload class EcalCondTowerObjectContainer<EcalDQMStatusCode>* found in the built-in library.

@thomreis thanks, i'll investigate.

Fix EcalDQMStatusWriter by watching runs and bring the config file up…

366bfbf

… to date. Remove config file for non-existing EcalDQMStatusReader plugin.

cmsbuild added this to the CMSSW_12_1_X milestone Sep 29, 2021

cmsbuild added code-checks-pending dqm-pending orp-pending pending-signatures tests-pending labels Sep 29, 2021

thomreis changed the title ~~Fix EcalDQMStatusWriter to watching runs and load the input file~~ Fix EcalDQMStatusWriter to watch runs and load the input file Sep 29, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Sep 29, 2021

cmsbuild added tests-started and removed tests-pending labels Sep 29, 2021

cmsbuild added tests-approved and removed tests-started labels Sep 29, 2021

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Sep 29, 2021

cmsbuild added hold pending-signatures and removed fully-signed labels Sep 29, 2021

cmsbuild added dqm-pending tests-pending labels Sep 29, 2021

cmsbuild removed the hold label Sep 29, 2021

cmsbuild added tests-started and removed tests-pending labels Sep 29, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Sep 29, 2021

cmsbuild added tests-approved and removed tests-started labels Sep 29, 2021

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Sep 30, 2021

cmsbuild added orp-approved and removed orp-pending labels Sep 30, 2021

cmsbuild merged commit 7a2f691 into cms-sw:master Sep 30, 2021

cmsbuild mentioned this pull request Sep 30, 2021

Utilities/StaticAnalyzer: Update ESR::get report generator with deprecated_get end point function. #35494

Merged

ggovi reviewed Oct 5, 2021

View reviewed changes

thomreis commented Oct 5, 2021

View reviewed changes

thomreis deleted the ecaldqm-dqmstatuswriter-update branch October 6, 2021 10:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix EcalDQMStatusWriter to watch runs and load the input file #35466

Fix EcalDQMStatusWriter to watch runs and load the input file #35466

thomreis commented Sep 29, 2021

cmsbuild commented Sep 29, 2021

cmsbuild commented Sep 29, 2021

thomreis commented Sep 29, 2021

jfernan2 commented Sep 29, 2021

cmsbuild commented Sep 29, 2021

jfernan2 commented Sep 29, 2021

cmsbuild commented Sep 29, 2021

thomreis commented Sep 29, 2021

cmsbuild commented Sep 29, 2021

thomreis commented Sep 29, 2021

jfernan2 commented Sep 29, 2021

cmsbuild commented Sep 29, 2021

cmsbuild commented Sep 29, 2021

cmsbuild commented Sep 29, 2021

jfernan2 commented Sep 29, 2021

thomreis commented Sep 29, 2021

jfernan2 commented Sep 30, 2021

cmsbuild commented Sep 30, 2021

perrotta commented Sep 30, 2021

ggovi Oct 5, 2021

thomreis Oct 5, 2021

thomreis Oct 5, 2021

ggovi Oct 5, 2021

thomreis Oct 5, 2021

ggovi Oct 5, 2021

Fix EcalDQMStatusWriter to watch runs and load the input file #35466

Fix EcalDQMStatusWriter to watch runs and load the input file #35466

Conversation

thomreis commented Sep 29, 2021

PR description:

PR validation:

cmsbuild commented Sep 29, 2021

cmsbuild commented Sep 29, 2021

thomreis commented Sep 29, 2021

jfernan2 commented Sep 29, 2021

cmsbuild commented Sep 29, 2021

Comparison Summary

jfernan2 commented Sep 29, 2021

cmsbuild commented Sep 29, 2021

thomreis commented Sep 29, 2021

cmsbuild commented Sep 29, 2021

thomreis commented Sep 29, 2021

jfernan2 commented Sep 29, 2021

cmsbuild commented Sep 29, 2021

cmsbuild commented Sep 29, 2021

cmsbuild commented Sep 29, 2021

Comparison Summary

jfernan2 commented Sep 29, 2021

thomreis commented Sep 29, 2021

jfernan2 commented Sep 30, 2021

cmsbuild commented Sep 30, 2021

perrotta commented Sep 30, 2021

ggovi Oct 5, 2021

Choose a reason for hiding this comment

thomreis Oct 5, 2021

Choose a reason for hiding this comment

thomreis Oct 5, 2021

Choose a reason for hiding this comment

ggovi Oct 5, 2021

Choose a reason for hiding this comment

thomreis Oct 5, 2021

Choose a reason for hiding this comment

ggovi Oct 5, 2021

Choose a reason for hiding this comment