online Beamspot monitor DQM client (backporting of PR 31904) #31924

gennai · 2020-10-23T13:57:38Z

PR description:

With the new BeamSpot workflow in Run3 two clients will run in DQM producing two different beamspot payloads. An ESProducer is run at HLT to make a reproducible choice of which payload to use. This new client is monitoring the values of the two payloads as well as plotting which one of the two is chosen by the ESProducer logic.

PR validation:

I have run the DQM/Integration/python/clients/onlinebeammonitor_dqm_sourceclient-live_cfg.py with unitTest=True option.
The plots looks right.

if this PR is a backport please specify the original PR and why you need to backport that PR:

this is a backporting of #31904, it is needed for next MWGR

cmsbuild · 2020-10-23T13:58:07Z

A new Pull Request was created by @gennai (simone gennai) for CMSSW_11_1_X.

It involves the following packages:

DQM/BeamMonitor
DQM/Integration

@andrius-k, @kmaeshima, @ErnestaP, @cmsbuild, @jfernan2, @fioriNTU can you please review it and eventually sign? Thanks.
@threus, @batinkov, @battibass this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

gennai · 2020-10-23T13:58:56Z

@boudoul @amassiro @mmusich @mtosi @francescobrivio @ggovi you may be interested in this PR as well

jfernan2 · 2020-10-23T14:46:11Z

please test

cmsbuild · 2020-10-23T14:46:33Z

The tests are being triggered in jenkins.

CMSSW_11_1_X_2020-10-23-1100/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/10249/console Started: 2020/10/23 16:47

cmsbuild · 2020-10-23T16:07:27Z

-1

Tested at: a15087c

CMSSW: CMSSW_11_1_X_2020-10-23-1100
SCRAM_ARCH: slc7_amd64_gcc820
You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fd4d6c/10249/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests

Unit Tests:

I found errors in the following unit tests:

---> test TestDQMOnlineClient-onlinebeammonitor_dqm_sourceclient had ERRORS

cmsbuild · 2020-10-23T16:07:29Z

Comparison job queued.

mmusich · 2020-10-23T16:11:55Z

DQM/Integration/python/clients/onlinebeammonitor_dqm_sourceclient-live_cfg.py

+
+                            ),
+                      ),
+                      connect = cms.string('oracle://cms_orcon_prod/CMS_CONDITIONS')


This won't work outside of P5 (e.g. in unit tests).

Monitoring file not found, disabling. ----- Begin Fatal Exception 23-Oct-2020 16:51:46 CEST----------------------- An exception of category 'ConditionDatabase' occurred while [0] Constructing the EventProcessor [1] Constructing ESSource: class=PoolDBESSource label='BeamSpotDBSource' Exception Message: Service "cms_orcon_prod" can't be open with the current key. from cond::CredentialStore::setUpConnection ----- End Fatal Exception ------------------------------------------------- ---> test TestDQMOnlineClient-onlinebeammonitor_dqm_sourceclient had ERRORS

but it is working on lxplus, at least it is working for me. Do you have the proper connection string to put?

Maybe because you have been given a key file with reader access priviliges by @ggovi ?
frontier://FrontierProd/CMS_CONDITIONS will work for any user, but not sure that's what you want (at P5).

mmusich · 2020-10-23T16:12:14Z

DQM/Integration/python/clients/onlinebeammonitor_dqm_sourceclient-live_cfg.py

+  process.source.streamLabel = cms.untracked.string('streamDQMOnlineBeamspot')
+
+#ESProducer
+process.load("CondCore.DBCommon.CondDBSetup_cfi")


please use CondCore.CondDB.CondDB_cfi instead

cmsbuild · 2020-10-23T17:53:43Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fd4d6c/10249/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 36
DQMHistoTests: Total histograms compared: 2784776
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2784725
DQMHistoTests: Total skipped: 50
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 35 files compared)
Checked 152 log files, 16 edm output root files, 36 DQM output files

…is set

cmsbuild · 2020-10-26T07:01:12Z

Pull request #31924 was updated. @andrius-k, @kmaeshima, @ErnestaP, @cmsbuild, @jfernan2, @fioriNTU can you please check and sign again.

jfernan2 · 2020-10-26T09:14:54Z

please test

cmsbuild · 2020-10-26T09:15:16Z

The tests are being triggered in jenkins.

CMSSW_11_1_X_2020-10-25-2300/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/10282/console Started: 2020/10/26 10:17

cmsbuild · 2020-10-29T18:25:25Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fd4d6c/10385/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 36
DQMHistoTests: Total histograms compared: 2784776
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2784725
DQMHistoTests: Total skipped: 50
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 35 files compared)
Checked 152 log files, 16 edm output root files, 36 DQM output files

gennai · 2020-11-03T08:16:59Z

Hi, I have checked DT_DQM error and in fact it does not depend on my PR:
A std::exception was thrown.
failed opening file: No such file or directory: iostream error
----- End Fatal Exception -------------------------------------------------
29-Oct-2020 16:17:14 CET Closed file root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/express/Commissioning2019/ExpressCosmics/FEVT/Express-v1/000/334/393/00000/FE21789A-F777-0B43-A1F5-E43F1FD52D19.root

---> test TestDQMOnlineClient-dt_dqm_sourceclient had ERRORS

It is more an issue with the files chosen to be run in the unitTest. @jfernan2, @silviodonato How shall I proceed?

mmusich · 2020-11-03T08:28:41Z

@cmsbuild, please test
let's try a new round of tests, perhaps was a transient glitch

cmsbuild · 2020-11-03T08:29:05Z

The tests are being triggered in jenkins.

CMSSW_11_1_X_2020-11-02-2300/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/10462/console Started: 2020/11/03 10:18

cmsbuild · 2020-11-03T10:42:48Z

+1
Tested at: 1a54f9a
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fd4d6c/10462/summary.html
CMSSW: CMSSW_11_1_X_2020-11-02-2300
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-11-03T10:42:50Z

Comparison job queued.

cmsbuild · 2020-11-03T12:29:32Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fd4d6c/10462/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 36
DQMHistoTests: Total histograms compared: 2784776
DQMHistoTests: Total failures: 2
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2784724
DQMHistoTests: Total skipped: 50
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 35 files compared)
Checked 152 log files, 16 edm output root files, 36 DQM output files

jfernan2 · 2020-11-03T12:49:34Z

+1

cmsbuild · 2020-11-03T12:50:02Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_11_1_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_11_2_X is complete. This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

silviodonato · 2020-11-03T17:30:17Z

+1

backporting of PR 31904

a15087c

cmsbuild added this to the CMSSW_11_1_X milestone Oct 23, 2020

cmsbuild added comparison-pending dqm-pending orp-pending pending-signatures tests-pending labels Oct 23, 2020

cmsbuild added tests-started and removed tests-pending labels Oct 23, 2020

cmsbuild added tests-rejected and removed tests-started labels Oct 23, 2020

mmusich reviewed Oct 23, 2020

View reviewed changes

cmsbuild added comparison-available and removed comparison-pending labels Oct 23, 2020

Modified the python scripts to run using frontier when unitTest=True …

4c6e354

…is set

cmsbuild added comparison-pending tests-pending and removed comparison-available tests-rejected labels Oct 26, 2020

cmsbuild removed the tests-pending label Oct 26, 2020

cmsbuild added the tests-rejected label Oct 29, 2020

cmsbuild added comparison-available and removed comparison-pending labels Oct 29, 2020

cmsbuild added comparison-pending tests-pending and removed comparison-available tests-rejected labels Nov 3, 2020

cmsbuild added tests-started and removed tests-pending labels Nov 3, 2020

cmsbuild added tests-approved and removed tests-started labels Nov 3, 2020

cmsbuild added comparison-available and removed comparison-pending labels Nov 3, 2020

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Nov 3, 2020

cmsbuild added orp-approved and removed orp-pending labels Nov 3, 2020

cmsbuild merged commit c63f534 into cms-sw:CMSSW_11_1_X Nov 3, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

online Beamspot monitor DQM client (backporting of PR 31904) #31924

online Beamspot monitor DQM client (backporting of PR 31904) #31924

gennai commented Oct 23, 2020

cmsbuild commented Oct 23, 2020

gennai commented Oct 23, 2020

jfernan2 commented Oct 23, 2020

cmsbuild commented Oct 23, 2020 •

edited

cmsbuild commented Oct 23, 2020

cmsbuild commented Oct 23, 2020

mmusich Oct 23, 2020

gennai Oct 23, 2020

mmusich Oct 23, 2020 •

edited

mmusich Oct 23, 2020

cmsbuild commented Oct 23, 2020

cmsbuild commented Oct 26, 2020

jfernan2 commented Oct 26, 2020

cmsbuild commented Oct 26, 2020 •

edited

cmsbuild commented Oct 29, 2020

gennai commented Nov 3, 2020

mmusich commented Nov 3, 2020

cmsbuild commented Nov 3, 2020 •

edited

cmsbuild commented Nov 3, 2020

cmsbuild commented Nov 3, 2020

cmsbuild commented Nov 3, 2020

jfernan2 commented Nov 3, 2020

cmsbuild commented Nov 3, 2020

silviodonato commented Nov 3, 2020

online Beamspot monitor DQM client (backporting of PR 31904) #31924

online Beamspot monitor DQM client (backporting of PR 31904) #31924

Conversation

gennai commented Oct 23, 2020

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented Oct 23, 2020

gennai commented Oct 23, 2020

jfernan2 commented Oct 23, 2020

cmsbuild commented Oct 23, 2020 • edited

cmsbuild commented Oct 23, 2020

cmsbuild commented Oct 23, 2020

mmusich Oct 23, 2020

Choose a reason for hiding this comment

gennai Oct 23, 2020

Choose a reason for hiding this comment

mmusich Oct 23, 2020 • edited

Choose a reason for hiding this comment

mmusich Oct 23, 2020

Choose a reason for hiding this comment

cmsbuild commented Oct 23, 2020

cmsbuild commented Oct 26, 2020

jfernan2 commented Oct 26, 2020

cmsbuild commented Oct 26, 2020 • edited

cmsbuild commented Oct 29, 2020

gennai commented Nov 3, 2020

mmusich commented Nov 3, 2020

cmsbuild commented Nov 3, 2020 • edited

cmsbuild commented Nov 3, 2020

cmsbuild commented Nov 3, 2020

cmsbuild commented Nov 3, 2020

jfernan2 commented Nov 3, 2020

cmsbuild commented Nov 3, 2020

silviodonato commented Nov 3, 2020

cmsbuild commented Oct 23, 2020 •

edited

mmusich Oct 23, 2020 •

edited

cmsbuild commented Oct 26, 2020 •

edited

cmsbuild commented Nov 3, 2020 •

edited