Fix DQM ZB workflows and moved DQM FS configuration to DQM package #28351

jfernan2 · 2019-11-05T17:55:59Z

PR description:

This PR fixes the DQM ZeroBias workflows which were crashing as patched in #28336

I also took the opportunity to move DQM FastSim configuration from FastSimulation/Configuration/python/DQMOfflineMC_cff.py
to
DQMOffline/Configuration/python/DQMOfflineFS_cff.py
which was a long standing issue and resides now where it should be with the rest of DQM configurations.

A new standardDQMFS sequence is being created renaming the standardDQM running on FastSim which was misleading. All the FS workflows have been adapted to follow this new naming

PR validation:

Tested with both ZeroBias and FastSim workflows:

runTheMatrix.py -l 134.71,136.789,136.801,136.88,136.856 -i all --ibeos

runTheMatrix.py -l 5.1,5.5,135.4,135.1,135.13,135.8,2017.1,2017.13,2017.8,2018.13 -i all --ibeos

as well as usual limited workflows

if this PR is a backport please specify the original PR:

No backport is expected

…dDQMFS sequence to decouple

cmsbuild · 2019-11-05T17:56:33Z

The code-checks are being triggered in jenkins.

jfernan2 · 2019-11-05T17:57:34Z

please test workflow 134.71,136.789,136.801,136.88,5.5,135.1,135.13,135.8,2017.1,2017.13,2017.8,2018.13

cmsbuild · 2019-11-05T18:02:52Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28351/12635

This PR adds an extra 76KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/Applications/python/ConfigBuilder.py modified in PR(s): Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983, ECAL for Time dependent MC. Premix option C. Handle for APD/PN tag for time dependence #28214

cmsbuild · 2019-11-05T18:08:31Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28351/12636

This PR adds an extra 76KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/Applications/python/ConfigBuilder.py modified in PR(s): Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983, ECAL for Time dependent MC. Premix option C. Handle for APD/PN tag for time dependence #28214

cmsbuild · 2019-11-05T18:16:15Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28351/12637

This PR adds an extra 76KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/Applications/python/ConfigBuilder.py modified in PR(s): Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983, ECAL for Time dependent MC. Premix option C. Handle for APD/PN tag for time dependence #28214

cmsbuild · 2019-11-05T18:16:38Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/3360/console Started: 2019/11/05 19:18

cmsbuild · 2019-11-05T18:16:40Z

A new Pull Request was created by @jfernan2 for master.

It involves the following packages:

Configuration/Applications
Configuration/PyReleaseValidation
DQMOffline/Configuration

@cmsbuild, @andrius-k, @chayanit, @zhenhu, @schneiml, @kpedro88, @pgunnell, @kmaeshima, @franzoni, @jfernan2, @fioriNTU, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @threus, @rociovilar this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

mmusich · 2019-11-05T19:34:54Z

DQMOffline/Configuration/python/autoDQM.py


-            'commonSiStripZeroBias': ['@dcs+DQMMessageLoggerSeq+DQMOfflineCommonSiStripZeroBias+@tracking+@L1TMon+@hlt+@beam+@castor+@physics',
+            'commonSiStripZeroBias': ['@dcs+@DQMMessageLogger+@strip+@pixel+@tracking+@L1TMon+@hlt+@beam+@castor+@physics',


@jfernan2 why you substitute here :
DQMOfflineCommonSiStripZeroBias with @strip+@pixel?
Shouldn't it be @stripZeroBias+@pixel instead?

I guess Marco is right here, please @jfernan2 have a look

@jfernan2 I had noticed the same when looking at the crash, but in itself it was not sufficient to prevent it

Sorry, my mistake, I was working with two solutions in parallel and mixed them up. Apologies for the inconvenience

mmusich · 2019-11-05T19:35:34Z

DQMOffline/Configuration/python/autoDQM.py

@@ -36,11 +40,11 @@

 	    'stripZeroBias': ['DQMOfflineTrackerStripMinBias',
 			      'PostDQMOffline',
-                              'DQMHarvestTrackerStrip+DQMCertTrackerStrip'],


why is DQMCertTrackerStrip removed?

This module was giving an error without crash, so I originally removed it until I found another solution. I am very sorry for the inconvenience. Thanks for the check!

mmusich · 2019-11-05T19:35:49Z

DQMOffline/Configuration/python/autoDQM.py


 	    'pixel': ['DQMOfflineTrackerPixel',
 		      'PostDQMOffline',
-                      'DQMHarvestTrackerPixel+DQMCertTrackerPixel'],


why is DQMCertTrackerPixel removed?

This module was the cause of the crash, since it was originally removed for Phase1 wf. at the end of DQMOffline/Configuration/python/DQMOffline_Certification_cff.py
Until I realized that it took me a while. Now the configuration is much clearer IMHO so that this kind of mistakes should be less frequent now
I am very sorry. Thanks for the check!

mmusich · 2019-11-05T19:36:34Z

@arossi83 @sroychow FYI

cmsbuild · 2019-11-05T20:37:21Z

+1
Tested at: ee225f0
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8c915d/3360/summary.html

cmsbuild · 2019-11-21T18:18:56Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/3582/console Started: 2019/11/21 19:54

cmsbuild · 2019-11-21T18:18:58Z

Pull request #28351 was updated. @cmsbuild, @andrius-k, @chayanit, @zhenhu, @schneiml, @kpedro88, @pgunnell, @kmaeshima, @franzoni, @jfernan2, @fioriNTU, @fabiocos, @davidlange6 can you please check and sign again.

jfernan2 · 2019-11-21T18:21:19Z

unhold

cmsbuild · 2019-11-21T21:25:00Z

+1
Tested at: b242359
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8c915d/3582/summary.html

cmsbuild · 2019-11-21T21:25:05Z

Comparison job queued.

cmsbuild · 2019-11-21T23:10:53Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8c915d/3582/summary.html

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-8c915d/134.71_RunZeroBias2015B+RunZeroBias2015B+HLTDR2_50ns+RECODR2_50nsreHLT_ZB_HIPM+HARVESTDR2ZB
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-8c915d/136.789_RunZeroBias2017B+RunZeroBias2017B+HLTDR2_2017+RECODR2_2017reHLT_ZBPrompt+HARVEST2017ZB
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-8c915d/136.801_RunZeroBias2017C+RunZeroBias2017C+HLTDR2_2017+RECODR2_2017reHLT_ZBPrompt+HARVEST2017ZB
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-8c915d/136.856_RunZeroBias2018A+RunZeroBias2018A+HLTDR2_2018+RECODR2_2018reHLT_ZBOffline+HARVEST2018ZB
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-8c915d/136.88_RunZeroBias2018C+RunZeroBias2018C+HLTDR2_2018+RECODR2_2018reHLT_ZBOffline+HARVEST2018ZB

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2785613
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2785271
DQMHistoTests: Total skipped: 341
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
Checked 147 log files, 16 edm output root files, 34 DQM output files

jfernan2 · 2019-11-22T06:50:30Z

+1
No changes as expected

fabiocos · 2019-11-22T08:20:16Z

+operations

kpedro88 · 2019-11-23T00:10:05Z

+upgrade

fabiocos · 2019-11-23T20:41:12Z

@chayanit I consider your previous signature are valid

fabiocos · 2019-11-23T20:41:40Z

+1

fabiocos · 2019-11-23T20:42:01Z

merge

jfernan2 added 4 commits November 4, 2019 18:17

Fix to patch set on PR28336

1dc047b

Final tuning

11da9d9

Removed Pixel Phase1 certification in the new configuration

0c0e89e

Moved FS DQM confiuration into DQMOffline package and created standar…

ee225f0

…dDQMFS sequence to decouple

cmsbuild added this to the CMSSW_11_0_X milestone Nov 5, 2019

cmsbuild added code-checks-pending comparison-pending dqm-pending operations-pending orp-pending pdmv-pending pending-signatures tests-pending upgrade-pending labels Nov 5, 2019

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 5, 2019

cmsbuild added tests-started and removed tests-pending labels Nov 5, 2019

mmusich reviewed Nov 5, 2019

View reviewed changes

cmsbuild added tests-started and removed tests-pending labels Nov 21, 2019

cmsbuild removed the hold label Nov 21, 2019

cmsbuild added tests-approved and removed tests-started labels Nov 21, 2019

cmsbuild added comparison-available and removed comparison-pending labels Nov 21, 2019

cmsbuild added dqm-approved and removed dqm-pending labels Nov 22, 2019

cmsbuild added operations-approved and removed operations-pending labels Nov 22, 2019

fabiocos mentioned this pull request Nov 22, 2019

Finalization of CMSSW_11_0_X open development #28455

Closed

9 tasks

cmsbuild added upgrade-approved and removed upgrade-pending labels Nov 23, 2019

cmsbuild mentioned this pull request Nov 23, 2019

Changes for new trained regression for UL 2016 - 110X #28422

Closed

cmsbuild added orp-approved and removed orp-pending labels Nov 23, 2019

cmsbuild merged commit 7e27750 into cms-sw:master Nov 23, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix DQM ZB workflows and moved DQM FS configuration to DQM package #28351

Fix DQM ZB workflows and moved DQM FS configuration to DQM package #28351

jfernan2 commented Nov 5, 2019

cmsbuild commented Nov 5, 2019

jfernan2 commented Nov 5, 2019

cmsbuild commented Nov 5, 2019

cmsbuild commented Nov 5, 2019

cmsbuild commented Nov 5, 2019

cmsbuild commented Nov 5, 2019 •

edited

cmsbuild commented Nov 5, 2019

mmusich Nov 5, 2019

fioriNTU Nov 6, 2019

fabiocos Nov 6, 2019

jfernan2 Nov 6, 2019

mmusich Nov 5, 2019

jfernan2 Nov 6, 2019

mmusich Nov 5, 2019

jfernan2 Nov 6, 2019

mmusich commented Nov 5, 2019

cmsbuild commented Nov 5, 2019

cmsbuild commented Nov 21, 2019 •

edited

cmsbuild commented Nov 21, 2019

jfernan2 commented Nov 21, 2019

cmsbuild commented Nov 21, 2019

cmsbuild commented Nov 21, 2019

cmsbuild commented Nov 21, 2019

jfernan2 commented Nov 22, 2019

fabiocos commented Nov 22, 2019

kpedro88 commented Nov 23, 2019

fabiocos commented Nov 23, 2019

fabiocos commented Nov 23, 2019

fabiocos commented Nov 23, 2019


		'commonSiStripZeroBias': ['@dcs+DQMMessageLoggerSeq+DQMOfflineCommonSiStripZeroBias+@tracking+@L1TMon+@hlt+@beam+@castor+@physics',
		'commonSiStripZeroBias': ['@dcs+@DQMMessageLogger+@strip+@pixel+@tracking+@L1TMon+@hlt+@beam+@castor+@physics',

Fix DQM ZB workflows and moved DQM FS configuration to DQM package #28351

Fix DQM ZB workflows and moved DQM FS configuration to DQM package #28351

Conversation

jfernan2 commented Nov 5, 2019

PR description:

PR validation:

if this PR is a backport please specify the original PR:

cmsbuild commented Nov 5, 2019

jfernan2 commented Nov 5, 2019

cmsbuild commented Nov 5, 2019

cmsbuild commented Nov 5, 2019

cmsbuild commented Nov 5, 2019

cmsbuild commented Nov 5, 2019 • edited

cmsbuild commented Nov 5, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mmusich commented Nov 5, 2019

cmsbuild commented Nov 5, 2019

cmsbuild commented Nov 21, 2019 • edited

cmsbuild commented Nov 21, 2019

jfernan2 commented Nov 21, 2019

cmsbuild commented Nov 21, 2019

cmsbuild commented Nov 21, 2019

cmsbuild commented Nov 21, 2019

jfernan2 commented Nov 22, 2019

fabiocos commented Nov 22, 2019

kpedro88 commented Nov 23, 2019

fabiocos commented Nov 23, 2019

fabiocos commented Nov 23, 2019

fabiocos commented Nov 23, 2019

cmsbuild commented Nov 5, 2019 •

edited

cmsbuild commented Nov 21, 2019 •

edited