Fix to allow concurrent processing of lumisections with ECAL DQM #31689

tanmaymudholkar · 2020-10-07T02:25:42Z

Change DQMOneLumiEDAnalyzer to DQMOneEDAnalyzer to allow concurrent processing of lumisections.

PR description:

This PR is meant to allow concurrent processing of lumisections with ECAL included in the workflow. This is done by changing the class from which the tasks inherit from DQMOneLumiEDAnalyzer to DQMOneEDAnalyzer.

No changes are expected in the output.

This PR supersedes earlier ECAL PR 30187 (#30187).

PR validation:

The standard suite of tests was run: runTheMatrix.py -l limited -i all --ibeos

if this PR is a backport please specify the original PR and why you need to backport that PR:

Not a backport.

Before submitting your pull requests, make sure you followed this checklist:

verify that the PR is really intended for the chosen branch
verify that changes follow CMS Naming, Coding, And Style Rules
verify that the PR passes the basic test procedure suggested in the CMSSW PR instructions

…rocessing of lumisections.

cmsbuild · 2020-10-07T02:26:02Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-10-07T02:32:03Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31689/18834

This PR adds an extra 20KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DQM/EcalCommon/interface/EcalDQMonitor.h modified in PR(s): Enable concurrent lumisection processing in ECAL #30187, Remove legacy dqm #31369
- File DQM/EcalCommon/src/EcalDQMonitor.cc modified in PR(s): Enable concurrent lumisection processing in ECAL #30187, Remove legacy dqm #31369
- File DQM/EcalMonitorTasks/interface/EcalDQMonitorTask.h modified in PR(s): Enable concurrent lumisection processing in ECAL #30187, Remove legacy dqm #31369
- File DQM/EcalMonitorTasks/plugins/EcalDQMonitorTask.cc modified in PR(s): Enable concurrent lumisection processing in ECAL #30187, Remove legacy dqm #31369

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31689/18834/code-format.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31689/18834/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2020-10-07T04:26:23Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-10-07T04:32:38Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31689/18835

This PR adds an extra 20KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DQM/EcalCommon/interface/EcalDQMonitor.h modified in PR(s): Enable concurrent lumisection processing in ECAL #30187, Remove legacy dqm #31369
- File DQM/EcalCommon/src/EcalDQMonitor.cc modified in PR(s): Enable concurrent lumisection processing in ECAL #30187, Remove legacy dqm #31369
- File DQM/EcalMonitorTasks/interface/EcalDQMonitorTask.h modified in PR(s): Enable concurrent lumisection processing in ECAL #30187, Remove legacy dqm #31369
- File DQM/EcalMonitorTasks/plugins/EcalDQMonitorTask.cc modified in PR(s): Enable concurrent lumisection processing in ECAL #30187, Remove legacy dqm #31369

cmsbuild · 2020-10-07T04:33:01Z

A new Pull Request was created by @tanmaymudholkar (Tanmay Mudholkar) for master.

It involves the following packages:

DQM/EcalCommon
DQM/EcalMonitorTasks

@andrius-k, @kmaeshima, @ErnestaP, @cmsbuild, @jfernan2, @fioriNTU can you please review it and eventually sign? Thanks.
@rchatter, @thomreis, @argiro this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

jfernan2 · 2020-10-07T06:53:39Z

please test

cmsbuild · 2020-10-07T06:54:01Z

The tests are being triggered in jenkins.

CMSSW_11_2_X_2020-10-06-2300/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/9765/console Started: 2020/10/07 08:56

cmsbuild · 2020-10-07T08:12:54Z

+1
Tested at: 8930cfd
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3e7a51/9765/summary.html
CMSSW: CMSSW_11_2_X_2020-10-06-2300
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-10-07T08:12:56Z

Comparison job queued.

cmsbuild · 2020-10-07T09:39:53Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3e7a51/9765/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 35
DQMHistoTests: Total histograms compared: 2542225
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2542196
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 34 files compared)
Checked 149 log files, 22 edm output root files, 35 DQM output files

cmsbuild · 2020-10-19T14:15:04Z

The tests are being triggered in jenkins.
Tested with other pull request(s) #31699,#31678

CMSSW_11_2_X_2020-10-19-1100/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/10081/console Started: 2020/10/19 16:17

cmsbuild · 2020-10-19T16:18:03Z

-1

Tested at: 5a3a782

CMSSW: CMSSW_11_2_X_2020-10-19-1100
SCRAM_ARCH: slc7_amd64_gcc820
You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-60df14/10081/summary.html

I found follow errors while testing this PR

Failed tests: RelVals

RelVals:

When I ran the RelVals I found an error in the following workflows:
4.22 step5

runTheMatrix-results/4.22_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC/step5_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC.log

140.53 step3

runTheMatrix-results/140.53_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI/step3_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI.log

8.0 step5

runTheMatrix-results/8.0_BeamHalo+BeamHalo+DIGICOS+RECOCOS+ALCABH+HARVESTCOS/step5_BeamHalo+BeamHalo+DIGICOS+RECOCOS+ALCABH+HARVESTCOS.log

4.53 step4

runTheMatrix-results/4.53_RunPhoton2012B+RunPhoton2012B+HLTD+RECODR1reHLT+HARVESTDR1reHLT/step4_RunPhoton2012B+RunPhoton2012B+HLTD+RECODR1reHLT+HARVESTDR1reHLT.log

9.0 step4

runTheMatrix-results/9.0_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST/step4_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST.log

25.0 step4

runTheMatrix-results/25.0_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT/step4_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT.log

136.731 step4

runTheMatrix-results/136.731_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2/step4_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2.log

25202.0 step4

runTheMatrix-results/25202.0_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25+NANOUP15_PU25/step4_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25+NANOUP15_PU25.log

cmsbuild · 2020-10-19T16:18:10Z

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

tanmaymudholkar · 2020-10-19T16:26:17Z

Hi @jfernan2 , it seems the tests have failed but due to reasons unrelated to this PR; all failures seem to be due to the SiPixelCertification module.

silviodonato · 2020-10-19T16:39:44Z

merge
yes, the errors are related to SiPixelCertification.
However, I still see

The following modules require synchronizing on LuminosityBlock boundaries:
ESIntegrityTask ecalPreshowerIntegrityTask

@tanmaymudholkar could you migrate that missing module?

silviodonato · 2020-10-19T16:52:08Z

merge
yes, the errors are related to SiPixelCertification.
However, I still see
The following modules require synchronizing on LuminosityBlock boundaries:
ESIntegrityTask ecalPreshowerIntegrityTask
@tanmaymudholkar could you migrate that missing module?

#31829 :-D

cmsbuild · 2020-10-19T17:03:30Z

The tests are being triggered in jenkins.
Tested with other pull request(s) #31699,#31678,#31829

CMSSW_11_2_X_2020-10-19-1100/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/10096/console Started: 2020/10/19 19:04

cmsbuild · 2020-10-19T19:11:02Z

+1
Tested at: 5a3a782
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2a187a/10096/summary.html
CMSSW: CMSSW_11_2_X_2020-10-19-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-10-19T19:11:13Z

Comparison job queued.

cmsbuild · 2020-10-19T20:26:26Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2a187a/10096/summary.html

Comparison Summary:

You potentially added 4870 lines to the logs
Reco comparison results: 7 differences found in the comparisons
DQMHistoTests: Total files compared: 35
DQMHistoTests: Total histograms compared: 2543640
DQMHistoTests: Total failures: 28
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2543590
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -773.364 KiB( 34 files compared)
DQMHistoSizes: changed ( 10024.0,... ): -48.369 KiB L1T/L1TStage2CaloLayer1
DQMHistoSizes: changed ( 136.731 ): -47.829 KiB L1T/L1TStage2CaloLayer1
Checked 149 log files, 22 edm output root files, 35 DQM output files

Change DQMOneLumiEDAnalyzer to DQMOneEDAnalyzer to allow concurrent p…

9c1687e

…rocessing of lumisections.

cmsbuild added this to the CMSSW_11_2_X milestone Oct 7, 2020

cmsbuild added code-checks-pending comparison-pending dqm-pending orp-pending pending-signatures tests-pending labels Oct 7, 2020

cmsbuild added code-checks-rejected and removed code-checks-pending labels Oct 7, 2020

Fixed code formatting issues.

8930cfd

cmsbuild added code-checks-pending and removed code-checks-rejected labels Oct 7, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 7, 2020

cmsbuild added tests-started and removed tests-pending labels Oct 7, 2020

cmsbuild added tests-approved and removed tests-started labels Oct 7, 2020

cmsbuild removed the comparison-pending label Oct 7, 2020

cmsbuild added the tests-started label Oct 19, 2020

cmsbuild added comparison-notrun tests-rejected and removed comparison-pending tests-started labels Oct 19, 2020

cmsbuild added orp-approved and removed orp-pending labels Oct 19, 2020

cmsbuild merged commit 3036e5d into cms-sw:master Oct 19, 2020

silviodonato mentioned this pull request Oct 19, 2020

Modules prohibiting concurrent lumis in RelVal workflows #25090

Closed

cmsbuild mentioned this pull request Oct 19, 2020

ES DQM: Enable concurrent processing of lumisections #31829

Merged

cmsbuild added tests-started and removed tests-rejected labels Oct 19, 2020

cmsbuild added comparison-pending tests-approved and removed comparison-notrun tests-started labels Oct 19, 2020

cmsbuild added comparison-available and removed comparison-pending labels Oct 19, 2020

jfernan2 mentioned this pull request Oct 20, 2020

Remove legacy dqm #31369

Closed

silviodonato mentioned this pull request Nov 26, 2020

DQMOnlineClient-beam_dqm_sourceclient error #31896

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix to allow concurrent processing of lumisections with ECAL DQM #31689

Fix to allow concurrent processing of lumisections with ECAL DQM #31689

tanmaymudholkar commented Oct 7, 2020

cmsbuild commented Oct 7, 2020

cmsbuild commented Oct 7, 2020

cmsbuild commented Oct 7, 2020

cmsbuild commented Oct 7, 2020

cmsbuild commented Oct 7, 2020

jfernan2 commented Oct 7, 2020

cmsbuild commented Oct 7, 2020 •

edited

cmsbuild commented Oct 7, 2020

cmsbuild commented Oct 7, 2020

cmsbuild commented Oct 7, 2020

cmsbuild commented Oct 19, 2020 •

edited

cmsbuild commented Oct 19, 2020

cmsbuild commented Oct 19, 2020

tanmaymudholkar commented Oct 19, 2020

silviodonato commented Oct 19, 2020

silviodonato commented Oct 19, 2020

cmsbuild commented Oct 19, 2020 •

edited

cmsbuild commented Oct 19, 2020

cmsbuild commented Oct 19, 2020

cmsbuild commented Oct 19, 2020

Fix to allow concurrent processing of lumisections with ECAL DQM #31689

Fix to allow concurrent processing of lumisections with ECAL DQM #31689

Conversation

tanmaymudholkar commented Oct 7, 2020

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented Oct 7, 2020

cmsbuild commented Oct 7, 2020

cmsbuild commented Oct 7, 2020

cmsbuild commented Oct 7, 2020

cmsbuild commented Oct 7, 2020

jfernan2 commented Oct 7, 2020

cmsbuild commented Oct 7, 2020 • edited

cmsbuild commented Oct 7, 2020

cmsbuild commented Oct 7, 2020

cmsbuild commented Oct 7, 2020

cmsbuild commented Oct 19, 2020 • edited

cmsbuild commented Oct 19, 2020

cmsbuild commented Oct 19, 2020

tanmaymudholkar commented Oct 19, 2020

silviodonato commented Oct 19, 2020

silviodonato commented Oct 19, 2020

cmsbuild commented Oct 19, 2020 • edited

cmsbuild commented Oct 19, 2020

cmsbuild commented Oct 19, 2020

cmsbuild commented Oct 19, 2020

cmsbuild commented Oct 7, 2020 •

edited

cmsbuild commented Oct 19, 2020 •

edited

cmsbuild commented Oct 19, 2020 •

edited