[DQM] Add dqmPerLSsaving modifier #34487

jfernan2 · 2021-07-14T17:22:35Z

PR description:

After the creation of pseudo data-Tier nanoDQMIO in:
#34220
and
#34301
It was suggested by @makortel (Thank you!!!) to use a Modifier instead ( #34301 (comment))

This PR introduces the dqmPerLSsaving Modifier and files which need to make use of it

So now in addition to
--customise_commands=process.DQMStore.saveByLumi = cms.untracked.bool(True)
one must set
--procModifiers=dqmPerLSsaving
at cmsDriver.py config

https://twiki.cern.ch/twiki/bin/view/CMS/PerLsDQMIO#nanoDQMIO_data_Tier

No changes expected in this PR since perLSsaving reamins switched off by default.

PR validation:

Tested locally with wfs 1000.0,4.22,4.17,18.0,136.858,4.37,136.8862,136.727,136.7801 with an equivalent config to this:

cmsDriver.py step2 -s DQM--datatier DQMIO --eventcontent DQM --data --customise_commands="process.DQMStore.saveByLumi = cms.untracked.bool(True)" --procModifiers=dqmPerLSsaving

cmsbuild · 2021-07-14T17:27:52Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34487/23952

This PR adds an extra 32KB to repository

cmsbuild · 2021-07-14T17:28:16Z

A new Pull Request was created by @jfernan2 for master.

It involves the following packages:

Configuration/ProcessModifiers (operations)
DQM/BeamMonitor (dqm)
DQM/CTPPS (dqm)
DQM/EcalPreshowerMonitorModule (dqm)
DQM/L1TMonitor (dqm)
DQM/SiPixelCommon (dqm)
DQMOffline/Configuration (dqm)
DQMOffline/Trigger (dqm)

@andrius-k, @kmaeshima, @ErnestaP, @ahmad3213, @cmsbuild, @silviodonato, @jfernan2, @qliphy, @davidlange6, @fabiocos, @rvenditti can you please review it and eventually sign? Thanks.
@argiro, @Martin-Grunewald, @fioriNTU, @thomreis, @threus, @cericeci, @hdelanno, @makortel, @jhgoh, @missirol, @sscruz, @simonepigazzini, @trocino, @rociovilar, @sroychow, @fabferro, @jandrea, @idebruyn, @mtosi, @fabiocos, @arossi83, @HuguesBrun, @rchatter, @jan-kaspar, @Fedespring, @calderona this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy, @perrotta you are the release manager for this.

cms-bot commands are listed here

jfernan2 · 2021-07-14T17:33:15Z

please test

makortel · 2021-07-14T17:47:43Z

Configuration/ProcessModifiers/python/perLSsaving_cff.py

+import FWCore.ParameterSet.Config as cms
+
+# This modifier sets the perLS saving in process.DQMStore.saveByLumi for nanoDQMIO
+perLSsaving = cms.Modifier()


How about e.g. dqmPerLSSaving to make it clear that the Modifier is related to DQM?

Done, thanks for the suggestion!

cmsbuild · 2021-07-14T18:10:34Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34487/23956

This PR adds an extra 32KB to repository
Found files with invalid states:
- Configuration/ProcessModifiers/python/perLSsaving_cff.py:
  - Added: 4a83346
  - Deleted: 857fdc9

cmsbuild · 2021-07-14T18:10:59Z

Pull request #34487 was updated. @andrius-k, @kmaeshima, @ErnestaP, @ahmad3213, @cmsbuild, @silviodonato, @jfernan2, @qliphy, @davidlange6, @fabiocos, @rvenditti can you please check and sign again.

jfernan2 · 2021-07-14T18:15:53Z

please test

cmsbuild · 2021-07-14T19:53:21Z

-1

Failed Tests: RelVals RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c4a8c6/16822/summary.html
COMMIT: 857fdc9
CMSSW: CMSSW_12_0_X_2021-07-13-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/34487/16822/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

140.53140.53_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI/step2_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI.log

RelVals-INPUT

140.52140.52_RunHI2010+RunHI2010+RECOHID10+RECOHIR10D11+HARVESTDHI/step2_RunHI2010+RunHI2010+RECOHID10+RECOHIR10D11+HARVESTDHI.log
140.53140.53_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI/step2_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI.log
1004.01004.0_RunHI2011+RunHI2011+TIER0EXPHI+ALCAEXPHI+ALCAHARVD1HI+ALCAHARVD2HI+ALCAHARVD3HI+ALCAHARVD5HI/step2_RunHI2011+RunHI2011+TIER0EXPHI+ALCAEXPHI+ALCAHARVD1HI+ALCAHARVD2HI+ALCAHARVD3HI+ALCAHARVD5HI.log

jfernan2 · 2021-07-15T08:48:02Z

please test

cmsbuild · 2021-07-15T16:43:03Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c4a8c6/16842/summary.html
COMMIT: 99b63ed
CMSSW: CMSSW_12_0_X_2021-07-14-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/34487/16842/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 38
DQMHistoTests: Total histograms compared: 2786281
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2786258
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -3.15 KiB( 37 files compared)
DQMHistoSizes: changed ( 1000.0,... ): -0.150 KiB EcalPreshower/ESIntegrityTask
Checked 160 log files, 37 edm output root files, 38 DQM output files
TriggerResults: no differences found

ES Good Channel Fraction ME was originally ON in perRun configuration

jfernan2 · 2021-07-15T17:15:05Z

please test

cmsbuild · 2021-07-15T17:21:31Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34487/23991

This PR adds an extra 12KB to repository
Found files with invalid states:
- Configuration/ProcessModifiers/python/perLSsaving_cff.py:
  - Added: 4a83346
  - Deleted: 857fdc9

cmsbuild · 2021-07-15T17:22:02Z

Pull request #34487 was updated. @andrius-k, @kmaeshima, @ErnestaP, @ahmad3213, @silviodonato, @jfernan2, @qliphy, @davidlange6, @fabiocos, @rvenditti can you please check and sign again.

cmsbuild · 2021-07-15T20:33:53Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c4a8c6/16859/summary.html
COMMIT: 3fe8ed2
CMSSW: CMSSW_12_0_X_2021-07-15-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/34487/16859/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 38
DQMHistoTests: Total histograms compared: 2786302
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2786279
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 37 files compared)
Checked 160 log files, 37 edm output root files, 38 DQM output files
TriggerResults: no differences found

jfernan2 · 2021-07-16T07:52:06Z

+1

qliphy · 2021-07-18T09:28:12Z

+1

cmsbuild · 2021-07-18T09:28:37Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.

Add perLSsaving modifier

4a83346

cmsbuild added this to the CMSSW_12_0_X milestone Jul 14, 2021

cmsbuild added code-checks-pending dqm-pending operations-pending orp-pending pending-signatures tests-pending labels Jul 14, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 14, 2021

cmsbuild added tests-started and removed tests-pending labels Jul 14, 2021

makortel reviewed Jul 14, 2021

View reviewed changes

Renamed to dqmPerLSsaving

857fdc9

cmsbuild added code-checks-pending tests-pending and removed code-checks-approved tests-started labels Jul 14, 2021

jfernan2 changed the title ~~[DQM] Add perLSsaving modifier~~ [DQM] Add dqmPerLSsaving modifier Jul 14, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 14, 2021

cmsbuild added tests-started and removed tests-pending labels Jul 14, 2021

cmsbuild added tests-approved and removed tests-started labels Jul 15, 2021

Restore original configuration for ECAL

3fe8ed2

ES Good Channel Fraction ME was originally ON in perRun configuration

cmsbuild added code-checks-pending tests-started and removed code-checks-approved tests-approved labels Jul 15, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Jul 15, 2021

cmsbuild added tests-approved and removed tests-started labels Jul 15, 2021

cmsbuild added dqm-approved and removed dqm-pending labels Jul 16, 2021

cmsbuild added fully-signed and removed operations-pending orp-pending pending-signatures labels Jul 18, 2021

cmsbuild added operations-approved orp-approved labels Jul 18, 2021

cmsbuild merged commit e5fa302 into cms-sw:master Jul 18, 2021

cmsbuild mentioned this pull request Jul 19, 2021

[BuildRules] cleanup PYTHON27; use python3 as default cms-sw/cmsdist#7146

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DQM] Add dqmPerLSsaving modifier #34487

[DQM] Add dqmPerLSsaving modifier #34487

jfernan2 commented Jul 14, 2021 •

edited

cmsbuild commented Jul 14, 2021

cmsbuild commented Jul 14, 2021

jfernan2 commented Jul 14, 2021

makortel Jul 14, 2021

jfernan2 Jul 14, 2021

cmsbuild commented Jul 14, 2021

cmsbuild commented Jul 14, 2021

jfernan2 commented Jul 14, 2021

cmsbuild commented Jul 14, 2021

jfernan2 commented Jul 15, 2021

cmsbuild commented Jul 15, 2021

jfernan2 commented Jul 15, 2021

cmsbuild commented Jul 15, 2021

cmsbuild commented Jul 15, 2021

cmsbuild commented Jul 15, 2021

jfernan2 commented Jul 16, 2021

qliphy commented Jul 18, 2021

cmsbuild commented Jul 18, 2021

[DQM] Add dqmPerLSsaving modifier #34487

[DQM] Add dqmPerLSsaving modifier #34487

Conversation

jfernan2 commented Jul 14, 2021 • edited

PR description:

PR validation:

cmsbuild commented Jul 14, 2021

cmsbuild commented Jul 14, 2021

jfernan2 commented Jul 14, 2021

makortel Jul 14, 2021

Choose a reason for hiding this comment

jfernan2 Jul 14, 2021

Choose a reason for hiding this comment

cmsbuild commented Jul 14, 2021

cmsbuild commented Jul 14, 2021

jfernan2 commented Jul 14, 2021

cmsbuild commented Jul 14, 2021

RelVals

RelVals-INPUT

jfernan2 commented Jul 15, 2021

cmsbuild commented Jul 15, 2021

Comparison Summary

jfernan2 commented Jul 15, 2021

cmsbuild commented Jul 15, 2021

cmsbuild commented Jul 15, 2021

cmsbuild commented Jul 15, 2021

Comparison Summary

jfernan2 commented Jul 16, 2021

qliphy commented Jul 18, 2021

cmsbuild commented Jul 18, 2021

jfernan2 commented Jul 14, 2021 •

edited