[12.4.X] Move `DiMuonMassBiasClient` to a regular harvesting path #39248

mmusich · 2022-08-30T07:30:38Z

backport of #39217

PR description:

Inspired by the discussion at #39180 (comment) I move the DiMuonMassBiasClient to a regular harvesting path, actually into the DQMHarvestMuon sequence which as per autoDQM is run in the @muon sequence

cmssw/DQMOffline/Configuration/python/autoDQM.py

Lines 96 to 99 in d9d4aec

    
           'muon': ['DQMOfflineMuon', 
        
                    'PostDQMOffline', 
        
                    'DQMHarvestMuon+DQMCertMuon'],

that would be run at Tier-0 for the Muon PD (cf also: https://github.com/dmwm/T0/blob/0f4f9d27c5366da57c456c4ae01da81c75524595/etc/ProdOfflineConfiguration.py#L698)

I profit of this PR to remove remove useless DMQToken produce and consumes as pointed out in #39180 (comment)

PR validation:

Run successfully several workflows that were failing before issue #39180 was solved in threading mode:

runTheMatrix.py -nel 1000.0,10042.0,10842.0 -j 8 -t 4

EDIT:
unfortunately I could not find a RelVal workflow in the matrix that mimics Tier0 in this respect (i.e. that runs ALCA:TkAlDiMuonAndVertex in the same step as DQM and then a regular harvesting, so I've created one for this purpose: 1002.4.

after that I've tested with :

runTheMatrix.py -l 1002.4 -t 4 -j 8 --ibeos

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

backport of #39217 needed for data-taking purposes.

cmsbuild · 2022-08-30T07:31:02Z

A new Pull Request was created by @mmusich (Marco Musich) for CMSSW_12_4_X.

It involves the following packages:

Configuration/PyReleaseValidation (pdmv, upgrade)
DQMOffline/Alignment (dqm)
DQMOffline/Configuration (dqm)

@jordan-martins, @bbilin, @emanueleusai, @ahmad3213, @cmsbuild, @AdrianoDee, @srimanob, @jfernan2, @kskovpen, @pmandrik, @micsucmed, @rvenditti can you please review it and eventually sign? Thanks.
@makortel, @kpedro88, @fabiocos, @Martin-Grunewald, @missirol, @trtomei, @threus, @beaucero, @slomeo, @rociovilar this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

Backported from Move DiMuonMassBiasClient to a regular harvesting path #39217

mmusich · 2022-08-30T07:31:02Z

test parameters:

workflow = 1002.4
relvals_opt = --what cleanedupgrade,standard,highstats,pileup,generator,extendedgen,production,ged,machine,premix

mmusich · 2022-08-30T07:31:10Z

@cmsbuild, please test

cmsbuild · 2022-08-30T07:57:04Z

-1

Failed Tests: UnitTests RelVals RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d57ece/27186/summary.html
COMMIT: caa6b22
CMSSW: CMSSW_12_4_X_2022-08-29-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39248/27186/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestDQMOfflineConfiguration_0 had ERRORS
---> test TestDQMOfflineConfiguration_110 had ERRORS
---> test TestDQMOfflineConfiguration_180 had ERRORS
---> test TestDQMOfflineConfiguration_160 had ERRORS
and more ...

RelVals

RelVals-INPUT

cmsbuild · 2022-08-30T09:31:37Z

Pull request #39248 was updated. @jordan-martins, @bbilin, @emanueleusai, @ahmad3213, @cmsbuild, @AdrianoDee, @srimanob, @jfernan2, @kskovpen, @pmandrik, @micsucmed, @rvenditti can you please check and sign again.

mmusich · 2022-08-30T09:38:10Z

@cmsbuild, please test

cmsbuild · 2022-08-30T13:52:12Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d57ece/27189/summary.html
COMMIT: 1ec6145
CMSSW: CMSSW_12_4_X_2022-08-29-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39248/27189/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-d57ece/1002.4_RunDoubleMuon2022B+RunDoubleMuon2022B+TIER0PROMPTRUN3+HARVESTPROMPTRUN3

Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3677280
DQMHistoTests: Total failures: 13
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3677244
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.004 KiB( 49 files compared)
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

emanueleusai · 2022-08-31T05:20:17Z

+1

srimanob · 2022-08-31T13:49:09Z

+Upgrade

Backport PR.

kskovpen · 2022-08-31T13:52:56Z

+pdmv

cmsbuild · 2022-08-31T13:53:20Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_4_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_6_X is complete. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

mmusich · 2022-09-02T06:38:51Z

@cms-sw/orp-l2 anything that prevents the integration of this PR in 12.4.X?
It would be a desirable feature for when we restart taking data after the LHC cryo stop.

perrotta · 2022-09-02T07:01:57Z

Configuration/PyReleaseValidation/python/relval_steps.py

@@ -481,6 +481,8 @@
 Run2022B={355130: [[1, 40]]}
 steps['RunSingleMuon2022B']={'INPUT':InputInfo(dataSet='/SingleMuon/Run2022B-v1/RAW',label='2022B',events=100000,location='STD', ls=Run2022B)}
 steps['RunZeroBias2022B']={'INPUT':InputInfo(dataSet='/ZeroBias/Run2022B-v1/RAW',label='2022B',events=100000,location='STD', ls=Run2022B)}
+Run2022B={355680: [[21, 24]]}


This was not in #39217: can you add a comment?

well, it's needed because the PR (#38733) that added them was not backported.

perrotta · 2022-09-02T07:03:26Z

Configuration/PyReleaseValidation/python/relval_steps.py

@@ -3167,6 +3169,18 @@ def gen2021HiMix(fragment,howMuch):
                                    '--era' : 'Run2_2018_pp_on_AA',
                                    '--filetype':'DQM',
                                    '-n':100},hiDefaults2018_ppReco])
+


Steps 'HARVESTDRUN3' and 'HARVESTRUN3' were not in #39217: can you add a comment?

well, it's needed because the PR (#38733) that added them was not backported.

DQMOffline/Alignment/python/ALCARECOTkAlDQM_cff.py

perrotta · 2022-09-02T09:06:56Z

+1

Backport of Move DiMuonMassBiasClient to a regular harvesting path #39217 and part of Add Run3 wfs (NanoV10) #38733

mmusich added 2 commits August 30, 2022 09:25

remove useless DMQToken produce and consumes

99b7619

move DiMuonMassBiasClient to be run in the '@Muon' harvesting sequence

6515a58

cmsbuild added this to the CMSSW_12_4_X milestone Aug 30, 2022

cmsbuild added backport-ok dqm-pending orp-pending pdmv-pending pending-signatures tests-pending upgrade-pending labels Aug 30, 2022

cmsbuild added tests-started and removed tests-pending labels Aug 30, 2022

cmsbuild added tests-rejected and removed tests-started labels Aug 30, 2022

include new Production workflow, including DQM harvesting

1ec6145

mmusich force-pushed the moveDiMuonMassBiasClientToHarvestingPath_12_4_X branch from caa6b22 to 1ec6145 Compare August 30, 2022 09:31

cmsbuild added tests-pending and removed tests-rejected labels Aug 30, 2022

cmsbuild added tests-started and removed tests-pending labels Aug 30, 2022

cmsbuild added tests-approved and removed tests-started labels Aug 30, 2022

cmsbuild added dqm-approved and removed dqm-pending labels Aug 31, 2022

cmsbuild added upgrade-approved and removed upgrade-pending labels Aug 31, 2022

cmsbuild added fully-signed pdmv-approved and removed pending-signatures pdmv-pending labels Aug 31, 2022

perrotta reviewed Sep 2, 2022

View reviewed changes

cmsbuild added orp-approved and removed orp-pending labels Sep 2, 2022

cmsbuild merged commit 04a6d57 into cms-sw:CMSSW_12_4_X Sep 2, 2022

mmusich deleted the moveDiMuonMassBiasClientToHarvestingPath_12_4_X branch September 2, 2022 09:08

mmusich mentioned this pull request Sep 7, 2022

[12.4.X] Add Tracker Alignment di-muon monitoring to other resonances #39332

Merged

makortel mentioned this pull request Dec 22, 2022

Simplify Alpaka ESProducer model by making the data copies to device implicit #40403

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[12.4.X] Move `DiMuonMassBiasClient` to a regular harvesting path #39248

[12.4.X] Move `DiMuonMassBiasClient` to a regular harvesting path #39248

mmusich commented Aug 30, 2022

cmsbuild commented Aug 30, 2022 •

edited

mmusich commented Aug 30, 2022

mmusich commented Aug 30, 2022

cmsbuild commented Aug 30, 2022

cmsbuild commented Aug 30, 2022

mmusich commented Aug 30, 2022

cmsbuild commented Aug 30, 2022

emanueleusai commented Aug 31, 2022

srimanob commented Aug 31, 2022

kskovpen commented Aug 31, 2022

cmsbuild commented Aug 31, 2022

mmusich commented Sep 2, 2022

perrotta Sep 2, 2022

mmusich Sep 2, 2022 •

edited

perrotta Sep 2, 2022

mmusich Sep 2, 2022 •

edited

perrotta commented Sep 2, 2022

	'muon': ['DQMOfflineMuon',
	'PostDQMOffline',
	'DQMHarvestMuon+DQMCertMuon'],

[12.4.X] Move DiMuonMassBiasClient to a regular harvesting path #39248

[12.4.X] Move DiMuonMassBiasClient to a regular harvesting path #39248

Conversation

mmusich commented Aug 30, 2022

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

cmsbuild commented Aug 30, 2022 • edited

mmusich commented Aug 30, 2022

mmusich commented Aug 30, 2022

cmsbuild commented Aug 30, 2022

Unit Tests

RelVals

RelVals-INPUT

cmsbuild commented Aug 30, 2022

mmusich commented Aug 30, 2022

cmsbuild commented Aug 30, 2022

Comparison Summary

emanueleusai commented Aug 31, 2022

srimanob commented Aug 31, 2022

kskovpen commented Aug 31, 2022

cmsbuild commented Aug 31, 2022

mmusich commented Sep 2, 2022

perrotta Sep 2, 2022

Choose a reason for hiding this comment

mmusich Sep 2, 2022 • edited

Choose a reason for hiding this comment

perrotta Sep 2, 2022

Choose a reason for hiding this comment

mmusich Sep 2, 2022 • edited

Choose a reason for hiding this comment

perrotta commented Sep 2, 2022

[12.4.X] Move `DiMuonMassBiasClient` to a regular harvesting path #39248

[12.4.X] Move `DiMuonMassBiasClient` to a regular harvesting path #39248

cmsbuild commented Aug 30, 2022 •

edited

mmusich Sep 2, 2022 •

edited

mmusich Sep 2, 2022 •

edited