Adding general track to ALCARECO for muonalign #15221

lpernie · 2016-07-18T18:18:11Z

Hi @mmusich,
This the the new PR on the addition of general tracks to the ALCARECO stream for muon alignment you requested.

cmsbuild · 2016-07-18T18:18:34Z

A new Pull Request was created by @lpernie (Luca) for CMSSW_8_1_X.

It involves the following packages:

Alignment/CommonAlignmentProducer
Configuration/PyReleaseValidation
Configuration/StandardSequences

@ghellwig, @franzoni, @cerminar, @fabozzi, @cmsbuild, @srimanob, @mmusich, @hengne, @davidlange6 can you please review it and eventually sign? Thanks.
@ghellwig, @pakhotin, @makortel, @GiacomoSguazzoni, @tocheng, @VinInn, @Martin-Grunewald, @rovere, @tlampen, @mschrode, @mmusich, @dgulhan this is something you requested to watch as well.
@slava77, @smuzaffar you are the release manager for this.

cms-bot commands are list here #13028

lpernie · 2016-07-18T18:19:10Z

Should I rebase it?
git fetch official-cmssw # get the latest updates from the official repository
git checkout newALCARECOstream # checkout your branch (if you have not done yet)
git rebase official-cmssw/CMSSW_8_1_X # rebase your changes on top of the official branch

mmusich · 2016-07-19T07:00:02Z

@lpernie, yes, this needs a rebase, but be careful that in your current version there are regressions in Configuration/StandardSequences/python/AlCaRecoStreams_cff.py, namely you shouldn't remove ALCARECOStreamHcalCalIsolatedBunchFilter

cmsbuild · 2016-07-19T15:30:39Z

Pull request #15221 was updated. @ghellwig, @franzoni, @cerminar, @fabozzi, @cmsbuild, @srimanob, @mmusich, @hengne, @davidlange6 can you please check and sign again.

lpernie · 2016-07-19T15:30:55Z

Done!
(I saw that Configuration/StandardSequences/python/AlCaRecoStreams_cff.py had no conflicts)

mmusich · 2016-07-19T15:40:33Z

@lpernie,
as far as I can see the regressions are still there.
Configuration/StandardSequences/python/AlCaRecoStreams_cff.py, namely, you are still removing:

 -ALCARECOStreamHcalCalIsolatedBunchFilter = cms.FilteredStream(
-   responsible = 'Sunanda Banerjee',
-   name = 'HcalCalIsolatedBunchFilter',
-   paths  = (pathALCARECOHcalCalIsolatedBunchFilter),
-   content = OutALCARECOHcalCalIsolatedBunchFilter.outputCommands,
-   selectEvents = OutALCARECOHcalCalIsolatedBunchFilter.SelectEvents,
-   dataTier = cms.untracked.string('ALCARECO')
-   )
-

can you please get it back?
Thanks,

Marco

cmsbuild · 2016-07-19T15:47:00Z

Pull request #15221 was updated. @ghellwig, @franzoni, @cerminar, @fabozzi, @cmsbuild, @srimanob, @mmusich, @hengne, @davidlange6 can you please check and sign again.

mmusich · 2016-07-19T15:47:33Z

Configuration/StandardSequences/python/AlCaRecoStreams_cff.py

@@ -75,9 +75,6 @@
 from Calibration.HcalAlCaRecoProducers.ALCARECOHcalCalNoise_cff import *
 #HCAL calibration iterative PhiSym
 from Calibration.HcalAlCaRecoProducers.ALCARECOHcalCalIterativePhiSym_cff import *
-# HCAL isolated bunch
-from Calibration.HcalAlCaRecoProducers.ALCARECOHcalCalIsolatedBunchFilter_cff import *
-


@lpernie also this needs to get back

lpernie · 2016-07-19T15:47:36Z

Ok, I just added it back. I hope I did correctly.

mmusich · 2016-07-19T15:47:50Z

Configuration/StandardSequences/python/AlCaRecoStreams_cff.py

@@ -170,10 +167,12 @@
 pathALCARECOHcalCalIsoTrkFilter = cms.Path(seqALCARECOHcalCalIsoTrkFilter)
 pathALCARECOHcalCalNoise = cms.Path(seqALCARECOHcalCalNoise)
 pathALCARECOHcalCalIterativePhiSym = cms.Path(seqALCARECOHcalCalIterativePhiSym*ALCARECOHcalCalPhisymDQM)
-pathALCARECOHcalCalIsolatedBunchFilter = cms.Path(seqALCARECOHcalCalIsolatedBunchFilter)


@lpernie this one as well!

cmsbuild · 2016-07-19T15:53:17Z

Pull request #15221 was updated. @ghellwig, @franzoni, @cerminar, @fabozzi, @cmsbuild, @srimanob, @mmusich, @hengne, @davidlange6 can you please check and sign again.

mmusich · 2016-07-19T15:53:54Z

thanks! should be in a testable shape now

mmusich · 2016-07-19T15:54:01Z

@cmsbuild please test

cmsbuild · 2016-07-19T15:54:16Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/14178/console

cmsbuild · 2016-07-19T20:49:19Z

+1
Tested at: b6e04f4
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-15221/14178/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
a54470e
1482302
ce11780
ef9979b
eeae522
cf672ff
22db4d8
9fba656
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-15221/14178/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-15221/14178/git-merge-result

cmsbuild · 2016-07-19T21:56:44Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-15221/14178/summary.html

cmsbuild · 2016-08-16T21:38:59Z

+1
Tested at: c03b999
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-15221/14555/summary.html

cmsbuild · 2016-08-16T22:35:06Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-15221/14555/summary.html

ghellwig · 2016-08-22T14:40:55Z

Hi @lpernie, @mmusich, @franzoni,

I have run an additional check regarding the change of the ALCARECO troughput at commit c03b999:

I modified workflow 1000, to take SingleMuon and DoubleMuon datasets as input and compared the file sizes of the produced output of the MuAlCalIsolatedMu and the MuAlZMuMu streams to get a measure of the expected change in ALCARECO disk usage:

Used input events: 500

Selected events (before #15221):

SingleMuon:
- MuAlZMuMu.root: 18
- MuAlCalIsolatedMu.root: 490
DoubleMuon:
- MuAlZMuMu.root: 30
- MuAlCalIsolatedMu.root: 478

Selected events (after #15221):

SingleMuon:
- MuAlZMuMu.root: 18
- MuAlCalIsolatedMu.root: 480
DoubleMuon:
- MuAlZMuMu.root: 31
- MuAlCalIsolatedMu.root: 403

Event size (before #15221):

SingleMuon:
- MuAlZMuMu.root: 162295 bytes/event
- MuAlCalIsolatedMu.root: 38866 bytes/event
DoubleMuon:
- MuAlZMuMu.root: 114372 bytes/event
- MuAlCalIsolatedMu.root: 41461 bytes/event

Event size (after #15221):

SingleMuon:
- MuAlZMuMu.root: 168392 bytes/event
- MuAlCalIsolatedMu.root: 42145 bytes/event
DoubleMuon:
- MuAlZMuMu.root: 118663 bytes/event
- MuAlCalIsolatedMu.root: 45909 bytes/event

relative throughput change = (file_size(after) - file_size(before))/file_size(before)

SingleMuon:
- MuAlZMuMu.root: (3031048 - 2921303) / 2921303 = 3.8%
- MuAlCalIsolatedMu.root: (20229479 - 19044339) / 19044339 = 6.2%
DoubleMuon:
- MuAlZMuMu.root: (3678552 - 3431168) / 3431168 = 7.2%
- MuAlCalIsolatedMu.root: (18501503 - 19818406) / 19818406 = -6.6%

Hope this helps to judge this PR in terms of ALCARECO throughput.

-Gregor

ghellwig · 2016-08-22T14:46:51Z

To be more specific: I used the following datasets:

franzoni · 2016-08-22T19:02:52Z

+1
with further removal of access to RECO some level of extra alcareco resources is inevitable; this will also need to reflect on the overall disk provision to alcs/db at tier 2's in general and at CAF

thanks @ghellwig for the study

mmusich · 2016-08-22T20:42:40Z

@fabozzi @srimanob @hengne, can you please cross-check and sign? Thanks
M.

mmusich · 2016-08-22T20:47:29Z

@lpernie in view of a possible reprocessing of 2016 data can you please backport to 80x as well?

lpernie · 2016-08-22T20:48:52Z

Sure.
This mean should I make a PR with the same changes in 80?

mmusich · 2016-08-22T20:50:47Z

Indeed. Thanks!

fabozzi · 2016-08-23T11:10:40Z

+1

cmsbuild · 2016-08-23T11:10:58Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_8_1_X IBs (tests are also fine). This pull request requires discussion in the ORP meeting before it's merged. @slava77, @davidlange6, @smuzaffar

davidlange6 · 2016-08-23T13:07:40Z

+1

mmusich · 2016-09-05T15:00:11Z

Configuration/PyReleaseValidation/python/relval_steps.py

-steps['RECOCOS']=merge([{'-s':'RAW2DIGI,L1Reco,RECO,ALCA:MuAlCalIsolatedMu,DQM','--scenario':'cosmics'},stCond,step3Defaults])
-steps['RECOHAL']=merge([{'-s':'RAW2DIGI,L1Reco,RECO,ALCA:MuAlCalIsolatedMu,DQM','--scenario':'cosmics'},step3Up2015Hal])
+steps['RECOCOS']=merge([{'-s':'RAW2DIGI,L1Reco,RECO,ALCA:MuAlGlobalCosmics,DQM','--scenario':'cosmics'},stCond,step3Defaults])
+steps['RECOHAL']=merge([{'-s':'RAW2DIGI,L1Reco,RECO,ALCA:MuAlBeamHalo+MuAlBeamHaloOverlaps,DQM','--scenario':'cosmics'},step3Up2015Hal])
 steps['RECOCOS_UP15']=merge([{'--conditions':'auto:run2_mc_cosmics','-s':'RAW2DIGI,L1Reco,RECO,ALCA:MuAlCalIsolatedMu,DQM','--scenario':'cosmics'},step3Up2015Hal])


@lpernie this line here shall be changed as done above:
https://github.com/cms-sw/cmssw/pull/15221/files#diff-819af2ec677272fddf1fef50e788a88bR1195"
this currently breaks wf 1307.0 @MengqingWu

@mmusich Do you mean like RECOHAL?
steps['RECOCOS']=merge([{'-s':'RAW2DIGI,L1Reco,RECO,ALCA:MuAlBeamHalo+MuAlBeamHaloOverlaps,DQM','--scenario':'cosmics'},stCond,step3Defaults])
Or like it was in the past (ALCA:MuAlCalIsolatedMu)?

@lpernie no I mean to change:

steps['RECOCOS_UP15']=merge([{'--conditions':'auto:run2_mc_cosmics','-s':'RAW2DIGI,L1Reco,RECO,ALCA:MuAlCalIsolatedMu,DQM','--scenario':'cosmics'},step3Up2015Hal])

to:

steps['RECOCOS_UP15']=merge([{'--conditions':'auto:run2_mc_cosmics','-s':'RAW2DIGI,L1Reco,RECO,ALCA:MuAlGlobalCosmics,DQM','--scenario':'cosmics'},step3Up2015Hal])

cmsbuild added this to the Next CMSSW_8_1_X milestone Jul 18, 2016

cmsbuild added alca-pending operations-pending pending-signatures tests-pending orp-pending comparison-pending pdmv-pending labels Jul 18, 2016

mmusich mentioned this pull request Jul 19, 2016

Adding (a lite version of) General Tracks to ALCARECO stream for Muon Alignment #15179

Closed

lpernie force-pushed the AddgeneralTrackALCARECOmuon branch from f03ac4a to a047260 Compare July 19, 2016 15:30

mmusich reviewed Jul 19, 2016
View reviewed changes

cmsbuild added tests-started and removed tests-pending labels Jul 19, 2016

cmsbuild added tests-approved and removed tests-started labels Jul 19, 2016

cmsbuild added the tests-started label Aug 16, 2016

cmsbuild added tests-approved and removed tests-started labels Aug 16, 2016

cmsbuild added comparison-available and removed comparison-pending labels Aug 16, 2016

cmsbuild added alca-approved operations-approved and removed alca-pending operations-pending labels Aug 22, 2016

lpernie mentioned this pull request Aug 22, 2016

Adding general track to ALCARECO for muonalign #15556

Merged

cmsbuild added fully-signed pdmv-approved and removed pdmv-pending pending-signatures labels Aug 23, 2016

cmsbuild added orp-approved and removed orp-pending labels Aug 23, 2016

cmsbuild merged commit 90a07cb into cms-sw:CMSSW_8_1_X Aug 23, 2016

mmusich reviewed Sep 5, 2016
View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding general track to ALCARECO for muonalign #15221

Adding general track to ALCARECO for muonalign #15221

lpernie commented Jul 18, 2016

cmsbuild commented Jul 18, 2016

lpernie commented Jul 18, 2016

mmusich commented Jul 19, 2016

cmsbuild commented Jul 19, 2016

lpernie commented Jul 19, 2016

mmusich commented Jul 19, 2016 •

edited

cmsbuild commented Jul 19, 2016

mmusich Jul 19, 2016

lpernie commented Jul 19, 2016

mmusich Jul 19, 2016

cmsbuild commented Jul 19, 2016

mmusich commented Jul 19, 2016

mmusich commented Jul 19, 2016

cmsbuild commented Jul 19, 2016 •

edited

cmsbuild commented Jul 19, 2016

cmsbuild commented Jul 19, 2016

cmsbuild commented Aug 16, 2016

cmsbuild commented Aug 16, 2016

ghellwig commented Aug 22, 2016 •

edited

ghellwig commented Aug 22, 2016

franzoni commented Aug 22, 2016

mmusich commented Aug 22, 2016

mmusich commented Aug 22, 2016

lpernie commented Aug 22, 2016

mmusich commented Aug 22, 2016

fabozzi commented Aug 23, 2016

cmsbuild commented Aug 23, 2016

davidlange6 commented Aug 23, 2016

mmusich Sep 5, 2016

lpernie Sep 5, 2016

mmusich Sep 6, 2016

Adding general track to ALCARECO for muonalign #15221

Adding general track to ALCARECO for muonalign #15221

Conversation

lpernie commented Jul 18, 2016

cmsbuild commented Jul 18, 2016

lpernie commented Jul 18, 2016

mmusich commented Jul 19, 2016

cmsbuild commented Jul 19, 2016

lpernie commented Jul 19, 2016

mmusich commented Jul 19, 2016 • edited

cmsbuild commented Jul 19, 2016

mmusich Jul 19, 2016

Choose a reason for hiding this comment

lpernie commented Jul 19, 2016

mmusich Jul 19, 2016

Choose a reason for hiding this comment

cmsbuild commented Jul 19, 2016

mmusich commented Jul 19, 2016

mmusich commented Jul 19, 2016

cmsbuild commented Jul 19, 2016 • edited

cmsbuild commented Jul 19, 2016

cmsbuild commented Jul 19, 2016

cmsbuild commented Aug 16, 2016

cmsbuild commented Aug 16, 2016

ghellwig commented Aug 22, 2016 • edited

ghellwig commented Aug 22, 2016

franzoni commented Aug 22, 2016

mmusich commented Aug 22, 2016

mmusich commented Aug 22, 2016

lpernie commented Aug 22, 2016

mmusich commented Aug 22, 2016

fabozzi commented Aug 23, 2016

cmsbuild commented Aug 23, 2016

davidlange6 commented Aug 23, 2016

mmusich Sep 5, 2016

Choose a reason for hiding this comment

lpernie Sep 5, 2016

Choose a reason for hiding this comment

mmusich Sep 6, 2016

Choose a reason for hiding this comment

mmusich commented Jul 19, 2016 •

edited

cmsbuild commented Jul 19, 2016 •

edited

ghellwig commented Aug 22, 2016 •

edited