Override split level in miniaod #19680

arizzi · 2017-07-11T12:47:25Z

This PR implements a dedicated list of what to split/not-to-split for MiniAOD.
Different branches can be optimal with full split or no-split depending on the number of items per event and/or the number of empty (zeroed) datamembers (sub-branches).

The list is obtained simply comparing what takes more or less space running with full split vs no-split.
Savings of 2.6kb/ev on ttbar with PU (out of ~45kb/ev) are expected

cmsbuild · 2017-07-11T12:47:45Z

A new Pull Request was created by @arizzi for master.

It involves the following packages:

Configuration/Applications
PhysicsTools/PatAlgos

@perrotta, @monttj, @cmsbuild, @franzoni, @slava77, @davidlange6 can you please review it and eventually sign? Thanks.
@ghellwig, @mmarionncern, @gouskos, @rappoccio, @imarches, @ahinzmann, @acaudron, @gpetruc, @TaiSakuma, @Martin-Grunewald, @jdolen, @nhanvtran, @JyothsnaKomaragiri, @gkasieczka, @schoef, @ferencek, @mverzett, @mariadalfonso, @pvmulder, @seemasharmafnal this is something you requested to watch as well.
@davidlange6 you are the release manager for this.

cms-bot commands are listed here

arizzi · 2017-07-11T12:49:08Z

@gpetruc

perrotta · 2017-07-11T13:01:41Z

please test

cmsbuild · 2017-07-11T13:02:00Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/21362/console Started: 2017/07/11 15:43

cmsbuild · 2017-07-11T15:37:33Z

+1
Tested at: c298aa3
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19680/21362/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
92ee798
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19680/21362/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19680/21362/git-merge-result

cmsbuild · 2017-07-11T15:37:36Z

Comparison job queued.

cmsbuild · 2017-07-11T16:18:44Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19680/21362/summary.html

Comparison Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 23
DQMHistoTests: Total histograms compared: 2012315
DQMHistoTests: Total failures: 133
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2012016
DQMHistoTests: Total skipped: 166
DQMHistoTests: Total Missing objects: 0
Checked 93 log files, 14 edm output root files, 23 DQM output files

arizzi · 2017-07-12T07:51:16Z

http://arizzi.web.cern.ch/arizzi/miniaod/split-custom.html
vs
http://arizzi.web.cern.ch/arizzi/miniaod/default-split.html

slava77 · 2017-07-13T06:17:21Z

Configuration/Applications/python/ConfigBuilder.py

-                    output.fastCloning= cms.untracked.bool(False)
-                    output.overrideInputFileSplitLevels = cms.untracked.bool(True)                      
+                    from PhysicsTools.PatAlgos.slimming.miniAOD_tools import miniAOD_customizeOutput
+		    miniAOD_customizeOutput(output)


is it another case of inconsistent indentation?

it is interesting that the python intepreter understands things correctly while github doesn't

slava77 · 2017-07-13T06:24:56Z

PhysicsTools/PatAlgos/python/slimming/MicroEventContent_cff.py

+MiniAODOverrideBranchesSplitLevel = cms.untracked.VPSet( [
+cms.untracked.PSet(branch = cms.untracked.string("patMuons_slimmedMuons__*"),splitLevel=cms.untracked.int32(0)),
+cms.untracked.PSet(branch = cms.untracked.string("patElectrons_slimmedElectrons__*"),splitLevel=cms.untracked.int32(0)),
+cms.untracked.PSet(branch = cms.untracked.string("patTaus_slimmedTaus__*"),splitLevel=cms.untracked.int32(0)),


is the list of branches here substantially different from what's in MicroEventContent ?
Are there any branches that have to stay in the full split or is the real intent of the changes here to make everything in miniAOD to be splitLevel 0?

It looks like most of the miniAOD branches in this PR are set to use the no-split mode.
It seems more practical to use the output module splitLevel parameter to set the default and then override a few branches to have the full split (like packedCandidates),

some reduced verbosity would be nice, especially for readability.
E.g.

def split0PSet(name): return cms.untracked.PSet(branch = cms.untracked.string(name),splitLevel=cms.untracked.int32(0)) MiniAODOverrideBranchesSplitLevel = cms.untracked.VPSet( [ split0PSet("patMuons_slimmedMuons__*"), split0PSet("patElectrons_slimmedElectrons__*")

arizzi · 2017-07-13T06:34:03Z

Yes, there are branches that have to stay in full split, and in fact we got better performance with everything in full split than with everything in no-split (that's the reason we use full split in first place!)

Concerning what is "cheaper" to do... there are 40 branches out of 80 to keep in full split and 40 to have un-split... so there is no cheaper choice is exactly the same (so given that full-split for all is better than no split for all, I'd keep default-full-split and white list the one to not-split)

arizzi · 2017-07-13T07:28:36Z

OTOH, if one would restrict only to the branches that show some significant difference with full-split / no-split we could reduce the list to ~10 branches (and in that case is probably cheaper, in term of number of python lines, to default no-split and override only a few).

PS: I was expecting more saving when running on less events (1k) that is more "production like" given that production uses fastmerge... but this is not what I see. I'll do more tests.

slava77 · 2017-07-14T20:28:37Z

PhysicsTools/PatAlgos/python/slimming/MicroEventContent_cff.py

+cms.untracked.PSet(branch = cms.untracked.string("patPackedCandidates_lostTracks_eleTracks_*"),splitLevel=cms.untracked.int32(0)),
+cms.untracked.PSet(branch = cms.untracked.string("recoConversions_reducedEgamma_reducedSingleLegConversions_*"),splitLevel=cms.untracked.int32(0)),
+cms.untracked.PSet(branch = cms.untracked.string("recoGsfElectronCores_reducedEgamma_reducedGedGsfElectronCores_*"),splitLevel=cms.untracked.int32(0)),
+cms.untracked.PSet(branch = cms.untracked.string("recoPhotonCores_reducedEgamma_reducedGedPhotonCores_*"),splitLevel=cms.untracked.int32(0)),


what about reducedOOT stuff?

slava77 · 2017-07-14T20:29:53Z

PhysicsTools/PatAlgos/python/slimming/MicroEventContent_cff.py

+cms.untracked.PSet(branch = cms.untracked.string("l1extraL1JetParticles_l1extraParticles_Central_*"),splitLevel=cms.untracked.int32(0)),
+cms.untracked.PSet(branch = cms.untracked.string("l1extraL1EmParticles_l1extraParticles_Isolated_*"),splitLevel=cms.untracked.int32(0)),
+cms.untracked.PSet(branch = cms.untracked.string("l1extraL1MuonParticles_l1extraParticles__*"),splitLevel=cms.untracked.int32(0)),
+cms.untracked.PSet(branch = cms.untracked.string("l1extraL1JetParticles_l1extraParticles_Tau_*"),splitLevel=cms.untracked.int32(0)),


how permissive is this input for wildcards?
Can we have just one *_l1extraParticles_*_* ?

cmsbuild · 2017-08-02T01:27:10Z

-1

Tested at: 2312220

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
ff15bd6
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19680/21994/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19680/21994/git-merge-result

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19680/21994/summary.html

I found follow errors while testing this PR

Failed tests: RelVals

RelVals:

When I ran the RelVals I found an error in the following worklfows:
136.788 step3

runTheMatrix-results/136.788_RunSinglePh2017B+RunSinglePh2017B+HLTDR2_2017+RECODR2_2017reHLT_skimSinglePh_Prompt+HARVEST2017/step3_RunSinglePh2017B+RunSinglePh2017B+HLTDR2_2017+RECODR2_2017reHLT_skimSinglePh_Prompt+HARVEST2017.log

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
ff15bd6
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19680/21994/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19680/21994/git-merge-result

cmsbuild · 2017-08-02T01:27:13Z

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

slava77 · 2017-08-02T01:37:49Z

@cmsbuild please test

the last few two failed with what seems like some random timeout or similar kill.
In the last attempt the step3 of 136.788 failed with ret= 35072
where are these codes defined? @smuzaffar @Dr15Jones

cmsbuild · 2017-08-02T01:38:06Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/21995/console Started: 2017/08/02 05:45

cmsbuild · 2017-08-02T05:11:05Z

+1
Tested at: 2312220
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19680/21995/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
ff15bd6
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19680/21995/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19680/21995/git-merge-result

cmsbuild · 2017-08-02T05:11:10Z

Comparison job queued.

cmsbuild · 2017-08-02T08:47:44Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-19680/21995/summary.html

Comparison Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 25
DQMHistoTests: Total histograms compared: 2651091
DQMHistoTests: Total failures: 31637
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2619273
DQMHistoTests: Total skipped: 181
DQMHistoTests: Total Missing objects: 0
Checked 102 log files, 14 edm output root files, 25 DQM output files

slava77 · 2017-08-03T02:39:28Z

+1

for #19680 2312220

default split level in miniAOD changed to 0, except for fairly large products (like packedPFCandidates ) which are left in full split mode
jenkins tests pass and comparisons with baseline show no differences
local test with 3K events running PAT/miniAOD on ttbar PU35 (an equivalent of wf 10224) confirm the decrease in the event size of about 6% or 3kB/evt. Output module CPU time is unchanged.

Some of the largest reductions (above 0.5% of the output file size)

orig      new    delta    delta/fullFile     name
   3071.8 ->      2669.3       -403      -0.80     patMuons_slimmedMuons__PAT.
    237.4 ->         4.3       -233      -0.46     patPhotons_slimmedOOTPhotons__PAT.
   1158.9 ->       691.3       -468      -0.93     patPhotons_slimmedPhotons__PAT.
   1837.5 ->      1395.5       -442      -0.88     patElectrons_slimmedElectrons__PAT.

slava77 · 2017-08-07T13:43:42Z

@davidlange6 @monttj @franzoni
analysis and operations signatures are missing for this PR.
Please check and suggest changes or sign.
Thank you.

davidlange6 · 2017-08-09T08:08:51Z

merge

arizzi added 2 commits July 11, 2017 14:30

use central function to customize miniaod out module

ad7f770

first version of override split level list for MiniAOD

c298aa3

cmsbuild added this to the CMSSW_9_3_X milestone Jul 11, 2017

cmsbuild added analysis-pending comparison-pending operations-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Jul 11, 2017

cmsbuild added tests-started and removed tests-pending labels Jul 11, 2017

cmsbuild added tests-approved and removed tests-started labels Jul 11, 2017

cmsbuild added comparison-available and removed comparison-pending labels Jul 11, 2017

slava77 reviewed Jul 13, 2017

View reviewed changes

slava77 reviewed Jul 14, 2017

View reviewed changes

cmsbuild added comparison-notrun tests-rejected and removed comparison-pending tests-started labels Aug 2, 2017

cmsbuild added comparison-pending tests-pending and removed comparison-notrun tests-rejected labels Aug 2, 2017

cmsbuild added tests-started and removed tests-pending labels Aug 2, 2017

cmsbuild added tests-approved and removed tests-started labels Aug 2, 2017

cmsbuild added comparison-available and removed comparison-pending labels Aug 2, 2017

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Aug 3, 2017

arizzi mentioned this pull request Aug 3, 2017

Override split level in miniaod 92X version #20037

Merged

cmsbuild merged commit 5d342d6 into cms-sw:master Aug 9, 2017

slava77 mentioned this pull request Aug 9, 2017

GenJetFlavourInfos in MiniAOD and Change in JetFlavour of LightJets to physicsPartons in CMSSW 92X #19048

Merged

arizzi deleted the overrideSplitMiniAOD branch October 11, 2017 08:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Override split level in miniaod #19680

Override split level in miniaod #19680

arizzi commented Jul 11, 2017

cmsbuild commented Jul 11, 2017

arizzi commented Jul 11, 2017

perrotta commented Jul 11, 2017

cmsbuild commented Jul 11, 2017 •

edited

cmsbuild commented Jul 11, 2017

cmsbuild commented Jul 11, 2017

cmsbuild commented Jul 11, 2017

arizzi commented Jul 12, 2017

slava77 Jul 13, 2017

arizzi Jul 13, 2017

slava77 Jul 13, 2017

slava77 Jul 14, 2017

arizzi commented Jul 13, 2017

arizzi commented Jul 13, 2017

slava77 Jul 14, 2017

slava77 Jul 14, 2017

cmsbuild commented Aug 2, 2017

cmsbuild commented Aug 2, 2017

slava77 commented Aug 2, 2017

cmsbuild commented Aug 2, 2017 •

edited

cmsbuild commented Aug 2, 2017

cmsbuild commented Aug 2, 2017

cmsbuild commented Aug 2, 2017

slava77 commented Aug 3, 2017

slava77 commented Aug 7, 2017

davidlange6 commented Aug 9, 2017

Override split level in miniaod #19680

Override split level in miniaod #19680

Conversation

arizzi commented Jul 11, 2017

cmsbuild commented Jul 11, 2017

arizzi commented Jul 11, 2017

perrotta commented Jul 11, 2017

cmsbuild commented Jul 11, 2017 • edited

cmsbuild commented Jul 11, 2017

cmsbuild commented Jul 11, 2017

cmsbuild commented Jul 11, 2017

arizzi commented Jul 12, 2017

slava77 Jul 13, 2017

Choose a reason for hiding this comment

arizzi Jul 13, 2017

Choose a reason for hiding this comment

slava77 Jul 13, 2017

Choose a reason for hiding this comment

slava77 Jul 14, 2017

Choose a reason for hiding this comment

arizzi commented Jul 13, 2017

arizzi commented Jul 13, 2017

slava77 Jul 14, 2017

Choose a reason for hiding this comment

slava77 Jul 14, 2017

Choose a reason for hiding this comment

cmsbuild commented Aug 2, 2017

cmsbuild commented Aug 2, 2017

slava77 commented Aug 2, 2017

cmsbuild commented Aug 2, 2017 • edited

cmsbuild commented Aug 2, 2017

cmsbuild commented Aug 2, 2017

cmsbuild commented Aug 2, 2017

slava77 commented Aug 3, 2017

slava77 commented Aug 7, 2017

davidlange6 commented Aug 9, 2017

cmsbuild commented Jul 11, 2017 •

edited

cmsbuild commented Aug 2, 2017 •

edited