Optimize NanoAOD buffers at 1k events. #119
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ROOT will automatically add 16 bytes per branch buffer per event in the event cluster; in the case of NanoAOD files, we've seen up to 143k events per cluster (and 800 branches), resulting in an excess of 900MB of memory usage. With this change, we explicitly optimize NanoAOD buffers after 1k events, then subsequently flush at more appropriate intervals. To achieve this, we disable the ROOT auto-flush mechanism and explicitly optimize / flush from the NanoAOD output module itself.
gpetruc-bot
added
test scheduled (bot)
and removed
to be tested by bot
test scheduled (bot)
labels
|
Automatic test started, see https://gitlab.cern.ch/cms-nanoAOD/nanoAOD-integration/pipelines/331820/builds |
gpetruc-bot
added
test started (bot)
code checks ok (bot)
and removed
test started (bot)
labels
gpetruc-bot
reviewed
Mar 15, 2018
There was a problem hiding this comment.
Automatic test report for 331820
- gitlab pipeline at https://gitlab.cern.ch/cms-nanoAOD/nanoAOD-integration/pipelines/331820/builds
- outputs at https://test-cms-nanoaod-integration.web.cern.ch/integration/test_pr_119/
Code integration
Code checks passed for this PR
Tests
- Long test data80X (10000 events): passed, no significant changes; dqm plots: all, diff
- Long test data80Xhip (3000 events): passed, no significant changes; dqm plots: all, diff
- Long test data94X (10000 events): passed, no significant changes; dqm plots: all, diff
- Long test mc80X (10000 events): passed, no significant changes; dqm plots: all, diff
- Long test mc94X (10000 events): passed, no significant changes; dqm plots: all, diff
- Test mc_94X: passed
- Test mc_80X: passed
- Test mc_92X: passed
- Test data_92X: passed
- Test data_80X: passed
Disk size report
| Sample | kb/event | ref kb/event | diff |
|---|---|---|---|
| TTbar MC 94X | 1.452 | 1.510 | -0.058 ( -3.9% ) |
| TTbar MC 80X | 1.501 | 1.557 | -0.056 ( -3.6% ) |
| Data 94X | 0.588 | 0.611 | -0.022 ( -3.6% ) |
| Data 80X | 0.556 | 0.577 | -0.021 ( -3.7% ) |
| Data 80X, Mu Run2016E | 0.556 | 0.507 | 0.049 ( +9.7% ) |
|
merged upstream, no need to merge here (PR was made just to run the bot) |
peruzzim
pushed a commit
that referenced
this pull request
Apr 25, 2019
Sync with 10_5_0_pre2 Only one trivial conflict solved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
copy of cms-sw#22445 in this repo so that I can have the bot run the tests on it and see if it affects compression