Optimize NanoAOD buffers at 1k events. #22445
Conversation
ROOT will automatically add 16 bytes per branch buffer per event in the event cluster; in the case of NanoAOD files, we've seen up to 143k events per cluster (and 800 branches), resulting in an excess of 900MB of memory usage. With this change, we explicitly optimize NanoAOD buffers after 1k events, then subsequently flush at more appropriate intervals. To achieve this, we disable the ROOT auto-flush mechanism and explicitly optimize / flush from the NanoAOD output module itself.
|
A new Pull Request was created by @bbockelm (Brian Bockelman) for CMSSW_9_4_X. It involves the following packages: PhysicsTools/NanoAOD @gpetruc, @cmsbuild, @arizzi, @monttj can you please review it and eventually sign? Thanks. cms-bot commands are listed here |
|
@bbockelm I think we would need this before in the master branch... |
|
please test |
|
The tests are being triggered in jenkins. |
|
Comparison job queued. |
|
Comparison is ready Comparison Summary:
|
|
@bbockelm thanks, looking at it |
|
+1 |
|
merge |
ROOT will automatically add 16 bytes per branch buffer per event in the event cluster; in the case of NanoAOD files, we've seen up to 143k events per cluster (and 800 branches), resulting in an excess of 900MB of memory usage.
With this change, we explicitly optimize NanoAOD buffers after 1k events, then subsequently flush at more appropriate intervals. To achieve this, we disable the ROOT auto-flush mechanism and explicitly optimize / flush from the NanoAOD output module itself.