Fix PFCluster and PFEGamma memory #9227
Conversation
|
please test |
|
I'll poke at PFEGProducer a bit more... it is very odd that it takes so -Lindsey On Fri, May 22, 2015 at 11:58 AM, Mark Grimes notifications@github.com
|
|
The tests are being triggered in jenkins. |
|
A new Pull Request was created by @mark-grimes (Mark Grimes) for CMSSW_6_2_X_SLHC. Fix PFCluster and PFEGamma memory It involves the following packages: RecoParticleFlow/PFClusterProducer @cmsbuild, @cvuosalo, @nclopezo, @slava77 can you please review it and eventually sign? Thanks. |
|
Is PFAlgo a service or something? I can only break down by module (EDProducer/EDAnalyzer/etcetera) and there's nothing in the list of 580 running modules that matches the name PFAlgo in any way; or in the class name of the 203 different module classes. |
|
Hi Mark, It's not from a service, the file is read in separately using TMVA::Reader -L On Fri, May 22, 2015 at 12:13 PM, Mark Grimes notifications@github.com
|
|
I'll hack at PFEGammaProducer and see which line that beginRun allocation comes in. My money is on a completely innocuous service call - could be a basic one used by loads of things but PFEGammaProducer is just the first to use it. |
|
Okay, so hacking in some printouts of the memory, the memory after each line (using SLHC25_patch6 line numbers) is: line 239 13.7829 MiB Blank lines are in there to get the memory at the start and end. So it's basically creating objects to hold the result of database records as far as I can see, which is unavoidable. Once another module asks for the same record I assume it gets a pointer to the already created object, hence it won't show up for anything else. |
|
merge |
Fix PFCluster and PFEGamma memory
Memory fixes from Lindsey.
Retained memory excluding products before applying this:
After applying this pull request:
Note that the yellow line (ElectronSeedProducer) was fixed in #9194, but I didn't have a "before" plot with that fix. So a big saving of 40-50 Mb from PFClusterProducer. No visible change in PFEGammaProducer, although that could be washed out from the big allocation in
beginRun. That's probably unavoidable from a geom service or something (though I haven't checked) - the way I count the memory, services get lumped in with the first module that calls them.So current state (this is basically the "after" plot above, but not range limited to match the "before" and the new top 25 rather than the old top 25):
Now a lot of the big allocations are in beginRun or beginLumi, which as mentioned above I'm going to assume are unavoidable for now. Might not be but I'll check those if and when memory becomes a problem again.
Only big thing left that makes the allocations in the events now is
SeedGeneratorFromRegionHitsProducer. I'll compare against 75X because I think Vincenzo made a lot of improvements. Once I get that down I'll run a check of VmSize and RSS and see where we stand in the big picture.@lgray