New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refresh the index map of pointers to hits and tracks in Pandora translator #8015
Refresh the index map of pointers to hits and tracks in Pandora translator #8015
Conversation
@vandreev11 @pfs @kpedro88 @sethzenz -- just to make sure |
A new Pull Request was created by @lgray (Lindsey Gray) for CMSSW_6_2_X_SLHC. Refresh the index map of pointers to hits and tracks in Pandora translator It involves the following packages: RecoParticleFlow/PandoraTranslator The following packages do not have a category, yet: RecoParticleFlow/PandoraTranslator @cmsbuild, @nclopezo can you please review it and eventually sign? Thanks. |
Please test |
The tests are being triggered in jenkins. |
@mark-grimes I'm probably going to move the .clear() to the end of produce() so the memory is cleared for the rest of the present event. |
If you are dealing with a map, then you really are better off putting it on the stack since you will not be amortizing any memory allocations. |
There is no substantial change to the runtime aside from retained memory, please go ahead and merge. Private tests indicate the crash is gone, or at least we no longer see errors within 20 PU 140 events, where we were able to make the crash reproducible. |
merge |
Refresh the index map of pointers to hits and tracks in Pandora translator
Runtime and memory use after applying this patch for 42 events (the rest are still running). I guess the cuts in #8023 could speed this up. |
@mark-grimes RSS is never over 3GB, is this OK for RelVals? Or will we get killed on VM size? |
@mark-grimes Speed ups from #8023 will likely be rather random in nature. |
I think so. The current plan is to submit the relvals and keep fingers crossed; release came out yesterday. @davidlange6 was reporting a few hits of ~3.1GB, my worry is that it depends on the sample used. I was using QCDForPF which is flat, he was using QCD 80-120 which will have more in the endcaps. For the record, full plots are below (same as above but for the finished job). The job crashed on event 63 with an error in SoftPFMuonTagInfoProducer, already fixed in #8022. I also added the user time split by module for the top 10, although it's not particularly informative since Pandora dominates as expected. Average time per event comes out at 13 +/- 4 minutes. |
Hi Mark
|
I'll start a job to check, I've always run with DQM on. |
First 23 Shashlik events are 2.4 +/- 0.4 minutes/event. |
If you could run the HGCal-Pandora jobs with igprof on the events that have tails in to 20 minutes or more it'll help too. At some point the algorithms need to be refined much further. Right now the speedups are from patches and short circuits around slow bits, doing a proper rewrite would improve this much more. |
An update here: Kevin and I are addressing some issues of memory locality (i.e. keeping workspaces in the same place, and not causing 'churn') that we found in the modified pandora algos. This seems to give an improvement to both RSS and timing. Stay tuned, we're checking the effects on physics performance. |
OK, not much of a timing improvement, but the memory usage patterns are certainly more sane. Definitely helps RSS a bit too. |
Fix issue where the index map was not reset each iteration, thus eventually causing a crash either due to bad read or memory usage.
@vandreev11 @pfs @kpedro88 @sethzenz