Huge histograms in HGCAL Validation #30853
Comments
|
A new Issue was created by @schneiml Marcel Schneider. @Dr15Jones, @silviodonato, @dpiparo, @smuzaffar, @makortel can you please review it and eventually sign/assign? Thanks. cms-bot commands are listed here |
|
assign dqm, upgrade |
|
New categories assigned: dqm,upgrade @jfernan2,@andrius-k,@schneiml,@fioriNTU,@kmaeshima,@kpedro88 you have been requested to review this Pull request/Issue and eventually sign? Thanks |
cmsbuild
added
dqm-pending
pending-signatures
upgrade-pending
and removed
pending-assignment
labels
|
The level of reduction should come from DQM and PPD convenors.
…________________________________
From: apsallid [notifications@github.com]
Sent: 22 July 2020 15:26
To: cms-sw/cmssw
Cc: Sunanda Banerjee; Mention
Subject: Re: [cms-sw/cmssw] Huge histograms in HGCAL Validation (#30853)
@bsunanda<https://github.com/bsunanda> @rovere<https://github.com/rovere> @schneiml<https://github.com/schneiml> Is reducing in half (e.g. 8200 to 4100) a good choice for all sides or more is needed? I don't know if this is a trial and error procedure.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#30853 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABGMZOR7U6Z6HXCWDG2EHOLR43SJPANCNFSM4PDZSZIA>.
|
|
@bsunanda the level or reduction should be derived from the usefulness and information that is supposed to be stored in the plots. If I were DQM convener, I'd refuse anything less than a factor 10 reduction in number of bins (for each axis). This is validation, not visualization. |
|
@apsallid at least factor 10 (or even 20 in line with the MEs for X and Y coordinates, unless Z is indeed as much as twice sensible) is requested yes, as @rovere pointed out, a factor 2 is totally insufficient. Please bear in mind the usual stats involved in a RelVal sample for validation and the stats you plan to get to populate those MEs. |
|
The current version of these plots is so big that DQMGUI refuses to render them, as I suspected (the limit is 8MB IIRC, so somewhere around 1M-2M bins). In general, we don't really care about a few thousand bins here and there (and that gives plenty of room to add all sorts of plots), but once it gets to millions of bins, memory consumption really starts to matter (no matter if it is in one huge plot or many small ones). As Javi wrote, there are a bunch of other questionably large histograms, but those two really seriously stand out. |
|
Plots are now rendered in Jenkins tests of #30879 |
|
Has this been improved sufficiently to close the issue? |
|
Ciao @kpedro88 |
|
Hi, from DQM side the solution is sufficient |
|
+upgrade |
|
+1 |
|
This issue can be closed IMHO |
|
This issue is fully signed and ready to be closed. |
Recently, we experienced some problems with DQMIO merge jobs running out of memory on Phase2 workflows (various threads on this issue exist). These problems are caused by two histograms:
Booked here: https://github.com/cms-sw/cmssw/blob/master/Validation/HGCalValidation/plugins/HGCalHitValidation.cc#L156-L163
Each consuming a significant fraction of the total memory used by the affected jobs.
Those huge histograms also triggered the bug ROOT-10927 [1] which lead us to uncover this issue.
It would be highly appreciated if these histograms could be replaced by less memory-hungry ones.
@srimanob FYI
[1] https://sft.its.cern.ch/jira/browse/ROOT-10927
The text was updated successfully, but these errors were encountered: