-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Study additional decrease in SIM AOD size #39449
Comments
A new Issue was created by @Dr15Jones Chris Jones. @Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks. cms-bot commands are listed here |
Change to branch structureThe default for the AOD event content is to preserve the branch splitting from the input file into the output file. This was probably initially done to allow fast cloning of those branches from the input to the output. But with our use of multi-threading, we already can not use fast cloning (as using multiple threads can cause the event order in the output file to differ from the event order in the input file). By allowing the branches read/written from the input source to be fully split, the size of the 1000 event SIM AOD file decreased by 4.7 %. |
assign core,reconstruction,generators |
New categories assigned: core,generators,reconstruction @mkirsano,@menglu21,@mandrenguyen,@Dr15Jones,@smuzaffar,@clacaputo,@alberto-sanchez,@SiewYan,@makortel,@GurpreetSinghChahal,@Saptaparna you have been requested to review this Pull request/Issue and eventually sign? Thanks |
Lossy Compression OverviewROOT offers several lossy compression options, all based on the double to floatUse of the typedef The use of fixed precision conversionIf a specially structured comment is placed on the same line as the declaration of a member variable of type This form is quite useful for storing values which have a fixed precision, such as position measurements or possibly angular measurements. relative precision conversionSimilar to fixed precision conversion, adding a special comment after the declaration of the member variable one can specify relative precision conversion. In this case, the full exponential range of a This is useful for storing values where the precision of the value is proportional to the value itself, such as the case for the momentum measurement. I create a test where I looked at values from the GenParticles momentum stored with relative precision using different bit settings. The largest relative differences measured were
Extrapolating to 12 bits I would expect a relative deviation of 1.2 E-4. |
reco::Track lossy compressionThe largest contributions to the stored
I tried various relative compression settings for For The code for this change can be found here: Dr15Jones#7 The results for different compression settings can be found below
|
reco::LeafCandidate lossy compressionThe Here is the relative sizes for the shared components for
The way ROOT encodes data into fEta makes it hard to apply lossy compression as very large values of eta are truncated and then the value of Pz is added to the the internal storage of fEta. This means for low values of Pz one needs very high absolute precision. Because of that no attempt was made to apply lossy compression to fEta. Different relative lossy precision was applied to fPt while fixed precision lossy was used on fPhi. The code testing this can be found here: Given the difficulties with the polar representation, I created storage only variables storing Px, Py and Pz and then applied lossy compression to them. The code testing this can be found here: The relative savings on AOD size is as follows
Applying fixed precision lossy compression to Eta on just the PFCandidate objects yields the savings
In theory, adding the savings on just PFCandidate to the savings for all Candidates would given an upper limit on applying the all the changes together. Even then, the savings is not as great as just using the Pxyz representation. |
@Dr15Jones please add compression details wrt I don't think this compression strategy for A more appropriate for the momentum would be to go to q/p (or 1/pt), theta, phi, or even the pt,eta,phi used in the Particle representation; the compression has to preserve the angle better than the absolute value. |
I'm afraid not.
I think we really need further work to be pursued by experts, not a meddling amateur like myself :).
I actually found the opposite when I did the work on PFCandidate and GenParticle. The polar notation is substantially worse when compressed and seemed to me to have worse precision behavior. But again, those judgements would be better made by professionals. |
Did you really see that with the PFCandidate? Please remind me if this functionality is configurable by product (instead of by class) |
compression of the covariance matrix elements leading to non-positively defined matrices is already now a limiting factor for analysis using miniAOD. If the plan is to do the same on AOD it needs to be done with care and carefully cross-validated. |
Is there a summary of the effect in miniaod somewhere? (Or eg, what is the rate seen there?)
… On Sep 20, 2022, at 2:15 PM, Marco Musich ***@***.***> wrote:
I tried various relative compression settings for covariance_ and momentum_.
compression of the covariance matrix elements leading to non-positively defined matrices is already now a limiting factor for analysis using miniAOD. If the plan is to do the same on AOD it needs to be done with care and carefully cross-validated.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.
|
https://indico.cern.ch/event/1155820/#7-track-covariance-matrices-in
60% of the tracks used for BPH analysis have this problem. |
simple rounding will likely have a smaller impact than what's done in miniAOD with the cov parameterization |
not objecting to that, but one would have to see the impact and have it agreed by the relevant parties. |
A major reason I made this issue is to start to get experts to begin looking at these options. |
It is only by class. (Looks like my previous reply was accidentally included in the quote I copied). But it is possible for us to have different classes that work like the LeafCandidate (I actually did that as part of my testing) so that different inheriting classes could have different storage options. |
It is a good point as I only studied GenParticle and have not looked at the details of PFCandidate. Matti pointed me to the PackedCandidate which is used in MiniAOD storage and there the eta is truncated at +-6. |
As a test of the lose of precision I ran RECO jobs on the same 1000 events using just 1 thread in order to get event ordering identical. I then wrote out the momentum related values from the GenParticles and wrote them to a test structure which stored the values using different compression values/methods. I then measured the deviation of the values stored with compression to the values stored fully. The results are below
I have now gone and applied a Eta fixed precision between -6 and 6 on all
|
what are the target gains in % expected from this effort? |
About the number of bit for px,py,pz, if I take a naive linear propagation of the relative uncertainty, for the W mass we'll need 1e-6 precision (0.1 MeV); so, it's 19 or 20 bits. Perhaps making some toy phase-space would show the more appropriate connection. |
Such a question is "above my pay grade" :). I'm just showing that we have the possibility for substantial gains by exploring this area. @dpiparo ? |
From what I can tell, this is what the MiniAOD is using cmssw/DataFormats/PatCandidates/src/PackedCandidate.cc Lines 13 to 23 in 8a3b152
where
which is cmssw/DataFormats/Math/interface/libminifloat.h Lines 24 to 36 in c9da596
which (I think) stores 11 bits of the mantissa and 4 bits of the exponent and 1 bit for the sign. |
On one hand, not everything in miniAOD is saved at this precision; electrons and muons have the same precision as AOD. |
@slava77 how does reconstruction want to proceed? |
I can only respond for tracking ( Please clarify what you did for the |
Going for 11m and keeping the same position accuracy (which is now about 1um) would mean at least 3 more bits of storage. |
If the plots are made during RECO then they would not reflect the change in precision since that only appears for later steps reading the data. |
good point, we need to split the request to run PAT/miniAOD in a separate step from the RECO step. |
@slava77 I updated the Track lossy section with new values where I used |
@slava77 how to proceed? Should I make to pull requests with just |
@cms-sw/orp-l2 please advise |
@slava77 do you want the validation test to use compression for the position as well? |
When do you need that validation? With the present schedule, a CMSSW_12_6_0_pre3 is expected for Oct 10. Once there is a pre-release, we can branch off a couple of releases specific for this test. |
I think that pre3 is fine, |
I don't have any insight into the time scale for which this would be deployed, other than my personal bias of 'sooner would be better' in order to help our recent resource constraints. |
please add some details in the table #39449 (comment) to accompany From #39449 (comment) I'm not sure there is much gain, if 24 bits (default float) gives the same size as 21 bits. But perhaps I misunderstood the test reported there. |
When comparing just using
Given the conversion of the |
Ugh, |
@slava77 it will probably take a few days (given the speed of the jobs) to make the measurements for
Given that to first order the different field compressions are independent of each other, I expect the addition of position compression will decrease the AOD size compared with just the |
@slava77 the table has been updated with the additional measurements. The savings are even a bit better than my original estimate. |
Thanks. Perhaps we can start with the PRs now? Perhaps with just one first, to confirm BW compatibility and what can be seen in the PR tests. The somewhat more aggressive option |
pre3 is out. I guess we can proceed with the plan . Should special branches be made? |
+reconstruction |
cms-bot internal usage |
-core
I guess at this point it seems clear the lossy compression won't be going forward. |
Studying a Run 3 pileup based workflow (11834.21) with 1000 events shows that the following branches take the bulk of the space on disk
Applications of different branch structure, object thinning, and lossy compression strategies (described below) to those branches could allow us to decrease the SIM AOD size by > 15% depending on how much loss of information is acceptable.
The text was updated successfully, but these errors were encountered: