[RFC:2] Use ROOT lossy compression for P3 and position of reco::Track #41018

Dr15Jones · 2023-03-09T19:40:14Z

PR description:

added classes used by ROOT when doing storage which specify lossy compression.
- 3 momentum has mantissa truncated to 13 bits
- covariance matrix has mantisa truncated to 10 bits
- position is truncated to 24 bits in the range +-1100
added iotypes rule to use new classes

PR validation:

Copied an AOD file from workflow 11834.21 into the new format. It reduced the file size by 13.5%. This compares to the previous PR (#39554) which on the exact same size reduced the file size by 13.9%.

- added classes used by ROOT when doing storage which specify lossy compression. - 3 momentum has mantissa truncated to 13 bits - covariance matrix has mantisa truncated to 10 bits - position is truncated to 24 bits in the range +-1100 - added iotypes rule to use new classes

Dr15Jones · 2023-03-09T19:40:27Z

please test

cmsbuild · 2023-03-09T20:00:54Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41018/34540

This PR adds an extra 16KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/GsfTrackReco/src/classes_def.xml modified in PR(s): [RFC] Use ROOT lossy compression for P3 and position of reco::Track #39554
- File DataFormats/TrackReco/interface/TrackBase.h modified in PR(s): [RFC] Use ROOT lossy compression for P3 and position of reco::Track #39554, Displaced regional tracking iteration #40643
- File DataFormats/TrackReco/src/TrackMomentumStorage.h modified in PR(s): [RFC] Use ROOT lossy compression for P3 and position of reco::Track #39554
- File DataFormats/TrackReco/src/TrackPositionStorage.h modified in PR(s): [RFC] Use ROOT lossy compression for P3 and position of reco::Track #39554
- File DataFormats/TrackReco/src/classes.h modified in PR(s): [RFC] Use ROOT lossy compression for P3 and position of reco::Track #39554
- File DataFormats/TrackReco/src/classes_def.xml modified in PR(s): [RFC] Use ROOT lossy compression for P3 and position of reco::Track #39554

cmsbuild · 2023-03-09T20:01:19Z

A new Pull Request was created by @Dr15Jones (Chris Jones) for master.

It involves the following packages:

DataFormats/GsfTrackReco (reconstruction)
DataFormats/TrackReco (reconstruction)

@mandrenguyen, @clacaputo can you please review it and eventually sign? Thanks.
@VourMa, @JanFSchulte, @rovere, @VinInn, @missirol, @gpetruc, @mmusich, @mtosi this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2023-03-10T00:03:48Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-299027/31176/summary.html
COMMIT: 37a505f
CMSSW: CMSSW_13_1_X_2023-03-09-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/41018/31176/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 5 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 303 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3530381
DQMHistoTests: Total failures: 3010
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3527349
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 213 log files, 164 edm output root files, 49 DQM output files
TriggerResults: no differences found

slava77 · 2023-03-13T17:28:44Z

@perrotta @rappoccio would you be OK to make a special build with this PR (similar to what was done for #39554) ?
This PR is consistent with doing that on top of CMSSW_13_1_0_pre1 (to avoid cherry-pick/backport).

I don't see a particularly strong motivation to backport and make that build in CMSSW_12_6_0_pre3. I think that for validation we will be able to get the needed confirmation from re-HLT workflows on standard relvals and there will be no need to do a 4-step special relval.

Is there an ORP on Mar 14?

rappoccio · 2023-03-14T14:33:51Z

Yes, we can make this special build, I will take care of it.

There will not be an ORP today.

rappoccio · 2023-03-14T17:03:34Z

Build started here

perrotta · 2023-03-15T17:05:50Z

@slava77 the release is now available, see https://cms-talk.web.cern.ch/t/development-release-cmssw-13-1-0-pre1-lossycompression-now-available/21477/1

slava77 · 2023-03-29T13:56:18Z

looking at 1TeV muon relvals with reHLT
(comparing /RelValSingleMuPt1000/CMSSW_13_1_0_pre1_LossyCompression-130X_mcRun3_2022_realistic_v2_RV172_reHLT-v2/DQMIO
/RelValSingleMuPt1000/CMSSW_13_1_0_pre1-130X_mcRun3_2022_realistic_v2-v1/DQMIO
)
https://tinyurl.com/2z2fkjm2

vs the older 10-bit variant

Similar with 100 GeV muons has even smaller difference https://tinyurl.com/2kx6cp4v

The new (compared to #39554) is clearly much better , but for the final version I think that it still may be safer to add 1 more bit and have 14.

slava77 · 2023-03-29T14:06:23Z

The new (compared to #39554) is clearly much better , but for the final version I think that it still may be safer to add 1 more bit and have 14.

this is mostly in view of slightly better tracking resolution in phase-2 (while the only DQM setup that works out of the box is done on phase-1).

slava77 · 2023-04-21T01:03:23Z

DataFormats/TrackReco/interface/TrackBase.h

@@ -453,7 +454,7 @@ namespace reco {
    HitPattern hitPattern_;

    /// perigee 5x5 covariance matrix
-    float covariance_[covarianceSize];
+    Float16_t covariance_[covarianceSize];  //[0,0,10]


to reduce confusion, it may be worth to be explicit and add a comment that Float16_t is a full float in memory and is streamed as char (8 bit) exponent and 12 active bits for mantissa
https://root.cern.ch/doc/master/TBufferFile_8cxx_source.html#l00590

The double32 is a bit less confusing

slava77 · 2023-04-26T14:57:23Z

I ran a D88 phase-2 workflow (based on 20811.0) with 10-mu pt gun events with pt from 100 to 1000 GeV and a fixed eta value of 2.5

the difference in resolution is around 17% based on a simple Gaussian fit.

I'm going to test a case with more bits.

slava77 · 2023-04-26T15:00:29Z

@AdrianoDee
in your tests of BPH displaced reco, do we have access to the displaced vertex parameters, prob or uncertainties.

I'm not sure we have tested the covariance changes to the full extent.

slava77 · 2023-04-26T20:09:38Z

@Dr15Jones @makortel
if I read an existing file using the old version of tracks with fastCloning = False and save the results, should I expect the saved result to be streamed with the new class version?

Dr15Jones · 2023-04-27T13:10:42Z

if I read an existing file using the old version of tracks with fastCloning = False and save the results, should I expect the saved result to be streamed with the new class version?

I believe so, yes.

slava77 · 2023-05-03T01:07:57Z

DataFormats/TrackReco/src/classes_def.xml

+  <ioread
+   sourceClass="ROOT::Math::DisplacementVector3D<ROOT::Math::Cartesian3D<Double32_t>,ROOT::Math::DefaultCoordinateSystemTag>"
+    targetClass="reco::storage::TrackMomentumStorage"
+    version="[1-]"


what's the effect of this; does it mean that any math::XYZVector will be mapped to reco::storage::TrackMomentumStorage even outside of reco::Track?

That is not what it means. The source is what is in the file so this would be for an older version of the class which used the Vector. The targetClass is the type within the data product in memory.

What causes the Vector class to be written out as the new type is the explicit declaration later in the file of

<field name="vertex_" iotype="reco::storage::TrackPositionStorage" /> <field name="momentum_" iotype="reco::storage::TrackMomentumStorage" />

as that is part of reco::TrackBase declaration it will only happen to that class.

pcanal · 2023-05-04T02:43:25Z

I don't think there is anything missing. @pcanal is there an easy way to check to see what lossy-compression is being used?

If the "lossy-compression" you are talking about is Double32_t/Float16_t you can talk a look at the TStreamerInfo recorded in the file:

myfile->GetStreamerInfoList()->FindObject(myclass_name)->Print(); 
myfile->GetStreamerInfoList()->Print();

slava77 · 2023-05-04T23:25:31Z

Here are some arguments to motivate at least one more bit in the vertex position:

Firstly, recall that this PR quantizes vx,vy in 24 bits in a range ±1100 cm, 2200./2^24 cm = 1.3 µm, which is fairly close to around 2-3 µm impact parameter resolution.

phase-2 muons with pt 100 to 1 TeV eta = 0.5 generated with production vertex at (0,0,0)

and an even more artificial part of it, requiring |sin(phi)|<0.2

In the first case the quantization introduces RMS increase by 1%, while in the second case the impact is 7% and quantization is a bit too extreme.

VinInn · 2023-05-05T07:34:15Z

for storage one my try to evaluate unums https://en.wikipedia.org/wiki/Unum_(number_format)

VinInn · 2023-05-05T09:24:05Z

I have tested the posit implementation in https://github.com/stillwater-sc/universal
in the range 10^6 10^-6: one can comfortably get ~10^-6 precision using 24 bits.
the format is highly flexible (for the vertex case one can easily go down to 20 bits or so and still get submicron precision in the "cm" range)
see
https://github.com/VinInn/ctest/blob/8a5bc614ffb3c8d4a47202f665ac6a130453d23b/floatPrec/testPosit.cpp
for a trivial test

slava77 · 2023-05-05T11:40:14Z

I have tested the posit implementation in https://github.com/stillwater-sc/universal in the range 10^6 10^-6: one can comfortably get ~10^-6 precision using 24 bits. the format is highly flexible (for the vertex case one can easily go down to 20 bits or so and still get submicron precision in the "cm" range) see https://github.com/VinInn/ctest/blob/8a5bc614ffb3c8d4a47202f665ac6a130453d23b/floatPrec/testPosit.cpp for a trivial test

what's the "cm" range here? I think that we need around 1 µm precision at 10-30 cm. Just to be sure, by precision do you mean "smallest delta"/value?

slava77 · 2023-05-05T11:43:11Z

Overall, I still think that a configurable solution is really what's needed. so that we can write reduced precision in "commodity" tiers like AOD, while full precision can be done for precision tiers (RECO, ALCA). Otherwise this global level choice means that limited scope cases will drive the needs.

VinInn · 2023-05-05T11:44:58Z

with 20 bits one can have
|val| abs err| rel err|
|1 |7.62939e-06| 7.62939e-06|
|0.0001 |3.05176e-09 |3.05176e-05|
|100 |0.00152588| 1.52588e-05|
|10000| 0.610352 |6.10352e-05|

VinInn · 2023-05-05T12:07:27Z

for one micron at 20cm one needs 21 bits

slava77 · 2023-05-05T12:20:15Z

with 20 bits one can have |val| abs err| rel err| |1 |7.62939e-06| 7.62939e-06| |0.0001 |3.05176e-09 |3.05176e-05| |100 |0.00152588| 1.52588e-05| |10000| 0.610352 |6.10352e-05|

Thanks. So, I guess at around 20-30 cm it will still be around 10 µm, which is likely going to be already visible on V0/conversion re-fits for vertices in O(10 cm) range.

There is still a point on having ability to save states for the muon system; for the 10 m scale it still needs 10 µm. But perhaps this case should still be done with a TrackExtra providing the innermost state. Actually, even the V0-like cases are perhaps better served by ability to save innermost hit state where needed.

For compression benefits saving uniform scale values e.g. with 0.1 µm precision (can be *sqrt(12) ~ 0.4 µm quantization) will still be better than posit: most tracks are within 1 cm.

Another random thought: for vx,vy compression needs for tracking we would have a better compression by saving uniform range e.g. with 0.1 µm units relative to the beamspot.

... I still don't have a detailed understanding of the impact of the covariance rounding proposed in this PR. And this is the part with largest gain in storage.

slava77 · 2023-05-25T00:19:02Z

... I still don't have a detailed understanding of the impact of the covariance rounding proposed in this PR. And this is the part with largest gain in storage.

Here is a bit overdue summary of analysis related to the covariance rounding in phase-1 setup.
The test is based on a sample of K_S decays, reconstructed in a job reading variants of AOD files with different compression settings; using generalV0Candidates.

The sim and reco steps took a bit of effort to make to get enough stats at high decay radius:

the starting point is KS gun events with E in 0.5 to 2 GeV range, made in Run3 geometry
pythia gun parameters changed KS lifetime to 200 cm '310:tau0 = 2000' to get mostly flat distribution; also the decay volume is set to 2x4 m to keep decays as in pythia (not geant) with 'ParticleDecays:xyMax = 2000', 'ParticleDecays:zMax = 4000',
reco efficiency for displaced tracks improved by increasing the originRadius in pixelLess seeding to 10 cm and in tobTec to 15 cm, as well as lowering pt to 0.4 GeV

The plot is for the fitted V0/K_S vertex error in xy direction transverse to the K_S momentum, plotting the fraction of vertices with this transverse error below 30 µm. This captures the decays happening very close to the sensors.

no loss is the reference CMSSW_13_1_0_pre1
p13 c10 is this PR (momentum with 13 bits compression and covariance with 10 bits)
p15 c12 is a more precise variant: covariance is using 12 bits
- as noted above, the momentum is in full float (setting of root compression to Double32_t fX; //[0,0,15] actually uses full float/Double32_t
- Note that 12 bits (Float16_t ... //[0,0,12]) is the best offered by ROOT now; IIUC at 13 it would default to a full float.

Observations:

The lossy compression p13 c10 variant (this PR) is getting noticeably worse at 25 cm and degrades quite significantly beyond that.
- Presumably, the much larger distribution of points in this case also indicates that p13 compression also degrades the vertex position reconstruction as well.
The precise momentum (p15) variant with c12 is visibly worse starting from around 40 cm and is not good enough.

Recall that reco::Track storage is parameterized at PCA to the beam. This means that the substantially displaced tracks undergo a fairly long distance propagation (analytical/precise) and the covariance which is relatively "compact" (position uncertainties of around the hit position resolution o(100µm)) at the innermost tracker hit "expands" (large values and off-diagonal terms) in position-related components. So, the storage precision of the covariance remains quite important.

I guess these observations also suggest that related issues for displaced tracks may be present in miniAOD packing as well. @AdrianoDee are you aware of some detailed studies comparing miniAOD to AOD for displaced decay reconstruction in BPH context with cascades or conversions for decays with L_xy >~ 30 cm?

@pcanal @Dr15Jones would it be possible to expand the compression to cover 13-15 bit range for the significand precision for Float16_t and 15-17 for Double32_t?
With the current schema it seems like even if the settings can be configurable per output file, the best available compression will not be good enough for momentum and covariance storage event in AOD for phase-1 [Run3 data].

Some mitigation strategies for the displaced tracks will be needed if there is still a strong motivation to get the lossy compression enabled. One option could be to keep some skimmed TrackExtra for a subset of tracks with track state at the innermost hit.

(note in part to self: plan to make some slides to present this in e.g. TRK POG meeting)

mandrenguyen · 2023-08-17T21:28:52Z

-1
Cleaning up reco queue.

smuzaffar · 2023-11-06T16:49:53Z

Milestone for this pull request has been moved to CMSSW_14_0_X.Please open a backport if it should also go in to CMSSW_13_3_X.

cmsbuild · 2024-02-06T10:10:04Z

Milestone for this pull request has been moved to CMSSW_14_1_X. Please open a backport if it should also go in to CMSSW_14_0_X.

makortel · 2024-04-03T15:03:25Z

@cmsbuild, please close

Per #39449 (comment)

cmsbuild · 2024-04-03T15:03:53Z

cms-bot internal usage

cmsbuild added this to the CMSSW_13_1_X milestone Mar 9, 2023

cmsbuild added reconstruction-pending pending-signatures orp-pending tests-started code-checks-pending tracking labels Mar 9, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 9, 2023

cmsbuild added tests-approved and removed tests-started labels Mar 10, 2023

rappoccio mentioned this pull request Mar 14, 2023

Build CMSSW_13_1_0_pre1_LossyCompression #41053

Closed

slava77 reviewed Apr 21, 2023

View reviewed changes

cmsbuild mentioned this pull request Apr 27, 2023

Displaced regional tracking iteration #40643

Merged

slava77 reviewed May 3, 2023

View reviewed changes

smuzaffar modified the milestones: CMSSW_13_1_X, CMSSW_13_2_X May 4, 2023

smuzaffar modified the milestones: CMSSW_13_2_X, CMSSW_13_3_X Jul 18, 2023

cmsbuild added reconstruction-rejected and removed reconstruction-pending labels Aug 17, 2023

smuzaffar modified the milestones: CMSSW_13_3_X, CMSSW_14_0_X Nov 6, 2023

cmsbuild modified the milestones: CMSSW_14_0_X, CMSSW_13_3_X Nov 6, 2023

cmsbuild modified the milestones: CMSSW_14_0_X, CMSSW_14_1_X Feb 6, 2024

cmsbuild closed this Apr 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC:2] Use ROOT lossy compression for P3 and position of reco::Track #41018

[RFC:2] Use ROOT lossy compression for P3 and position of reco::Track #41018

Dr15Jones commented Mar 9, 2023

Dr15Jones commented Mar 9, 2023

cmsbuild commented Mar 9, 2023

cmsbuild commented Mar 9, 2023

cmsbuild commented Mar 10, 2023

slava77 commented Mar 13, 2023

rappoccio commented Mar 14, 2023

rappoccio commented Mar 14, 2023

perrotta commented Mar 15, 2023

slava77 commented Mar 29, 2023

slava77 commented Mar 29, 2023

slava77 Apr 21, 2023

slava77 commented Apr 26, 2023

slava77 commented Apr 26, 2023

slava77 commented Apr 26, 2023

Dr15Jones commented Apr 27, 2023

slava77 May 3, 2023

Dr15Jones May 3, 2023

pcanal commented May 4, 2023

slava77 commented May 4, 2023

VinInn commented May 5, 2023

VinInn commented May 5, 2023

slava77 commented May 5, 2023

slava77 commented May 5, 2023

VinInn commented May 5, 2023

VinInn commented May 5, 2023

slava77 commented May 5, 2023

slava77 commented May 25, 2023

mandrenguyen commented Aug 17, 2023

smuzaffar commented Nov 6, 2023

cmsbuild commented Feb 6, 2024

makortel commented Apr 3, 2024

cmsbuild commented Apr 3, 2024

[RFC:2] Use ROOT lossy compression for P3 and position of reco::Track #41018

[RFC:2] Use ROOT lossy compression for P3 and position of reco::Track #41018

Conversation

Dr15Jones commented Mar 9, 2023

PR description:

PR validation:

Dr15Jones commented Mar 9, 2023

cmsbuild commented Mar 9, 2023

cmsbuild commented Mar 9, 2023

cmsbuild commented Mar 10, 2023

Comparison Summary

slava77 commented Mar 13, 2023

rappoccio commented Mar 14, 2023

rappoccio commented Mar 14, 2023

perrotta commented Mar 15, 2023

slava77 commented Mar 29, 2023

slava77 commented Mar 29, 2023

slava77 Apr 21, 2023

Choose a reason for hiding this comment

slava77 commented Apr 26, 2023

slava77 commented Apr 26, 2023

slava77 commented Apr 26, 2023

Dr15Jones commented Apr 27, 2023

slava77 May 3, 2023

Choose a reason for hiding this comment

Dr15Jones May 3, 2023

Choose a reason for hiding this comment

pcanal commented May 4, 2023

slava77 commented May 4, 2023

VinInn commented May 5, 2023

VinInn commented May 5, 2023

slava77 commented May 5, 2023

slava77 commented May 5, 2023

VinInn commented May 5, 2023

VinInn commented May 5, 2023

slava77 commented May 5, 2023

slava77 commented May 25, 2023

mandrenguyen commented Aug 17, 2023

smuzaffar commented Nov 6, 2023

cmsbuild commented Feb 6, 2024

makortel commented Apr 3, 2024

cmsbuild commented Apr 3, 2024