Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce memory allocation size for compression buffer #14651

Open
pcanal opened this issue Feb 10, 2024 · 0 comments
Open

Reduce memory allocation size for compression buffer #14651

pcanal opened this issue Feb 10, 2024 · 0 comments

Comments

@pcanal
Copy link
Member

pcanal commented Feb 10, 2024

Explain what you would like to see improved and how.

In TKey.cxx, TBufferXML.cxx, TMessage.cxx, and TBasket.cxx the compression buffer is allocated with a size like:

Int_t buflen = TMath::Max(512,fKeylen + fObjlen + 9*nbuffers + 28); //add 28 bytes in case object is placed in a deleted gap

where

Int_t buflen = TMath::Max(512,fKeylen + fObjlen);

is likely enough (i.e. there is no point in keeping the result of the compression if it is larger (including any headers) than the uncompressed data.

A related issue is the test at TKey.cxx:267 (similar test is present is other places).

         if (nout == 0 || nout >= fObjlen) { //this happens when the buffer cannot be compressed

where the nout of a single fragment (thus at most kMAXZIPBUF with the total size of the object. The check is accurate if and only if there is only one fragment (i.e. (1 + (fObjlen - 1)/kMAXZIPBUF) == 1) other wise one can construct a scenario with some of the fragment compressing poorly (but being accepted because of the wiggle room mentioned above) and leading to case were we store the compressed data but the reader used is as uncompressed data because: noutot == fObjlen

Finally one should consider unifying those 4 similar code fragment (and maybe also the one in RNTuple) by factoring it ou.

See #14523 for related discussions.

ROOT version

master ; 6.32/00 or later as the fix for #14508 is required before attempted to address this is.

Installation method

any

Operating system

all

Additional context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants