Using LZMA compression with RNTupleWriter leads to memory corruption #14508
Assignees
Labels
Comments
hahnjo
added
in:RNTuple
experiment
|
Thanks @Dr15Jones for this excellent report and reproducer; here's the test case updated for ROOT #include <ROOT/RNTupleModel.hxx>
#include <ROOT/RField.hxx>
#include <ROOT/RNTuple.hxx>
#include <vector>
int main() {
auto model = ROOT::Experimental::RNTupleModel::Create();
model->AddField( ROOT::Experimental::RFieldBase::Create("ints","std::vector<int>").Unwrap());
model->AddField( ROOT::Experimental::RFieldBase::Create("floats","std::vector<float>").Unwrap());
auto writeOptions = ROOT::Experimental::RNTupleWriteOptions();
writeOptions.SetCompression(ROOT::RCompressionSetting::EAlgorithm::kLZMA, 4);
auto writer = ROOT::Experimental::RNTupleWriter::Recreate(std::move(model), "Events", "dummy.root", writeOptions);
std::vector<int> ints ={1,2,3};
void* pInts = &ints;
void** ppInts = &pInts;
std::vector<float> floats = {2.0, 4.0, 6.0};
void* pFloats = &floats;
void** ppFloats = &pFloats;
{
auto entry = writer->CreateEntry();
entry->BindRawPtr("ints", *ppInts);
entry->BindRawPtr("floats", *ppFloats);
writer->Fill(*entry);
}
writer.reset();
return 0;
}I see the problem, we are hitting the exact case where the buffer size minus the compression header (which is not properly accounted for!) is not sufficient to hold the compressed content. I have a fix, PR will come soon. |
hahnjo
added a commit
to hahnjo/root
that referenced
this issue
Feb 2, 2024
lzma_code must only see the buffers without the header, so the sizes have to be adjusted accordingly. Fixes root-project#14508
hahnjo
added a commit
to hahnjo/root
that referenced
this issue
Feb 2, 2024
lzma_code must only see the buffers without the header, so the sizes have to be adjusted accordingly. Fixes root-project#14508
hahnjo
added a commit
to hahnjo/root
that referenced
this issue
Feb 3, 2024
lzma_code must only see the buffers without the header, so the sizes have to be adjusted accordingly. Fixes root-project#14508
hahnjo
added a commit
to hahnjo/root
that referenced
this issue
Feb 7, 2024
lzma_code must only see the buffers without the header, so the sizes have to be adjusted accordingly. Fixes root-project#14508
hahnjo
added a commit
that referenced
this issue
Feb 8, 2024
lzma_code must only see the buffers without the header, so the sizes have to be adjusted accordingly. Fixes #14508
hahnjo
added a commit
to hahnjo/root
that referenced
this issue
Feb 8, 2024
lzma_code must only see the buffers without the header, so the sizes have to be adjusted accordingly. Fixes root-project#14508 (cherry picked from commit 17e3561)
hahnjo
added a commit
to hahnjo/root
that referenced
this issue
Feb 8, 2024
lzma_code must only see the buffers without the header, so the sizes have to be adjusted accordingly. Fixes root-project#14508 (cherry picked from commit 17e3561)
hahnjo
added a commit
to hahnjo/root
that referenced
this issue
Feb 8, 2024
lzma_code must only see the buffers without the header, so the sizes have to be adjusted accordingly. Fixes root-project#14508 (cherry picked from commit 17e3561)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Check duplicate issues.
Description
When I used RNTupleWriter to store CMS miniAOD like data using LZMA compression with level 4 I got libc errors about problems with free. Running valgrind on the job turned up many 'invalid writes' in calls to lzma
I see the same valgrind error when just storing one
std::vector<int>and onestd::vector<float>to RNTuple. In this case, the job completes without the error from 'free'.Reproducer
The following shows the problem when run using valgrind.
ROOT version
Installation method
CMSSW release
Operating system
linux
Additional context
No response
The text was updated successfully, but these errors were encountered: