Memory leak when using snapshot to write RNTuples using RDataFrame

[jminguez2001]

Check duplicate issues.

• Checked for duplicates

Description

Following the issue #21962, I recompiled root from the branch that solved the issue and tested again in the FastFrames code. This time most of the original "definitely lost" memory leaks disappeared but I found new ones:

==8511== 72 bytes in 1 blocks are definitely lost in loss record 8,490 of 14,373
==8511==    at 0x484DFD3: operator new(unsigned long) (vg_replace_malloc.c:487)
==8511==    by 0x67605CD: new_ROOTcLcLRNTuple (G__ROOTNTuple.cxx:164)
==8511==    by 0x67605CD: ROOT::new_ROOTcLcLRNTuple(void*) (G__ROOTNTuple.cxx:163)
==8511==    by 0x6FBBDD8: TClass::NewObject(TClass::ENewType, bool) const (TClass.cxx:5058)
==8511==    by 0x6FBC283: TClass::New(TClass::ENewType, bool) const (TClass.cxx:5035)
==8511==    by 0x69B8BE9: TKey::ReadObjectAny(TClass const*) (TKey.cxx:1095)
==8511==    by 0x6972419: TDirectoryFile::GetObjectChecked(char const*, TClass const*) (TDirectoryFile.cxx:1127)
==8511==    by 0x644F550: Get<ROOT::RNTuple> (TDirectory.h:208)
==8511==    by 0x644F550: Get<ROOT::RNTuple> (TDirectoryFile.h:84)
==8511==    by 0x644F550: (anonymous namespace)::EnsureValidSnapshotRNTupleOutput(ROOT::RDF::RSnapshotOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (RDFSnapshotHelpers.cxx:263)
==8511==    by 0x644FD80: ROOT::Internal::RDF::UntypedSnapshotRNTupleHelper::UntypedSnapshotRNTupleHelper(unsigned int, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, ROOT::RDF::RSnapshotOptions const&, ROOT::Detail::RDF::RLoopManager*, ROOT::Detail::RDF::RLoopManager*, std::vector<std::type_info const*, std::allocator<std::type_info const*> > const&) (RDFSnapshotHelpers.cxx:887)
==8511==    by 0x61EE230: std::unique_ptr<ROOT::Internal::RDF::RActionBase, std::default_delete<ROOT::Internal::RDF::RActionBase> > ROOT::Internal::RDF::BuildAction<ROOT::Detail::RDF::RNodeBase>(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::shared_ptr<ROOT::Internal::RDF::SnapshotHelperArgs> const&, unsigned int, std::shared_ptr<ROOT::Detail::RDF::RNodeBase>, ROOT::Internal::RDF::RColumnRegister const&, std::vector<std::type_info const*, std::allocator<std::type_info const*> > const&) (InterfaceUtils.hxx:353)
==8511==    by 0x61F0081: ROOT::RDF::RInterface<ROOT::Detail::RDF::RNodeBase>::Snapshot(std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, ROOT::RDF::RSnapshotOptions const&) (RInterface.hxx:1402)
==8511==    by 0x60F5872: MainFrame::processSingleTruthTreeNtuple(std::shared_ptr<Truth> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::shared_ptr<Sample> const&, UniqueSampleID const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) (MainFrame.cc:2606)
==8511==    by 0x60F7066: MainFrame::processUniqueSampleNtuple(std::shared_ptr<Sample> const&, UniqueSampleID const&) (MainFrame.cc:600)
==8511== 
==8511== 72 bytes in 1 blocks are definitely lost in loss record 8,491 of 14,373
==8511==    at 0x484DFD3: operator new(unsigned long) (vg_replace_malloc.c:487)
==8511==    by 0x67605CD: new_ROOTcLcLRNTuple (G__ROOTNTuple.cxx:164)
==8511==    by 0x67605CD: ROOT::new_ROOTcLcLRNTuple(void*) (G__ROOTNTuple.cxx:163)
==8511==    by 0x6FBBDD8: TClass::NewObject(TClass::ENewType, bool) const (TClass.cxx:5058)
==8511==    by 0x6FBC283: TClass::New(TClass::ENewType, bool) const (TClass.cxx:5035)
==8511==    by 0x69B8BE9: TKey::ReadObjectAny(TClass const*) (TKey.cxx:1095)
==8511==    by 0x6972419: TDirectoryFile::GetObjectChecked(char const*, TClass const*) (TDirectoryFile.cxx:1127)
==8511==    by 0x644F550: Get<ROOT::RNTuple> (TDirectory.h:208)
==8511==    by 0x644F550: Get<ROOT::RNTuple> (TDirectoryFile.h:84)
==8511==    by 0x644F550: (anonymous namespace)::EnsureValidSnapshotRNTupleOutput(ROOT::RDF::RSnapshotOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (RDFSnapshotHelpers.cxx:263)
==8511==    by 0x644FD80: ROOT::Internal::RDF::UntypedSnapshotRNTupleHelper::UntypedSnapshotRNTupleHelper(unsigned int, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, ROOT::RDF::RSnapshotOptions const&, ROOT::Detail::RDF::RLoopManager*, ROOT::Detail::RDF::RLoopManager*, std::vector<std::type_info const*, std::allocator<std::type_info const*> > const&) (RDFSnapshotHelpers.cxx:887)
==8511==    by 0x62096B3: std::unique_ptr<ROOT::Internal::RDF::RActionBase, std::default_delete<ROOT::Internal::RDF::RActionBase> > ROOT::Internal::RDF::BuildAction<ROOT::Detail::RDF::RLoopManager>(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::shared_ptr<ROOT::Internal::RDF::SnapshotHelperArgs> const&, unsigned int, std::shared_ptr<ROOT::Detail::RDF::RLoopManager>, ROOT::Internal::RDF::RColumnRegister const&, std::vector<std::type_info const*, std::allocator<std::type_info const*> > const&) (InterfaceUtils.hxx:353)
==8511==    by 0x620BB2B: ROOT::RDF::RInterface<ROOT::Detail::RDF::RLoopManager>::Snapshot(std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, ROOT::RDF::RSnapshotOptions const&) (RInterface.hxx:1402)
==8511==    by 0x61FF27D: ObjectCopier::copyTreesTo(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, bool, bool, bool) const (ObjectCopier.cc:184)
==8511==    by 0x60F7177: MainFrame::processUniqueSampleNtuple(std::shared_ptr<Sample> const&, UniqueSampleID const&) (MainFrame.cc:619)

They seem to point to Snapshot, so I tried to reproduce them with a minimal macro and got the same error plus a new one:

==17653== 72 bytes in 1 blocks are definitely lost in loss record 6,992 of 9,905
==17653==    at 0x48D3FD3: operator new(unsigned long) (vg_replace_malloc.c:487)
==17653==    by 0x5B535CD: new_ROOTcLcLRNTuple (G__ROOTNTuple.cxx:164)
==17653==    by 0x5B535CD: ROOT::new_ROOTcLcLRNTuple(void*) (G__ROOTNTuple.cxx:163)
==17653==    by 0x4BAADD8: TClass::NewObject(TClass::ENewType, bool) const (TClass.cxx:5058)
==17653==    by 0x4BAB283: TClass::New(TClass::ENewType, bool) const (TClass.cxx:5035)
==17653==    by 0x66C9BE9: TKey::ReadObjectAny(TClass const*) (TKey.cxx:1095)
==17653==    by 0x6683419: TDirectoryFile::GetObjectChecked(char const*, TClass const*) (TDirectoryFile.cxx:1127)
==17653==    by 0x4E9D550: Get<ROOT::RNTuple> (TDirectory.h:208)
==17653==    by 0x4E9D550: Get<ROOT::RNTuple> (TDirectoryFile.h:84)
==17653==    by 0x4E9D550: (anonymous namespace)::EnsureValidSnapshotRNTupleOutput(ROOT::RDF::RSnapshotOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (RDFSnapshotHelpers.cxx:263)
==17653==    by 0x4E9DD80: ROOT::Internal::RDF::UntypedSnapshotRNTupleHelper::UntypedSnapshotRNTupleHelper(unsigned int, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, ROOT::RDF::RSnapshotOptions const&, ROOT::Detail::RDF::RLoopManager*, ROOT::Detail::RDF::RLoopManager*, std::vector<std::type_info const*, std::allocator<std::type_info const*> > const&) (RDFSnapshotHelpers.cxx:887)
==17653==    by 0x4057522: std::unique_ptr<ROOT::Internal::RDF::RActionBase, std::default_delete<ROOT::Internal::RDF::RActionBase> > ROOT::Internal::RDF::BuildAction<ROOT::Detail::RDF::RLoopManager>(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::shared_ptr<ROOT::Internal::RDF::SnapshotHelperArgs> const&, unsigned int, std::shared_ptr<ROOT::Detail::RDF::RLoopManager>, ROOT::Internal::RDF::RColumnRegister const&, std::vector<std::type_info const*, std::allocator<std::type_info const*> > const&) (InterfaceUtils.hxx:353)
==17653==    by 0x40548E4: ROOT::RDF::RInterface<ROOT::Detail::RDF::RLoopManager>::Snapshot(std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, ROOT::RDF::RSnapshotOptions const&) (RInterface.hxx:1402)
==17653==    by 0x40516CE: ROOT::RDF::RInterface<ROOT::Detail::RDF::RLoopManager>::Snapshot(std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::initializer_list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, ROOT::RDF::RSnapshotOptions const&) (RInterface.hxx:1511)
==17653==    by 0x404B48C: snapshot_again(char const*, char const*) (repro_rntuple_leak.cxx:17)
==17653== 
==17653== 216 (184 direct, 32 indirect) bytes in 1 blocks are definitely lost in loss record 8,282 of 9,905
==17653==    at 0x48D3FD3: operator new(unsigned long) (vg_replace_malloc.c:487)
==17653==    by 0x4B06A3C: TStorage::ObjectAlloc(unsigned long) (TStorage.cxx:292)
==17653==    by 0x6696D9B: operator new (TObject.h:189)
==17653==    by 0x6696D9B: TFile::Recover() (TFile.cxx:2174)
==17653==    by 0x66A258A: TFile::Init(bool) (TFile.cxx:869)
==17653==    by 0x66A332F: TFile::TFile(char const*, char const*, char const*, int) (TFile.cxx:583)
==17653==    by 0x66A763F: TFile::Open(char const*, char const*, char const*, int, int) (TFile.cxx:3979)
==17653==    by 0x4E9E5C9: ROOT::Internal::RDF::UntypedSnapshotRNTupleHelper::Initialize() (RDFSnapshotHelpers.cxx:940)
==17653==    by 0x4067167: ROOT::Internal::RDF::RActionSnapshot<ROOT::Internal::RDF::UntypedSnapshotRNTupleHelper, ROOT::Detail::RDF::RLoopManager>::Initialize() (RActionSnapshot.hxx:137)
==17653==    by 0x4ED757C: ROOT::Detail::RDF::RLoopManager::InitNodes() (RLoopManager.cxx:804)
==17653==    by 0x4EE0C6E: ROOT::Detail::RDF::RLoopManager::Run(bool) (RLoopManager.cxx:943)
==17653==    by 0x405C881: ROOT::RDF::RResultPtr<ROOT::RDF::RInterface<ROOT::Detail::RDF::RLoopManager> >::TriggerRun() (RResultPtr.hxx:418)
==17653==    by 0x405ADEC: ROOT::RDF::RResultPtr<ROOT::RDF::RInterface<ROOT::Detail::RDF::RLoopManager> >::GetSharedPtr() (RResultPtr.hxx:232)

Reproducer

The minimal reproducer is:

#include <ROOT/RDataFrame.hxx>

void create_initial(const char* fname, const char* ntname) {
    ROOT::RDataFrame df(5);
    auto df2 = df.Define("x", [](){ return 42; });
    ROOT::RDF::RSnapshotOptions opts;
    opts.fOutputFormat = ROOT::RDF::ESnapshotOutputFormat::kRNTuple;
    df2.Snapshot(ntname, fname, {"x"}, opts);
}

void snapshot_again(const char* fname, const char* ntname) {
    ROOT::RDataFrame df(ntname, fname);
    ROOT::RDF::RSnapshotOptions opts;
    opts.fOutputFormat = ROOT::RDF::ESnapshotOutputFormat::kRNTuple;
    opts.fMode = "UPDATE";
    opts.fOverwriteIfExists = true;
    df.Snapshot(ntname, fname, {"x"}, opts);
}

int main() {
    const char* file  = "repro_leak.root";
    const char* ntuple = "myNTuple";

    create_initial(file, ntuple);   
    snapshot_again(file, ntuple);   
    return 0;
}

Compiled with: g++ -g -O0 -o repro_rntuple_leak repro_rntuple_leak.cxx $(root-config --cflags --libs)
Run with valgrind as: valgrind --leak-check=full --show-leak-kinds=definite --track-origins=yes --error-exitcode=1 --log-file=valgrind-repro-leak.txt --suppressions=$ROOTSYS/etc/valgrind-root.supp ./repro_rntuple_leak

ROOT version

https://github.com/vepadulano/root/tree/gh-21962

Installation method

Build from source

Operating system

Ubuntu 25.10

Additional context

No response

[ferdymercury]

recompiled root from the branch

If you want to try out, maybe this (untested) additional changes help: #21990
EDIT: I tested now and it seems and it only helps partially.

[vepadulano]

This PR #21964 is attempting to fix all the similar usages of TFile that require resource management in RDF. This issue falls in that same logic, so let's keep discussion/fixes over that PR 👍 Let's see if that's enough or we need more still.