[RF] Memory leak when running FrequentistCalculator scan in RooStats

[lmoneta]

The problem has been reported in https://root-forum.cern.ch/t/significantly-increased-memory-consumption-of-standardhypotestinvdemo-c-in-6-20-06-compared-to-6-14-04/44405

When running the FrequentistCalculator scan in RooStats with the StandardHypoTestInvDemo.C a large memory increase is observed after 6.14.
To reproduce is enough to use the workspace created from the example.xml of the histfactory tutorial. Do for example :

/usr/bin/time --verbose root.exe -l -b -q "StandardHypoTestInvDemo.C(\"example_combined_GaussExample_model.root\")"

Running bisect the cause of this memory increase is this PR:
#2866

[lmoneta]

Valgrind reports these leaks in RooArgSet:

==759== 1,008,000 bytes in 1 blocks are still reachable in loss record 18,282 of 18,292
==759==    at 0x4C2A593: operator new(unsigned long) (vg_replace_malloc.c:344)
==759==    by 0x15B5CF61: RooArgSet::operator new(unsigned long) (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15B60B12: RooArgSet::create(char const*) const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15AF77B6: RooAbsCollection::snapshot(bool) const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15C353A9: RooProdPdf::RooProdPdf(RooProdPdf const&, char const*) (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15C3EC34: RooProdPdf::clone(char const*) const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15AF6694: RooAbsCollection::snapshot(RooAbsCollection&, bool) const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15AF77D6: RooAbsCollection::snapshot(bool) const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15B24D7C: RooAbsReal::fillDataHist(RooDataHist*, RooArgSet const*, double, bool, bool) const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15B62CCA: RooBinnedGenContext::generate(double, bool, bool) (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15C6F42F: RooSimSplitGenContext::generate(double, bool, bool) (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15B1A91B: RooAbsPdf::generate(RooAbsPdf::GenSpec&) const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759== 
==759== 1,008,000 bytes in 1 blocks are still reachable in loss record 18,283 of 18,292
==759==    at 0x4C2A593: operator new(unsigned long) (vg_replace_malloc.c:344)
==759==    by 0x15B5CF61: RooArgSet::operator new(unsigned long) (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15C4FC8F: RooRealIntegral::RooRealIntegral(char const*, char const*, RooAbsReal const&, RooArgSet const&, RooArgSet const*, RooNumIntConfig const*, char const*) (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15B28A45: RooAbsReal::createIntObj(RooArgSet const&, RooArgSet const*, RooNumIntConfig const*, char const*) const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15B2915F: RooAbsReal::createIntegral(RooArgSet const&, RooArgSet const*, RooNumIntConfig const*, char const*) const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15C376A2: RooProdPdf::processProductTerm(RooArgSet const*, RooArgSet const*, char const*, RooArgSet const*, RooArgSet const&, RooArgSet const&, bool&, bool) const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15C3CF87: RooProdPdf::getPartIntList(RooArgSet const*, RooArgSet const*, RooArgList*&, RooLinkedList*&, int&, char const*) const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15C3DC47: RooProdPdf::evaluate() const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15B12590: RooAbsPdf::getValV(RooArgSet const*) const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15B24E8E: RooAbsReal::fillDataHist(RooDataHist*, RooArgSet const*, double, bool, bool) const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15B62CCA: RooBinnedGenContext::generate(double, bool, bool) (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15C6F42F: RooSimSplitGenContext::generate(double, bool, bool) (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759== 
==759== 1,008,000 bytes in 1 blocks are still reachable in loss record 18,284 of 18,292
==759==    at 0x4C2A593: operator new(unsigned long) (vg_replace_malloc.c:344)
==759==    by 0x15B5CF61: RooArgSet::operator new(unsigned long) (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15C35B09: RooProdPdf::factorizeProduct(RooArgSet const&, RooArgSet const&, RooLinkedList&, RooLinkedList&, RooLinkedList&, RooLinkedList&, RooLinkedList&) const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15C39FF7: RooProdPdf::getPartIntList(RooArgSet const*, RooArgSet const*, RooArgList*&, RooLinkedList*&, int&, char const*) const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15C3DC47: RooProdPdf::evaluate() const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15B12590: RooAbsPdf::getValV(RooArgSet const*) const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15B24E8E: RooAbsReal::fillDataHist(RooDataHist*, RooArgSet const*, double, bool, bool) const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15B62CCA: RooBinnedGenContext::generate(double, bool, bool) (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15C6F42F: RooSimSplitGenContext::generate(double, bool, bool) (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x15B1A91B: RooAbsPdf::generate(RooAbsPdf::GenSpec&) const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooFitCore.so.6.16.00)
==759==    by 0x1A650F7D: RooStats::ToyMCSampler::Generate(RooAbsPdf&, RooArgSet&, RooDataSet const*, int) const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooStats.so.6.16.00)
==759==    by 0x1A651FCA: RooStats::ToyMCSampler::GenerateToyData(RooArgSet&, double&, RooAbsPdf&) const (in /cvmfs/sft.cern.ch/lcg/app/releases/ROOT/6.16.00/x86_64-centos7-gcc48-opt/lib/libRooStats.so.6.16.00)
==759== 

[hageboeck]

Leaks found (and fixed in #7904; merging soon), but still needs backports to:

• 6.18
• 6.20
• 6.22
• 6.24

[hageboeck]

Github closed this automatically. Reopen since backports are missing.

[guitargeek]

This issue can be closed once all the backports are merged:

• 6.18: [v618][RF] Improve MemPoolForRooSets and fix memory leak in ToyMCSampler #7972
• 6.20: [v620][RF] Improve MemPoolForRooSets and fix memory leak in ToyMCSampler #7973
• 6.22: [v622][RF] Improve MemPoolForRooSets and fix memory leak in ToyMCSampler #7974
• 6.24: [v624][RF] Improve MemPoolForRooSets and fix memory leak in ToyMCSampler #7975

[luilongo]

Dear All,
sorry if I comment on a closed issue but I would like to understand if the issue shown in this slide can be related to the same problem. We have observed an increase in the memory when method GetInterval of the HypoTestInverter is called. What do you think?
Best,
Luigi

[guitargeek]

Hi @luilongo, I think there are very likely related. Can you please open a new issue with this problem, so it gets the spotlight it deserves and is better to track for us? If possible, please also add some code to reproduce the problem in that new issue (or point to the repository where we can find the code from the slides, if its public).

Thanks!