[RF] Segmentation fault in plotOn methid in PyROOT for ROOT>6.25

[VanyaBelyaev]

I am working with dev3 LCG nigthlies slot.
Starting from this week, my PyROOT/RooFit scripts with complicated PDFs with multiple components produce segmentation faults for plotOn invocation. Unfortunately I can not reproduce the problem with simple PDFs.
There were no problem with last week nighlies as well as there are no problems with any released verisons of ROOT.

• Checked for duplicates

Describe the bug

Segmentation fault for drawing complicated multicomponent PDFs in PyROOT/RooFit in calling for plotOn method.
Problem appears from the start of this week. There were no problem till with last week nightlies
as well as there are no problems with any released versions of ROOT.

Expected behavior

To Reproduce

Setup

1. ROOT version 6.25.01 from dev3 nightlies slot
2. lxplus.cern.ch
3. LCG nightlies dev3 slot

Additional context

There are no problems with any released ROOT versions, and my tests with dev-3 LCG nightly slot till the mid mid of the previous week were successfull (unfortnuately I'venot run test Last Thursday and Friday)

There are no problems with "simple" PDFs, and there is no problems with PDF-only plots.
Problem appears when I make plot of DATA with multicomponent PDFs.

[eguiraud]

Hi @VanyaBelyaev , do you think the fact that you are using RooFit through PyROOT is relevant? Would you be able to test in pure C++ ?

[guitargeek]

Hi @VanyaBelyaev!

I guess you can't share all the data and scripts to reproduce the problem, but could you please post the full stack trace of the segmentation fault? I would also be helpful to see your code around the call to plotOn. With which command arguments are you calling plotOn?

And as @eguiraud said, please indicate if this problem also happens with the C++ interface, if you can easily check this.
Thanks a lot, I hope we can fix this issue quickly!

Cheers,
Jonas

[VanyaBelyaev]

Hi @VanyaBelyaev , do you think the fact that you are using RooFit through PyROOT is relevant? Would you be able to test in pure C++ ?

Dear @eguiraud
I am not using RooFit througt C++ . It would be expremently difficult or even impossible for me to conver tmy python stuff into C++.
Note that for simple models I have no problems.
It happens only for multidimensional multicomponent model, when I draw projecton

• first draw data set with Invisible flag
• draw fit components
• draw data with normal vizibility

And it segfaults at the second step.
The relevant part of the stack is here:

The lines below might hint at the cause of the crash.
You may get help by asking at the ROOT forum https://root.cern.ch/forum
Only if you are really convinced it is a bug in ROOT then please submit a
report at https://root.cern.ch/bugs Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#7  0x00007f36325eec01 in (anonymous namespace)::removeCommon(std::vector<RooAbsArg*, std::allocator<RooAbsArg*> >&, std::__ROOT::span<RooAbsArg* const>) () from /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-opt/lib/libRooFitCore.so
#8  0x00007f36325f2568 in RooProdPdf::factorizeProduct(RooArgSet const&, RooArgSet const&, RooLinkedList&, RooLinkedList&, RooLinkedList&, RooLinkedList&, RooLinkedList&) const () from /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-opt/lib/libRooFitCore.so
#9  0x00007f36325f94b9 in RooProdPdf::getPartIntList(RooArgSet const*, RooArgSet const*, char const*) const () from /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-opt/lib/libRooFitCore.so
#10 0x00007f36325fd52e in RooProdPdf::getAnalyticalIntegralWN(RooArgSet&, RooArgSet&, RooArgSet const*, char const*) const () from /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-opt/lib/libRooFitCore.so
#11 0x00007f36324ba870 in RooAddPdf::getAnalyticalIntegralWN(RooArgSet&, RooArgSet&, RooArgSet const*, char const*) const () from /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-opt/lib/libRooFitCore.so
#12 0x00007f36324ba870 in RooAddPdf::getAnalyticalIntegralWN(RooArgSet&, RooArgSet&, RooArgSet const*, char const*) const () from /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-opt/lib/libRooFitCore.so
#13 0x00007f3632618152 in RooRealIntegral::RooRealIntegral(char const*, char const*, RooAbsReal const&, RooArgSet const&, RooArgSet const*, RooNumIntConfig const*, char const*) () from /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-opt/lib/libRooFitCore.so
#14 0x00007f3632485592 in RooAbsReal::createIntObj(RooArgSet const&, RooArgSet const*, RooNumIntConfig const*, char const*) const () from /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-opt/lib/libRooFitCore.so
#15 0x00007f36324861a6 in RooAbsReal::createIntegral(RooArgSet const&, RooArgSet const*, RooNumIntConfig const*, char const*) const () from /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-opt/lib/libRooFitCore.so
#16 0x00007f3632486b77 in RooAbsReal::createPlotProjection(RooArgSet const&, RooArgSet const*, RooArgSet*&, char const*, RooArgSet const*) const () from /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-opt/lib/libRooFitCore.so
#17 0x00007f3632493d0a in RooAbsReal::plotOn(RooPlot*, RooAbsReal::PlotOpt) const () from /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-opt/lib/libRooFitCore.so
#18 0x00007f363249baf8 in RooAbsReal::plotOn(RooPlot*, RooLinkedList&) const () from /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-opt/lib/libRooFitCore.so
#19 0x00007f363246eea1 in RooAbsPdf::plotOn(RooPlot*, RooLinkedList&) const () from /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-opt/lib/libRooFitCore.so
#20 0x00007f363249794a in RooAbsReal::plotOn(RooPlot*, RooCmdArg const&, RooCmdArg const&, RooCmdArg const&, RooCmdArg const&, RooCmdArg const&, RooCmdArg const&, RooCmdArg const&, RooCmdArg const&, RooCmdArg const&, RooCmdArg const&) const () from /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-opt/lib/libRooFitCore.so
#21 0x00007f3629c7b038 in ?? ()
#22 0x000000000f89f240 in ?? ()
#23 0x00007f3632a77580 in ?? () from /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-opt/lib/libRooFitCore.so
#24 0x00007f3632a77580 in ?? () from /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-opt/lib/libRooFitCore.so
#25 0x00007f3632a77580 in ?? () from /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-opt/lib/libRooFitCore.so
#26 0x00007f3632a77580 in ?? () from /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-opt/lib/libRooFitCore.so
#27 0x00007f3632a77580 in ?? () from /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-opt/lib/libRooFitCore.so
#28 0x00007f36324e2820 in ?? () from /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-opt/lib/libRooFitCore.so
#29 0x00007ffd1e2c9950 in ?? ()
#30 0x000000000fdc2008 in ?? ()
#31 0x0000000000000014 in ?? ()
#32 0x00007ffd1e2c9950 in ?? ()
#33 0x00000000000e6f50 in ?? ()
#34 0x00000000000000cc in ?? ()
#35 0x000000000fdf20b0 in ?? ()
#36 0x000000000fcf14a8 in ?? ()
#37 0x00007f3656b22760 in ?? () from /lib64/libc.so.6
#38 0x000000000000002d in ?? ()
#39 0x00000000000000bc in ?? ()
#40 0x0000000000000014 in ?? ()
#41 0x0000000000000080 in ?? ()
#42 0x0000000000000080 in ?? ()
#43 0x00000000000000c8 in ?? ()
#44 0x0000000000000080 in ?? ()
#45 0x0000000000000000 in ?? ()
===========================================================

[VanyaBelyaev]

Hi @VanyaBelyaev!

I guess you can't share all the data and scripts to reproduce the problem, but could you please post the full stack trace of the segmentation fault? I would also be helpful to see your code around the call to plotOn. With which command arguments are you calling plotOn?

And as @eguiraud said, please indicate if this problem also happens with the C++ interface, if you can easily check this.
Thanks a lot, I hope we can fix this issue quickly!

Cheers,
Jonas

Dear Jonas @guitargeek

See a relevant part of stack in my reply to Enrico

Unfortunately there is no way to prepare C++ reproducer
The options are rather trivial

MarkerStyle(20), DrawOptions('zp'), MarkerSize (0.75), Invisible()

for data, then I have next call for the model component with options

Components( 'BBB_fitSym_compnsbghY' ), Precision(0.0001), LineColor(600), LineStyle(14), LineWidth(1)

and I get sef fault.

[eguiraud]

@VanyaBelyaev can you please post that stacktrace but using a debug build instead of an optimized build? e.g. /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-dbg/

[VanyaBelyaev]

@VanyaBelyaev can you please post that stacktrace but using a debug build instead of an optimized build? e.g. /cvmfs/sft-nightlies.cern.ch/lcg/nightlies/dev3/Fri/ROOT/HEAD/x86_64-centos7-gcc9-dbg/

Dear Enrico @eguiraud

I'll do it tomorrow

[guitargeek]

Hi @VanyaBelyaev, yes the stack trace with a debug build is indeed helpful because of the line numbers.

Seeing that RooProdPdf::factorizeProduct is involved, I am sure that the recently merged PR #7907 causes your problem. No need for a C++ producer then, this problem has nothing to do with pyROOT.

But even knowing that, I don't know enough about the problem to solve it. Do you mind sharing your RooFit code that produces the problem? Even better of course with the data, but I understand if you can't share it because of the guidelines of your experiment. But even without the data, seeing the RooFit model would be super helpful for me to understand how the RooProdPdf enters the game.

Thanks!

[VanyaBelyaev]

Dear Jonas @guitargeek

my PDF is just a simple 3D functon in each dimenstion is is just a Gaussian + linear polynomial. In total one has 8 components: SSS + SSB + SBS + BSS + SBB + BSB + BBS + BBB

Creation of the function is done via my ostap project. https://github.com/OstapHEP/ostap
You can see the actual (failing) test code in
https://github.com/OstapHEP/ostap/blob/master/ostap/fitting/tests/test_fitting_components_3D.py
It does not rely any external data - all data is produced internally
It is a bit more complicated that is actually needed, since the initial purpose of this particular test fiel was to see the change of the fit resutls with addion "additional" components mixed to the the base line model.
But it relies on "external" project https://github.com/OstapHEP/ostap

cheers, Vanya

P.S. I'll provide the stack with debug build tomorrow

[guitargeek]

Thanks a lot for the detailed information! It was due to a stupid bug in recently introduced. I suggest a fix in #8486.