[MRG] ROOT6 Support

[ndawe]

Include a possible -std=c++11 when compiling, taken directly from root-config --cflags.

Currently building fine against ROOT6 locally, but I am sorting out what appears to be some segmentation violation that causes gdb to hang when running the tests.

The include path is part of --cflags so removal of --incdir should be fine.

[coveralls]

Coverage remained the same when pulling 1ed2bc5 on ndawe:master into ecdda0c on rootpy:master.

[ndawe]

Segfault seems to be an issue with file and chain destructors:

#7  <signal handler called>
#8  0x00007fcb816b6860 in __dynamic_cast () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9  0x00007fcb823b80bd in TFile::Close (this=0x2e73b10, option=0x7fcb82607ea1 "") at /home/endw/software/root/root/io/io/src/TFile.cxx:905
#10 0x00007fcb823b5b6b in TFile::~TFile (this=0x2e73b10, __in_chrg=<optimized out>) at /home/endw/software/root/root/io/io/src/TFile.cxx:527
#11 0x00007fcb823b6082 in TFile::~TFile (this=0x2e73b10, __in_chrg=<optimized out>) at /home/endw/software/root/root/io/io/src/TFile.cxx:553
#12 0x00007fcb81ff7e7d in TChain::~TChain (this=0x2d954d0, __in_chrg=<optimized out>) at /home/endw/software/root/root/tree/tree/src/TChain.cxx:190
#13 0x00007fcb81ff7f7e in TChain::~TChain (this=0x2d954d0, __in_chrg=<optimized out>) at /home/endw/software/root/root/tree/tree/src/TChain.cxx:204
#14 0x00007fcb73289fbe in __pyx_pf_13_librootnumpy_6root2array_fromFname (__pyx_self=<optimized out>, __pyx_v_weight_name=<optimized out>, __pyx_v_include_weight=<optimized out>, __pyx_v_step=<optimized out>, __pyx_v_stop=<optimized out>, __pyx_v_start=<optimized out>, __pyx_v_selection=<optimized out>, __pyx_v_branches=<optimized out>, __pyx_v_treename=<optimized out>, __pyx_v_fnames=<optimized out>) at root_numpy/src/tree.pyx:713
#15 __pyx_pw_13_librootnumpy_7root2array_fromFname (__pyx_self=0x96000000a1, __pyx_args=0x7fcb716baee0, __pyx_kwds=0x93acc0 <PyByteArray_Type>) at root_numpy/src/tree.pyx:698
#16 0x00000000004caa35 in PyEval_EvalFrameEx ()
#17 0x00000000004c93d1 in PyEval_EvalCodeEx ()
#18 0x00000000004caf49 in PyEval_EvalFrameEx ()
#19 0x00000000004c93d1 in PyEval_EvalCodeEx ()
#20 0x0000000000503c3f in ?? ()
#21 0x000000000044a933 in PyRun_InteractiveOneFlags ()
#22 0x000000000044a725 in PyRun_InteractiveLoopFlags ()
#23 0x000000000042dbed in ?? ()
#24 0x000000000049882d in Py_Main ()
#25 0x00007fcb84815ec5 in __libc_start_main (main=0x4982c0 <main>, argc=1, argv=0x7fff6e369d38, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff6e369d28) at libc-start.c:287
#26 0x00000000004981ec in _start ()

When I remove the finally: del ttree in root2array_fromFname the test_ntuple test runs fine, but then test_single_chain hangs.

[ndawe]

Even simple stuff is failing right now:

>>> import ROOT
>>> f = ROOT.TFile()
>>> f

 *** Break *** segmentation violation

So this is some deeper issue with ROOT6 at the moment.

[kratsg]

@ndawe I've seen a segfault like this before. I know one of the things ROOT6 changed is memory management. In particular, one of the segfaults I encountered was deleting an object from memory while still using it later on, or deleting an object from memory and then ROOT segfaults because it expects to clean it up itself.

[kratsg]

Can you produce a stack trace or grab one? The current dump above doesn't give me enough information to dig through the code.

[ndawe]

@kratsg for my last post above? Simple TFile repr()?

[kratsg]

Yep. With the TFile. It shouldn't just crash like that. And as a proof of concept, can you just use something like root -l input.root and see what happens with ROOT6 if you use ROOT itself? I'm having a hard time believing that PyROOT is doing this.

[ndawe]

Yeah, me too. Probably a problem on my end. Testing what you suggest.

[ndawe]

No problem with root -l input.root

[kratsg]

Oh, you know what? You might have multiple pythons running amok. If you use PyROOT compiled with ROOT5 against ROOT6 - it might muck up? I would double-check versioning here. EG: if you can check config.log when you setup ROOT6 - grep for Python.h and check paths.

[ndawe]

Hmm... I am pretty sure I only have ROOT6 setup.

>>> import ROOT
>>> ROOT.__file__
'/home/endw/software/root/root_v6.02/lib/root/ROOT.pyc'

This is interesting:

>>> from ROOT import TFile
>>> f = TFile.Open('single1.root')
>>> 
>>> f
<ROOT.TFile object ("single1.root") at 0x409a0d0>
>>> 
>>> f = TFile()
>>> f

 *** Break *** segmentation violation

[kratsg]

That is... huh. Let me confer the documentation for a minute because I'm going crazy.

[ndawe]

Both my ROOT5 and 6 builds are built against the same python and I only have python 2.7.8 on this system.

[ndawe]

I finally got a trace after letting gdb run at 100% for some time:

===========================================================
There was a crash (kSigSegmentationViolation).
This is the entire stack trace of all threads:
===========================================================

Thread 2 (Thread 0x7f21c0912700 (LWP 6088)):
#0  sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:85
#1  0x00000000005bc117 in ?? ()
#2  0x00000000004caa35 in PyEval_EvalFrameEx ()
#3  0x00000000004e6970 in ?? ()
#4  0x00000000004cd00f in PyEval_EvalFrameEx ()
#5  0x00000000004cb212 in PyEval_EvalFrameEx ()
#6  0x00000000004cb212 in PyEval_EvalFrameEx ()
#7  0x00000000004e6970 in ?? ()
#8  0x0000000000505128 in ?? ()
#9  0x00000000004d25fb in PyEval_CallObjectWithKeywords ()
#10 0x00000000005bbc82 in ?? ()
#11 0x00007f21cf2c50a5 in start_thread (arg=0x7f21c0912700) at pthread_create.c:309
#12 0x00007f21ceff277d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7f21cf6d2740 (LWP 6075)):
#0  0x00007f21cefb94e9 in __libc_waitpid (pid=6099, stat_loc=stat_loc
entry=0x7fff640247f0, options=options
entry=0) at ../sysdeps/unix/sysv/linux/waitpid.c:40
#1  0x00007f21cef3c7d2 in do_system (line=<optimized out>) at ../sysdeps/posix/system.c:148
#2  0x00007f21cd336c34 in TUnixSystem::Exec (this=0x248b980, shellcmd=0x3e15400 "/home/endw/software/root/root_v6.02/etc/root/gdb-backtrace.sh 6075 1>&2") at /home/endw/software/root/root/core/unix/src/TUnixSystem.cxx:2069
#3  0x00007f21cd3374de in TUnixSystem::StackTrace (this=0x248b980) at /home/endw/software/root/root/core/unix/src/TUnixSystem.cxx:2297
#4  0x00007f21cd33b02b in TUnixSystem::DispatchSignals (this=0x248b980, sig=kSigSegmentationViolation) at /home/endw/software/root/root/core/unix/src/TUnixSystem.cxx:3533
#5  0x00007f21cd332e43 in SigHandler (sig=kSigSegmentationViolation) at /home/endw/software/root/root/core/unix/src/TUnixSystem.cxx:395
#6  0x00007f21cd33af7a in sighandler (sig=11) at /home/endw/software/root/root/core/unix/src/TUnixSystem.cxx:3510
#7  <signal handler called>
#8  0x00007f21ccaad613 in TDirectoryFile::Get (this=0x42340e0, namecycle=0x2ef3418 "__deref__") at /home/endw/software/root/root/io/io/src/TDirectoryFile.cxx:868
#9  0x00007f21c77639e5 in ?? ()
#10 0x00007fff64027a60 in ?? ()
#11 0x00007fff64027be0 in ?? ()
#12 0x00007fff640279f8 in ?? ()
#13 0x00000001c821699c in ?? ()
#14 0x00000000042340e0 in ?? ()
#15 0x00007f21c77639a0 in ?? ()
#16 0x00007fff64027cb0 in ?? ()
#17 0x00007f21c8211779 in TClingCallFunc::exec (this=0x414c9e0, address=0x42340e0, ret=0x7fff64027be0) at /home/endw/software/root/root/core/meta/src/TClingCallFunc.cxx:1873
#18 0x00007f21c8211bf9 in TClingCallFunc::exec_with_valref_return (this=0x414c9e0, address=0x42340e0, ret=0x7fff64027be0) at /home/endw/software/root/root/core/meta/src/TClingCallFunc.cxx:1936
#19 0x00007f21c82170dc in TClingCallFunc::ExecT<long> (this=0x414c9e0, address=0x42340e0) at /home/endw/software/root/root/core/meta/src/TClingCallFunc.cxx:2130
#20 0x00007f21c8212499 in TClingCallFunc::ExecInt (this=0x414c9e0, address=0x42340e0) at /home/endw/software/root/root/core/meta/src/TClingCallFunc.cxx:2143
#21 0x00007f21c81563dc in TCling::CallFunc_ExecInt (this=0x24d64c0, func=0x414c9e0, address=0x42340e0) at /home/endw/software/root/root/core/meta/src/TCling.cxx:6060
#22 0x00007f21cd7a1b6e in PRCallFuncExecInt (func=0x1c821699c, self=0x7fff640279f8, release_gil=255) at /home/endw/software/root/root/bindings/pyroot/src/Executors.cxx:51
===========================================================


The lines below might hint at the cause of the crash.
If they do not help you then please submit a bug report at
http://root.cern.ch/bugs. Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#8  0x00007f21ccaad613 in TDirectoryFile::Get (this=0x42340e0, namecycle=0x2ef3418 "__deref__") at /home/endw/software/root/root/io/io/src/TDirectoryFile.cxx:868
#9  0x00007f21c77639e5 in ?? ()
#10 0x00007fff64027a60 in ?? ()
#11 0x00007fff64027be0 in ?? ()
#12 0x00007fff640279f8 in ?? ()
#13 0x00000001c821699c in ?? ()
#14 0x00000000042340e0 in ?? ()
#15 0x00007f21c77639a0 in ?? ()
#16 0x00007fff64027cb0 in ?? ()
#17 0x00007f21c8211779 in TClingCallFunc::exec (this=0x414c9e0, address=0x42340e0, ret=0x7fff64027be0) at /home/endw/software/root/root/core/meta/src/TClingCallFunc.cxx:1873
#18 0x00007f21c8211bf9 in TClingCallFunc::exec_with_valref_return (this=0x414c9e0, address=0x42340e0, ret=0x7fff64027be0) at /home/endw/software/root/root/core/meta/src/TClingCallFunc.cxx:1936
#19 0x00007f21c82170dc in TClingCallFunc::ExecT<long> (this=0x414c9e0, address=0x42340e0) at /home/endw/software/root/root/core/meta/src/TClingCallFunc.cxx:2130
#20 0x00007f21c8212499 in TClingCallFunc::ExecInt (this=0x414c9e0, address=0x42340e0) at /home/endw/software/root/root/core/meta/src/TClingCallFunc.cxx:2143
#21 0x00007f21c81563dc in TCling::CallFunc_ExecInt (this=0x24d64c0, func=0x414c9e0, address=0x42340e0) at /home/endw/software/root/root/core/meta/src/TCling.cxx:6060
#22 0x00007f21cd7a1b6e in PRCallFuncExecInt (func=0x1c821699c, self=0x7fff640279f8, release_gil=255) at /home/endw/software/root/root/bindings/pyroot/src/Executors.cxx:51
===========================================================


<ROOT.TFile object at 0x42340e0>

[kratsg]

Ok, so two things.

1. stack trace: it bothers me that it's threading here. It seems like it's accessing memory from the original thread in the new thread which doesn't exist (it doesn't have access to it, so memory isn't being copied?) -- but they must be doing the same thing for TFile.Open()
2. what if we ran PyROOT from inside ROOT since we know ROOT seems fine? Try the following:

gSystem->Load( "libPyROOT" );
TPython::Exec("print1+1");
TFile* f = (void*)TPython::Eval("ROOT.TFile()");

The first line loads it, second line just checks to make sure you can do something stupid in python, and third line is the one that screwed you up before (and passes it into the ROOT interpreter).

[ndawe]

Yes, that all works fine (made minor changes to what you wrote).

[ndawe]

>>> ROOT.TTree()
<ROOT.TTree object at 0x2c49db0>
>>> 
>>> 
>>> ROOT.TChain()
<ROOT.TChain object at 0x2d26b60>
>>> 
>>> 
>>> ROOT.TFile()

 *** Break *** segmentation violation

[kratsg]

That's magical. We should probably raise it up to the ROOT devs. This Friday, I'll be able to jump on to ROOT6 on the computer in my office and should be able to report back on the simple loads and see if I can reproduce this (on a Mac).

Forum post is here: https://root.cern.ch/phpBB3/viewtopic.php?f=14&t=19097

[ndawe]

OK great! Thanks! I'll take a break from this until tomorrow.

[ndawe]

Hmm... false alarm? That also crashes in 5.34.

>>> import ROOT
>>> ROOT.TFile()

 *** Break *** segmentation violation

I suppose I just never noticed that?

[kratsg]

... whoa. I can confirm the same. Python 2.7.8 and Root 5.34.

 *** Break *** segmentation violation
 Generating stack trace...
 0x0000000101636110 in Cint::G__CallFunc::Execute(void*) (in libCint.so) + 192
 0x0000000100a307e1 in PyROOT::TRootObjectExecutor::Execute(Cint::G__CallFunc*, void*, bool) (in libPyROOT.so) + 111
 0x0000000100a35361 in PyROOT::TMethodHolder<PyROOT::TScopeAdapter, PyROOT::TMemberAdapter>::CallFast(void*, bool) (in libPyROOT.so) + 43
 0x0000000100a354c4 in PyROOT::TMethodHolder<PyROOT::TScopeAdapter, PyROOT::TMemberAdapter>::CallSafe(void*, bool) (in libPyROOT.so) + 178
 0x0000000100a352b0 in PyROOT::TMethodHolder<PyROOT::TScopeAdapter, PyROOT::TMemberAdapter>::Execute(void*, bool) (in libPyROOT.so) + 76
 0x0000000100a346de in PyROOT::TMethodHolder<PyROOT::TScopeAdapter, PyROOT::TMemberAdapter>::operator()(PyROOT::ObjectProxy*, _object*, _object*, long, bool) (in libPyROOT.so) + 406
 0x0000000100a360b6 in PyROOT::(anonymous namespace)::mp_call(PyROOT::MethodProxy*, _object*, _object*) (in libPyROOT.so) + 185
 0x000000010069d409 in PyObject_Call (in Python) + 99
 0x000000010069d59a in call_function_tail (in Python) + 72
 0x000000010069d796 in PyObject_CallMethod (in Python) + 231
 0x0000000100a41b7a in (anonymous namespace)::TFileGetAttr(_object*, _object*) (in libPyROOT.so) + 43
 0x0000000100a48bb2 in PyROOT::im_call(_object*, _object*, _object*) (in libPyROOT.so) + 61
 0x000000010069d409 in PyObject_Call (in Python) + 99
 0x0000000100699b2d in PyObject_CallFunctionObjArgs (in Python) + 190
 0x00000001006e717e in call_attribute (in Python) + 70
 0x00000001006e3d16 in slot_tp_getattr_hook (in Python) + 360
 0x00000001006cddeb in PyObject_HasAttr (in Python) + 11
 0x0000000100a39098 in PyROOT::(anonymous namespace)::op_repr(PyROOT::ObjectProxy*) (in libPyROOT.so) + 145
 0x00000001006cce91 in PyObject_Repr (in Python) + 51
 0x00000001006ccd44 in internal_print (in Python) + 230
 0x00000001006b41d9 in PyFile_WriteObject (in Python) + 297
 0x000000010073cb2a in sys_displayhook (in Python) + 183
 0x000000010071af6f in PyEval_EvalFrameEx (in Python) + 15788
 0x0000000100716f8f in PyEval_EvalCodeEx (in Python) + 1409
 0x00000001006bb4ba in function_call (in Python) + 350
 0x000000010069d409 in PyObject_Call (in Python) + 99
 0x000000010071cf3f in PyEval_CallObjectWithKeywords (in Python) + 93
 0x00000001007188cd in PyEval_EvalFrameEx (in Python) + 5898
 0x0000000100716f8f in PyEval_EvalCodeEx (in Python) + 1409
 0x0000000100716a08 in PyEval_EvalCode (in Python) + 54
 0x0000000100736a0b in run_mod (in Python) + 53
 0x0000000100736827 in PyRun_InteractiveOneFlags (in Python) + 353
 0x0000000100736336 in PyRun_InteractiveLoopFlags (in Python) + 192
 0x00000001007361e0 in PyRun_AnyFileExFlags (in Python) + 60
 0x0000000100747cd7 in Py_Main (in Python) + 3051
 0x00007fff8fdd85fd in start (in libdyld.dylib) + 1
<ROOT.TFile object at 0x7f9d3c4549c0>

[ndawe]

I just submitted a report to the ROOT devs here:

https://sft.its.cern.ch/jira/browse/ROOT-7005

[kratsg]

Voted and am watching the issue. Updated the forum post with a link to the JIRA. I guess we're back to #165 (comment) now?

[ndawe]

Thanks! Yes, that was a dead end. Something else is up.

[kratsg]

For reference, I looked at:

• https://github.com/rootpy/root_numpy/blob/master/root_numpy/tests.py#L74-L78
• https://github.com/rootpy/root_numpy/blob/master/root_numpy/_tree.py#L95-L199
• https://github.com/rootpy/root_numpy/blob/master/root_numpy/src/tree.pyx#L698-L714

Just looking at it, it seems like it should segfault as you're deleting the pointer, but not the memory allocated to it. It looks like a memory leak to me, but ROOT6 would catch these better than ROOT5 since ROOT6 will clean up a lot of stuff automatically.

[ndawe]

The dynamic_cast here in TFile.cxx must be failing:

 899    // Finish any concurrent I/O operations before we close the file handles.    
 900    if (fCacheRead) fCacheRead->Close();                                         
 901    {                                                                            
 902       TIter iter(fCacheReadMap);                                                
 903       TObject *key = 0;                                                         
 904       while ((key = iter()) != 0) {                                             
 905          TFileCacheRead *cache = dynamic_cast<TFileCacheRead *>(fCacheReadMap->GetValue(key));
 906          cache->Close();                                                        
 907       }                                                                         
 908    }

[ndawe]

Just looking at it, it seems like it should segfault as you're deleting the pointer, but not the memory allocated to it. It looks like a memory leak to me, but ROOT6 would catch these better than ROOT5 since ROOT6 will clean up a lot of stuff automatically.

Hmm. I think this is the correct way to delete a heap-allocated object in Cython.

[ndawe]

Yes. Must be some race condition. One of two/three possible crashes depending on if c.Delete() is there.

[ndawe]

I updated the ROOT bug report here: https://sft.its.cern.ch/jira/browse/ROOT-7015 with minimal code in Python and C++ that reproduces these crashes. ROOT 6 is broken indeed.

[ndawe]

@kratsg are you able to confirm a crash with this?

#include "TChain.h"

int main() {
    TChain* c = new TChain("tree");
    c->Add("root_numpy/testdata/single1.root");
    c->LoadTree(0);
    c->AddBranchToCache("f_float");
    c->SetCacheSize(1000000);
    delete c;
}

[kratsg]

Can't confirm a crash.

[kratsg]

Can confirm crash with python code

(ROOT)Lord Stark:~/MultiBJets$ python main.py 
100

 *** Break *** segmentation violation
 Generating stack trace...
 0x0000000104986ff4 in PyROOT::(anonymous namespace)::op_dealloc(PyROOT::ObjectProxy*) (in libPyROOT.so) + 14
 0x000000010467bade in subtype_dealloc (in Python) + 692
 0x000000010465da17 in insertdict_by_entry (in Python) + 262
 0x000000010465b60e in insertdict (in Python) + 51
 0x000000010465adb9 in dict_set_item_by_hash_or_entry (in Python) + 40
 0x000000010465f9bc in _PyModule_Clear (in Python) + 361
 0x00000001046be304 in PyImport_Cleanup (in Python) + 503
 0x00000001046c8ec4 in Py_Finalize (in Python) + 305
 0x00000001046dad8a in Py_Main (in Python) + 3230
 0x00007fff8fdd85fd in start (in libdyld.dylib) + 1

[ndawe]

Strange. What if you instead compile the C++? But it also crashes in the cling interpreter for me.

g++ -o test test.C -I/home/edawe/software/root/root-HEAD/include/root -std=c++11 -L/home/edawe/software/root/root-HEAD/lib/root -lTree

[ndawe]

What commit are you on in ROOT? I compiled (and installed with --prefix) 09d1ffa42ccf27 (Jan 8).

[kratsg]

I'm on ROOT 5.34/23 (heads/v5-34-00-patches@v5-34-22-106-g4a0dea3.

[ndawe]

Right. This C++ doesn't crash in 5.X. Only 6. I also don't have it crashing in 5.

[kratsg]

I have ROOT6 on my other machine, but I'm not near it at the moment. I can test it tomorrow.

[ndawe]

That would be great. Thanks!

[ndawe]

@kratsg Philippe can now reproduce the crash and knows what's happening. Hopefully they have a fix soon.

[kratsg]

@ndawe I asked a friend of mine with ROOT6 - he can reproduce the crash.

[coveralls]

Coverage remained the same when pulling 6eda9eb on ndawe:master into 2232c43 on rootpy:master.

[ndawe]

Waiting for https://sft.its.cern.ch/jira/browse/ROOT-7015 to be fixed before continuing here. Hopefully it's soon.

[coveralls]

Coverage remained the same at 100.0% when pulling f25ed15 on ndawe:master into bec6544 on rootpy:master.

[ndawe]

Since we know that the problems above are on the ROOT side, I am going to merge this so that root_numpy at least compiles with C++11.

[ndawe]

https://sft.its.cern.ch/jira/browse/ROOT-7015 is fixed but there is a new issue: https://sft.its.cern.ch/jira/browse/ROOT-7200

[kratsg]

This is hilarious and sad simultaneously. I'm curious about how ROOT6 is maintaining backwards-compatibility. I've run into the secondary issue you mentioned today on some of my own unit testing (JIRA#7200).

[ndawe]

Well, 7200 is apparently an intentional change, but I easily worked around it in root_numpy. Now we have another ROOT 6 issue:

https://sft.its.cern.ch/jira/browse/ROOT-7218