Failure when reading complex root file

[ibab]

I'm having problems reading a complex ROOT file (containing many branches with variable-length arrays) using root_numpy.
I've uploaded the first entry of the dataset here

Running the following script

from root_numpy import *

arr = root2array('out_tree.root', 'tree')
print(arr)

produces the following stack trace:

 *** Break *** segmentation violation



===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x00007fdabed70aaa in waitpid () from /usr/lib/libc.so.6
#1  0x00007fdabecf840b in do_system () from /usr/lib/libc.so.6
#2  0x00007fdab629aed1 in TUnixSystem::StackTrace() () from /usr/lib/root/libCore.so.5.34
#3  0x00007fdab629cf5c in TUnixSystem::DispatchSignals(ESignals) () from /usr/lib/root/libCore.so.5.34
#4  <signal handler called>
#5  OBJECT_getitem (ip=0x122deeb "C", ap=0x7fdab6c48b20) at numpy/core/src/multiarray/arraytypes.c.src:525
#6  0x00007fdabdf75758 in VOID_getitem (ip=0x122db50 "\345{\230\345\225w\250?kυ\t\317\024\v
", ap=0x7fdab6c48b20) at numpy/core/src/multiarray/arraytypes.c.src:617
#7  0x00007fdabe020fab in array_toscalar (self=0x7fdab6c48b20, args=<optimized out>) at numpy/core/src/multiarray/methods.c:680
#8  0x00007fdabfa7d153 in call_function (oparg=<optimized out>, pp_stack=0x7ffc7f132018) at Python/ceval.c:4033
#9  PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#10 0x00007fdabfa7ec6e in PyEval_EvalCodeEx (co=0x7fdabda9b7b0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=7, kws=0x7fdab6c70828, kwcount=0, defs=0x7fdab9b1d5a8, defcount=7, closure=0x0) at Python/ceval.c:3265
#11 0x00007fdabfa7d2aa in fast_function (nk=<optimized out>, na=7, n=<optimized out>, pp_stack=0x7ffc7f132218, func=0x7fdab9b229b0) at Python/ceval.c:4129
#12 call_function (oparg=<optimized out>, pp_stack=0x7ffc7f132218) at Python/ceval.c:4054
#13 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#14 0x00007fdabfa7ec6e in PyEval_EvalCodeEx (co=0x7fdabfdcbc30, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=1, kws=0x0, kwcount=0, defs=0x7fdabda931a8, defcount=3, closure=0x0) at Python/ceval.c:3265
#15 0x00007fdabf9fc851 in function_call (func=0x7fdab9b25398, arg=0x7fdab0af84d0, kw=0x0) at Objects/funcobject.c:526
#16 0x00007fdabf9cd323 in PyObject_Call (func=0x7fdab9b25398, arg=<optimized out>, kw=<optimized out>) at Objects/abstract.c:2529
#17 0x00007fdabfa76823 in PyEval_CallObjectWithKeywords (func=0x7fdab9b25398, arg=0x7fdab0af84d0, kw=<optimized out>) at Python/ceval.c:3902
#18 0x00007fdabdf5bac4 in array_str (self=<optimized out>) at numpy/core/src/multiarray/arrayobject.c:664
#19 0x00007fdabfa1999b in _PyObject_Str (v=0x7fdab6c48b20) at Objects/object.c:430
#20 0x00007fdabfa19a8a in PyObject_Str (v=<optimized out>) at Objects/object.c:451
#21 0x00007fdabe04e80e in gentype_repr (self=<optimized out>) at numpy/core/src/multiarray/scalartypes.c.src:372
#22 0x00007fdabfa19625 in PyObject_Repr (v=0x7fdab0b306c0) at Objects/object.c:381
#23 0x00007fdabfa7e2a2 in call_function (oparg=<optimized out>, pp_stack=0x7ffc7f1325b8) at Python/ceval.c:4021
#24 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#25 0x00007fdabfa7ec6e in PyEval_EvalCodeEx (co=0x7fdabda9b4b0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=1, kws=0xd60ae0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3265
#26 0x00007fdabfa7d2aa in fast_function (nk=<optimized out>, na=1, n=<optimized out>, pp_stack=0x7ffc7f1327b8, func=0x7fdab9b22848) at Python/ceval.c:4129
#27 call_function (oparg=<optimized out>, pp_stack=0x7ffc7f1327b8) at Python/ceval.c:4054
#28 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#29 0x00007fdabfa7ec6e in PyEval_EvalCodeEx (co=0x7fdabda9ba30, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=8, kws=0xd5e978, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3265
#30 0x00007fdabfa7d2aa in fast_function (nk=<optimized out>, na=8, n=<optimized out>, pp_stack=0x7ffc7f1329b8, func=0x7fdab9b22aa0) at Python/ceval.c:4129
#31 call_function (oparg=<optimized out>, pp_stack=0x7ffc7f1329b8) at Python/ceval.c:4054
#32 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#33 0x00007fdabfa7ec6e in PyEval_EvalCodeEx (co=0x7fdabda9b5b0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=6, kws=0x6ca8a0, kwcount=1, defs=0x7fdab9b151a8, defcount=3, closure=0x0) at Python/ceval.c:3265
#34 0x00007fdabfa7d2aa in fast_function (nk=<optimized out>, na=6, n=<optimized out>, pp_stack=0x7ffc7f132bb8, func=0x7fdab9b228c0) at Python/ceval.c:4129
#35 call_function (oparg=<optimized out>, pp_stack=0x7ffc7f132bb8) at Python/ceval.c:4054
#36 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#37 0x00007fdabfa7ec6e in PyEval_EvalCodeEx (co=0x7fdabda9b7b0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=7, kws=0x7fdab6c70628, kwcount=0, defs=0x7fdab9b1d5a8, defcount=7, closure=0x0) at Python/ceval.c:3265
#38 0x00007fdabfa7d2aa in fast_function (nk=<optimized out>, na=7, n=<optimized out>, pp_stack=0x7ffc7f132db8, func=0x7fdab9b229b0) at Python/ceval.c:4129
#39 call_function (oparg=<optimized out>, pp_stack=0x7ffc7f132db8) at Python/ceval.c:4054
#40 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#41 0x00007fdabfa7ec6e in PyEval_EvalCodeEx (co=0x7fdabfdcbc30, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=1, kws=0x0, kwcount=0, defs=0x7fdabda931a8, defcount=3, closure=0x0) at Python/ceval.c:3265
#42 0x00007fdabf9fc851 in function_call (func=0x7fdab9b25398, arg=0x7fdabfe1ebd0, kw=0x0) at Objects/funcobject.c:526
#43 0x00007fdabf9cd323 in PyObject_Call (func=0x7fdab9b25398, arg=<optimized out>, kw=<optimized out>) at Objects/abstract.c:2529
#44 0x00007fdabfa76823 in PyEval_CallObjectWithKeywords (func=0x7fdab9b25398, arg=0x7fdabfe1ebd0, kw=<optimized out>) at Python/ceval.c:3902
#45 0x00007fdabdf5bac4 in array_str (self=<optimized out>) at numpy/core/src/multiarray/arrayobject.c:664
#46 0x00007fdabfa1999b in _PyObject_Str (v=0x7fdab0a48df0) at Objects/object.c:430
#47 0x00007fdabfa19a8a in PyObject_Str (v=<optimized out>) at Objects/object.c:451
#48 0x00007fdabfa19be2 in internal_print (op=0x7fdab0a48df0, fp=0x7fdabf057740 <_IO_2_1_stdout_>, flags=<optimized out>, nesting=0) at Objects/object.c:303
#49 0x00007fdabf9f4e48 in file_PyObject_Print (flags=1, f=0x7fdabff69150, op=0x7fdab0a48df0) at Objects/fileobject.c:110
#50 PyFile_WriteObject (v=<optimized out>, f=0x7fdabff69150, flags=1) at Objects/fileobject.c:2579
#51 0x00007fdabfa7b600 in PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:1771
#52 0x00007fdabfa7ec6e in PyEval_EvalCodeEx (co=0x7fdabfe77e30, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3265
#53 0x00007fdabfa7ed82 in PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>) at Python/ceval.c:667
#54 0x00007fdabfa9eba0 in run_mod (arena=0x6825b0, flags=0x7ffc7f133480, locals=0x7fdabff17168, globals=0x7fdabff17168, filename=<optimized out>, mod=0x6c4f48) at Python/pythonrun.c:1371
#55 PyRun_FileExFlags (fp=0x6c8d10, filename=<optimized out>, start=<optimized out>, globals=0x7fdabff17168, locals=0x7fdabff17168, closeit=1, flags=0x7ffc7f133480) at Python/pythonrun.c:1357
#56 0x00007fdabfa9ed7f in PyRun_SimpleFileExFlags (fp=0x6c8d10, filename=0x7ffc7f135031 "bla.py", closeit=1, flags=0x7ffc7f133480) at Python/pythonrun.c:949
#57 0x00007fdabfab4664 in Py_Main (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:645
#58 0x00007fdabecd9800 in __libc_start_main () from /usr/lib/libc.so.6
#59 0x0000000000400649 in _start ()
===========================================================


The lines below might hint at the cause of the crash.
If they do not help you then please submit a bug report at
http://root.cern.ch/bugs. Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.
===========================================================
#5  OBJECT_getitem (ip=0x122deeb "C", ap=0x7fdab6c48b20) at numpy/core/src/multiarray/arraytypes.c.src:525
#6  0x00007fdabdf75758 in VOID_getitem (ip=0x122db50 "345{230345225w250?kυt317024v
", ap=0x7fdab6c48b20) at numpy/core/src/multiarray/arraytypes.c.src:617
#7  0x00007fdabe020fab in array_toscalar (self=0x7fdab6c48b20, args=<optimized out>) at numpy/core/src/multiarray/methods.c:680
#8  0x00007fdabfa7d153 in call_function (oparg=<optimized out>, pp_stack=0x7ffc7f132018) at Python/ceval.c:4033
#9  PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#10 0x00007fdabfa7ec6e in PyEval_EvalCodeEx (co=0x7fdabda9b7b0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=7, kws=0x7fdab6c70828, kwcount=0, defs=0x7fdab9b1d5a8, defcount=7, closure=0x0) at Python/ceval.c:3265
#11 0x00007fdabfa7d2aa in fast_function (nk=<optimized out>, na=7, n=<optimized out>, pp_stack=0x7ffc7f132218, func=0x7fdab9b229b0) at Python/ceval.c:4129
#12 call_function (oparg=<optimized out>, pp_stack=0x7ffc7f132218) at Python/ceval.c:4054
#13 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#14 0x00007fdabfa7ec6e in PyEval_EvalCodeEx (co=0x7fdabfdcbc30, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=1, kws=0x0, kwcount=0, defs=0x7fdabda931a8, defcount=3, closure=0x0) at Python/ceval.c:3265
#15 0x00007fdabf9fc851 in function_call (func=0x7fdab9b25398, arg=0x7fdab0af84d0, kw=0x0) at Objects/funcobject.c:526
#16 0x00007fdabf9cd323 in PyObject_Call (func=0x7fdab9b25398, arg=<optimized out>, kw=<optimized out>) at Objects/abstract.c:2529
#17 0x00007fdabfa76823 in PyEval_CallObjectWithKeywords (func=0x7fdab9b25398, arg=0x7fdab0af84d0, kw=<optimized out>) at Python/ceval.c:3902
#18 0x00007fdabdf5bac4 in array_str (self=<optimized out>) at numpy/core/src/multiarray/arrayobject.c:664
#19 0x00007fdabfa1999b in _PyObject_Str (v=0x7fdab6c48b20) at Objects/object.c:430
#20 0x00007fdabfa19a8a in PyObject_Str (v=<optimized out>) at Objects/object.c:451
#21 0x00007fdabe04e80e in gentype_repr (self=<optimized out>) at numpy/core/src/multiarray/scalartypes.c.src:372
#22 0x00007fdabfa19625 in PyObject_Repr (v=0x7fdab0b306c0) at Objects/object.c:381
#23 0x00007fdabfa7e2a2 in call_function (oparg=<optimized out>, pp_stack=0x7ffc7f1325b8) at Python/ceval.c:4021
#24 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#25 0x00007fdabfa7ec6e in PyEval_EvalCodeEx (co=0x7fdabda9b4b0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=1, kws=0xd60ae0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3265
#26 0x00007fdabfa7d2aa in fast_function (nk=<optimized out>, na=1, n=<optimized out>, pp_stack=0x7ffc7f1327b8, func=0x7fdab9b22848) at Python/ceval.c:4129
#27 call_function (oparg=<optimized out>, pp_stack=0x7ffc7f1327b8) at Python/ceval.c:4054
#28 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#29 0x00007fdabfa7ec6e in PyEval_EvalCodeEx (co=0x7fdabda9ba30, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=8, kws=0xd5e978, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3265
#30 0x00007fdabfa7d2aa in fast_function (nk=<optimized out>, na=8, n=<optimized out>, pp_stack=0x7ffc7f1329b8, func=0x7fdab9b22aa0) at Python/ceval.c:4129
#31 call_function (oparg=<optimized out>, pp_stack=0x7ffc7f1329b8) at Python/ceval.c:4054
#32 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#33 0x00007fdabfa7ec6e in PyEval_EvalCodeEx (co=0x7fdabda9b5b0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=6, kws=0x6ca8a0, kwcount=1, defs=0x7fdab9b151a8, defcount=3, closure=0x0) at Python/ceval.c:3265
#34 0x00007fdabfa7d2aa in fast_function (nk=<optimized out>, na=6, n=<optimized out>, pp_stack=0x7ffc7f132bb8, func=0x7fdab9b228c0) at Python/ceval.c:4129
#35 call_function (oparg=<optimized out>, pp_stack=0x7ffc7f132bb8) at Python/ceval.c:4054
#36 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#37 0x00007fdabfa7ec6e in PyEval_EvalCodeEx (co=0x7fdabda9b7b0, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=7, kws=0x7fdab6c70628, kwcount=0, defs=0x7fdab9b1d5a8, defcount=7, closure=0x0) at Python/ceval.c:3265
#38 0x00007fdabfa7d2aa in fast_function (nk=<optimized out>, na=7, n=<optimized out>, pp_stack=0x7ffc7f132db8, func=0x7fdab9b229b0) at Python/ceval.c:4129
#39 call_function (oparg=<optimized out>, pp_stack=0x7ffc7f132db8) at Python/ceval.c:4054
#40 PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:2679
#41 0x00007fdabfa7ec6e in PyEval_EvalCodeEx (co=0x7fdabfdcbc30, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=1, kws=0x0, kwcount=0, defs=0x7fdabda931a8, defcount=3, closure=0x0) at Python/ceval.c:3265
#42 0x00007fdabf9fc851 in function_call (func=0x7fdab9b25398, arg=0x7fdabfe1ebd0, kw=0x0) at Objects/funcobject.c:526
#43 0x00007fdabf9cd323 in PyObject_Call (func=0x7fdab9b25398, arg=<optimized out>, kw=<optimized out>) at Objects/abstract.c:2529
#44 0x00007fdabfa76823 in PyEval_CallObjectWithKeywords (func=0x7fdab9b25398, arg=0x7fdabfe1ebd0, kw=<optimized out>) at Python/ceval.c:3902
#45 0x00007fdabdf5bac4 in array_str (self=<optimized out>) at numpy/core/src/multiarray/arrayobject.c:664
#46 0x00007fdabfa1999b in _PyObject_Str (v=0x7fdab0a48df0) at Objects/object.c:430
#47 0x00007fdabfa19a8a in PyObject_Str (v=<optimized out>) at Objects/object.c:451
#48 0x00007fdabfa19be2 in internal_print (op=0x7fdab0a48df0, fp=0x7fdabf057740 <_IO_2_1_stdout_>, flags=<optimized out>, nesting=0) at Objects/object.c:303
#49 0x00007fdabf9f4e48 in file_PyObject_Print (flags=1, f=0x7fdabff69150, op=0x7fdab0a48df0) at Objects/fileobject.c:110
#50 PyFile_WriteObject (v=<optimized out>, f=0x7fdabff69150, flags=1) at Objects/fileobject.c:2579
#51 0x00007fdabfa7b600 in PyEval_EvalFrameEx (f=<optimized out>, throwflag=<optimized out>) at Python/ceval.c:1771
#52 0x00007fdabfa7ec6e in PyEval_EvalCodeEx (co=0x7fdabfe77e30, globals=<optimized out>, locals=<optimized out>, args=<optimized out>, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3265
#53 0x00007fdabfa7ed82 in PyEval_EvalCode (co=<optimized out>, globals=<optimized out>, locals=<optimized out>) at Python/ceval.c:667
#54 0x00007fdabfa9eba0 in run_mod (arena=0x6825b0, flags=0x7ffc7f133480, locals=0x7fdabff17168, globals=0x7fdabff17168, filename=<optimized out>, mod=0x6c4f48) at Python/pythonrun.c:1371
#55 PyRun_FileExFlags (fp=0x6c8d10, filename=<optimized out>, start=<optimized out>, globals=0x7fdabff17168, locals=0x7fdabff17168, closeit=1, flags=0x7ffc7f133480) at Python/pythonrun.c:1357
#56 0x00007fdabfa9ed7f in PyRun_SimpleFileExFlags (fp=0x6c8d10, filename=0x7ffc7f135031 "bla.py", closeit=1, flags=0x7ffc7f133480) at Python/pythonrun.c:949
#57 0x00007fdabfab4664 in Py_Main (argc=<optimized out>, argv=<optimized out>) at Modules/main.c:645
#58 0x00007fdabecd9800 in __libc_start_main () from /usr/lib/libc.so.6
#59 0x0000000000400649 in _start ()
===========================================================

The error also changes depending on which variables are read in.

[ndawe]

Wow, thanks for reporting. I'll take a look. This could be something not covered by our unit tests.

[ndawe]

Yes, I see this:

*** Error in `python': malloc(): memory corruption: 0x0000000003594310 ***
Aborted

I'm using ROOT 6.03/03 and Python 2.7.9.

There is something strange about this file though. ROOT is reporting a negative size on disk for each branch:

https://gist.github.com/ndawe/d054453e5899769281b1

[ndawe]

I can reproduce the same crash after OBJECT_getitem when cloning your tree into a new file. I suspect it's either some corruption in the tree or there is a memory management problem with an np.object type.

[ibab]

Looks like I messed up the ROOT file when taking only one entry from it.
Somehow I managed to save an in-memory tree to disk, which is why it returned -1 for the branch sizes.
I've uploaded a new version.
This one gives sensible branch sizes with rootpy tree, but still segfaults when reading it in via root_numpy.

It could very well be some corruption in the tree, although it seems to look fine when opening it with TBrowser and looking at the individual branches.

[ndawe]

Thanks! Yes, I still get the same segfault. My guess is that an array (from one of the array-type branches) is being deleted / not INCREF'd properly leaving numpy with pointers to invalid memory. I'll take a closer look with valgrind.

[ndawe]

The problematic branch is B0_DECAYMTRX (I discovered this by iterating over each branch and converting it separately). If I include all other branches except this one, root_numpy gives me an array and does not segfault. root_numpy isn't aware that B0_DECAYMTRX_LENGTH is associated to B0_DECAYMTRX and is probably accessing invalid memory.

[ibab]

Just wanted to write the exact same thing :D
I'm not quite sure how ROOT manages to make sense of the variable.
It looks like the number of elements in B_DECAYMTRX is determined by a fixed number of columns (in this case 25) * B_DECAYMTRX_LENGTH

[ndawe]

I was also surprised at how long it took to convert just a single branch. After enabling profiling in the cython code, I discovered that the majority of the time was spent in our TreeChain destructor where we revert the status of each branch back to the original status. I need to think about that some more. There might be a faster way to do this, or we should give the user the option of not reverting the branches to save time.

[ibab]

The variable was written using this method: http://lhcb-release-area.web.cern.ch/LHCb-release-area/DOC/davinci/releases/latest/doxygen/d5/d75/class_tuples_1_1_tuple_obj.html#a0284acaa6c3329e2a9247f3dd649d69e

I'm surprised that ROOT can read/write variable-size matrices.
When only reading the B_DECAYMTRX variable with root_numpy, I don't get a segfault, and it looks like root_numpy only retrieves a single row (25 values) of the matrix.

Everything seems to be working fine when I exclude the variable, so I'm just going to do that for now.

Thanks for taking the time to look at this!

[ndawe]

@ibab thanks again for reporting this. I'd still call this a bug in root_numpy. It should know to back away from a branch it doesn't know how to convert properly. I'll investigate.

This also raised the issue of how long tree2array took to convert trees with thousands of branches. Most of the time was wasted on non-essential tasks like reverting branch statuses. I've made changes in #193 that really speeds up tree2array a lot. If you update to the latest commit on the master branch, you should see this increased speed on your large tree. A new tag will come soon.

[ndawe]

I've cornered the issue in root_numpy and I'll have a PR coming soon that fixes this.