Segmentation fault when reversing an array enough times (Trac #466)

[numpy-gitbot]

Original ticket http://projects.scipy.org/numpy/ticket/466 on 2007-03-08 by @FrancescAlted, assigned to @dmcooke.

Hi,

I'm getting a consistent seg fault by running this code:

import numpy

print "numpy version-->", numpy.__version__
N = 1000*1000
a = numpy.array([1,2])
for i in xrange(N):
    a = a[::-1]

and the output for my machine:

numpy version--> 1.0.2.dev3546
Violació de segment

I need 1 million of iterations for getting the segfault, but your mileage may vary.

[numpy-gitbot]

@alberts wrote on 2007-03-10

Confirmed with 1.0.2.dev3572. I ran the provided code through Valgrind. After a while Valgrind crashes with:

==3912== Stack overflow in thread 1: can't grow stack to 0xBE416FEC
==3912== 
==3912== Process terminating with default action of signal 11 (SIGSEGV)
==3912==  Access not within mapped region at address 0xBE416FEC
==3912==    at 0x444CC9B: array_dealloc (arrayobject.c:1907)

Running under gdb, I get the following backtrace when it crashes:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1208523072 (LWP 3998)]
0x00262c9b in array_dealloc (self=0xa8c42a0) at numpy/core/src/arrayobject.c:1907
1907    array_dealloc(PyArrayObject *self) {
(gdb) bt
#0  0x00262c9b in array_dealloc (self=0xa8c42a0) at numpy/core/src/arrayobject.c:1907
#1  0x00262d32 in array_dealloc (self=0xa8c42e0) at numpy/core/src/arrayobject.c:1933
#2  0x00262d32 in array_dealloc (self=0xa8c4320) at numpy/core/src/arrayobject.c:1933
#3  0x00262d32 in array_dealloc (self=0xa8c4360) at numpy/core/src/arrayobject.c:1933
... many more of the same ...

[numpy-gitbot]

@pv wrote on 2007-04-01

Not only does [::-1] crash, but also the following loops cause a crash:

for i in xrange(N):
    a = a.T




for i in xrange(N):
    a = a[:]

I notice that on each iteration, some memory is reserved, and the size of the python process grows continuously while the loop is running.

The following does not exhibit this behavior:

for i in xrange(N):
    a = numpy.array(a[::-1], copy=True)

I think that the code makes numpy to generate a chain of views, unlimited in length, each referring to its parent. The bottom views are never freed before the topmost one, which causes memory hogging and everything to be freed only when the program ends. When freeing, DECREFing self->base calls array_dealloc recursively until stack overflows.

If this hypothesis is correct, could the problem be fixed by walking self->base pointers upwards always when creating new views?

[numpy-gitbot]

@teoliphant wrote on 2007-05-11

I think the hypothesis is correct. This definitely looks like a stack-overflow. I am hesitant to walk self-base pointers upwards as there may be situations where this creates other problems.

There are other ways in Python to create stack over-flows like this, so I'm not sure a lot of effort should be spent on fixing this.

[numpy-gitbot]

Milestone changed to 1.0.3 Release by @dmcooke on 2007-05-11

[numpy-gitbot]

@sebhaase wrote on 2007-05-14

Hi,
Is there really no way of converting this stack-over-flow into an exception ?
I thought that in (non-numpy / standard) python there was absolutely no way of creating a segmentation fault ?

-Sebastian

[numpy-gitbot]

@dmcooke wrote on 2007-05-14

I don't agree with the wontfix. I don't think walking the self-base pointers upwards will cause other problems: you'll get a very wide tree with the base as the root, instead of a long skinny one. Using the original base would make views always views of the original array, instead of the previous.

Reopening, will assign this to me. I don't think it'll be fixed for 1.0.3, though.

[numpy-gitbot]

Milestone changed to 1.1 by @dmcooke on 2007-05-14

[numpy-gitbot]

Milestone changed to 1.3.0 by @cournape on 2008-08-13

[numpy-gitbot]

Milestone changed to Unscheduled by @cournape on 2009-03-02

[numpy-gitbot]

@mwiebe wrote on 2011-03-23

There should be a clean API mechanism for creating views, since currently fixing this requires hand-modifying many places.

One fix will be to do a separation between "arrays" and "buffers," where a buffer holds elements, and arrays are always a view into a buffer. A view into another view would end up looking into the buffer, so no chain of references could exist.

[numpy-gitbot]

@teoliphant wrote on 2011-03-23

I am not sure what is meant by a "clean API mechanism". I'm pretty sure that all that needs to be done is to push the reference to the original object exposing the memory instead of leaving it to the "most recent" array exposing the view.

This could have been changed and it is not that difficult of a change.

I dont' know what you mean by a "separation" of an array and a "buffer". An array is an example of a buffer. There are many examples of "buffers" and an array is a kind of "buffer"

[numpy-gitbot]

@mwiebe wrote on 2011-07-19

I've made a new function PyArray_SetBase(arr, obj), which encapsulates setting the base. This is what I meant by a clean API mechanism. I've changed the many places this was being done to use PyArray_SetBase.

The NPY_ARRAY_UPDATEIFCOPY flag needed special handling, and the memmap ndarray subclass had bad assumptions about how views worked, and hopefully the way I've dealt with those is good. Anything that's not should be caught with a long testing period just like during the 1.6 release.

The changes are here:

#116

[numpy-gitbot]

@charris wrote on 2011-08-13

Can this be closed?

[numpy-gitbot]

@mwiebe wrote on 2011-08-13

The examples in the original report don't crash for me, I'll go ahead and close it.

[numpy-gitbot]

Milestone changed to NumPy 2.0 by @mwiebe on 2011-08-13

[numpy-gitbot]

Milestone changed to NumPy 1.7 by @charris on 2012-04-26