BUG: Segfault when trying to access a returned numpy array backed by a buffer of named shared memory

[ikeboy]

I'm not 100% sure if this is a numpy or python bug, but the behavior is quite weird. I've narrowed it down to two simple test cases, one which works and one which segfaults.

I can create a SharedMemory object, I can create a numpy array backed by it and read and write to it. I can do the same with a second SharedMemory object referring to the same space, created using the name. But I can't have a function return this np array, or accesses will segfault.

Reproducing code example:

from multiprocessing import shared_memory
import numpy as np

def np_array_from_shm_name(name, shape=(2,)):
    shm = shared_memory.SharedMemory(name=name)
    np_array = np.ndarray(shape, buffer=shm.buf, dtype=np.int32)
    np_array[:] = (0,0)
    print(np_array)  # prints [0 0]
    return np_array


if __name__ == "__main__":
    shm = shared_memory.SharedMemory(create=True, size=8)
    shm2 = shared_memory.SharedMemory(name=shm.name)
    np_array = np.ndarray((2,), buffer=shm.buf, dtype=np.int32)
    np_array[:] = (0, 0)
    print(np_array)  # prints [0 0]
    np_array = np.ndarray((2,), buffer=shm2.buf, dtype=np.int32)
    np_array[:] = (0, 0)
    print(np_array)  # prints [0 0]
    np_array = np_array_from_shm_name(shm.name)
    np_array[:] = (0, 0)  # segfaults
    print(np_array)
    shm.unlink()
    shm2.unlink()

Error message:

Thread 1 "python3-dbg" received signal SIGSEGV, Segmentation fault.
0x00007ffff692e0e9 in INT_setitem (op=0, ov=0x7ffff7491000, vap=0x7fffd68cbd60) at numpy/core/src/multiarray/arraytypes.c.src:235
235 numpy/core/src/multiarray/arraytypes.c.src: No such file or directory.
(gdb) py-bt
Traceback (most recent call first):
File "segfault_poc_simple.py", line 66, in
np_array[:] = (0, 0) # segfaults

NumPy/Python version information:

1.17.4 3.8.6 (default, Sep 25 2020, 09:36:53)
[GCC 10.2.0]

[seberg]

If you use the memoryviews directly (replace [:] = 0, 0 with [0] = 0 and use shm.buf directly), you will see that Python complains about the buffer being closed. I am not sure about shared memory, so did not quite follow it all.

Further, if you use np.frombuffer(shm.buf), it seems that you will get an exception that the shared memory cannot be closed. np.ndarray behaves differently with respect to the base object though, it additionally has this bit of code:

    if (PyMemoryView_Check(obj)) {
        buf->base = PyMemoryView_GET_BASE(obj);
    }

Which is in PyArray_BufferConverter, which is used solely in this one place (although it is public API). I think those lines should simply be removed, getting the base of a memoryview here apparently circumvents the safety checks that the Python devs carefully put into place.

[athompson673]

shm.buf is basically a memoryview of an mmap.mmap, which gets closed as soon as shm goes out of scope. my suggestion is to not let shm go out of scope by subclassing ndarray and attaching the shm as an attribute. This way, the shm lives as long as the array does.

class SHMArray(np.ndarray): #copied from https://numpy.org/doc/stable/user/basics.subclassing.html#slightly-more-realistic-example-attribute-added-to-existing-array

    def __new__(cls, input_array, shm=None):
        obj = np.asarray(input_array).view(cls)
        obj.shm = shm
        return obj

    def __array_finalize__(self, obj):
        if obj is None: return
        self.shm = getattr(obj, 'shm', None)

[athompson673]

this is a duplicate of #9537

[seberg]

Yeah, although it highlights a slight variation in np.ndarray rather than np.frombuffer. It would be nice if shared_memory.SharedMemory implemented the buffer protocol, then numpy could directly pick it as a base. Right now, I guess the trick with an array subclass is best. You can still call np.asarray() on that to get a base-class array if you like.

[seberg]

Marking as bug, because I think we should add the workaround (which will cause a Python error rather than a segfault). That won't actually help you @ikeboy but it would at least make it more clear that this pattern is invalid (and maybe why).

We do have np.memmap, I wonder if we should consider including an array-from-shared-memory factory in NumPy independently. Although, there are likely good libraries that include that and more.

[seberg]

The core of this issue is identical to gh-23305 so closing.