BUG: Stack overflow on double inheritance from numpy.flexible and numpy.ma.core.MaskedArray

[JelleZijlstra]

Describe the issue:

>>> import numpy
>>> numpy.__version__
'1.24.3'
>>> class X(numpy.flexible, numpy.ma.core.MaskedArray): pass
... 
>>> with memoryview(X()): pass
... 
zsh: segmentation fault  python

Reproduce the code example:

import numpy
class X(numpy.flexible, numpy.ma.core.MaskedArray): pass
with memoryview(X()): pass

Error message:

(lldb) target create "/Users/jelle/py/venvs/py311/bin/python"
Current executable set to '/Users/jelle/py/venvs/py311/bin/python' (arm64).
(lldb) r 
Process 62767 launched: '/Users/jelle/py/venvs/py311/bin/python' (arm64)
Python 3.11.1 (main, Dec 21 2022, 16:19:04) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__version__
'1.24.3'
>>> class X(numpy.flexible, numpy.ma.core.MaskedArray):
...     pass
... 
>>> with memoryview(X()): pass
... 
Process 62767 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x16edfffc0)
    frame #0: 0x00000001a4226994 libsystem_malloc.dylib`nanov2_allocate_from_block + 8
libsystem_malloc.dylib`nanov2_allocate_from_block:
->  0x1a4226994 <+8>:  stp    x28, x27, [sp, #0x20]
    0x1a4226998 <+12>: stp    x26, x25, [sp, #0x30]
    0x1a422699c <+16>: stp    x24, x23, [sp, #0x40]
    0x1a42269a0 <+20>: stp    x22, x21, [sp, #0x50]
Target 0: (python) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x16edfffc0)
  * frame #0: 0x00000001a4226994 libsystem_malloc.dylib`nanov2_allocate_from_block + 8
    frame #1: 0x00000001a42261e0 libsystem_malloc.dylib`nanov2_allocate + 128
    frame #2: 0x00000001a42260fc libsystem_malloc.dylib`nanov2_malloc + 64
    frame #3: 0x00000001a4243748 libsystem_malloc.dylib`_malloc_zone_malloc + 156
    frame #4: 0x00000001000a0998 python`_PyObject_Malloc + 64
    frame #5: 0x00000001001eedcc python`state_init + 164
    frame #6: 0x00000001001ecf1c python`_sre_SRE_Pattern_match + 224
    frame #7: 0x0000000100058a2c python`method_vectorcall_FASTCALL_KEYWORDS_METHOD + 144
    frame #8: 0x000000010004e02c python`PyObject_VectorcallMethod + 144
    frame #9: 0x0000000100104b30 python`check_matched + 108
    frame #10: 0x00000001001035ac python`warn_explicit + 1284
    frame #11: 0x000000010010487c python`do_warn + 828
    frame #12: 0x0000000100102eac python`PyErr_WarnEx + 72
    frame #13: 0x00000001036ada78 _multiarray_umath.cpython-311-darwin.so`PyArray_DescrFromTypeObject + 496
    frame #14: 0x00000001036acbb8 _multiarray_umath.cpython-311-darwin.so`PyArray_DescrFromScalar + 212
    frame #15: 0x000000010009ef28 python`_PyObject_GenericGetAttrWithDict + 200
    frame #16: 0x000000010009e7dc python`PyObject_GetAttr + 100
    frame #17: 0x000000010009e720 python`PyObject_GetAttrString + 72
    frame #18: 0x00000001036acccc _multiarray_umath.cpython-311-darwin.so`PyArray_DescrFromScalar + 488
    frame #19: 0x000000010009ef28 python`_PyObject_GenericGetAttrWithDict + 200
    frame #20: 0x000000010009e7dc python`PyObject_GetAttr + 100
    frame #21: 0x000000010009e720 python`PyObject_GetAttrString + 72
    frame #22: 0x00000001036acccc _multiarray_umath.cpython-311-darwin.so`PyArray_DescrFromScalar + 488
    frame #23: 0x000000010009ef28 python`_PyObject_GenericGetAttrWithDict + 200
    frame #24: 0x000000010009e7dc python`PyObject_GetAttr + 100
    frame #25: 0x000000010009e720 python`PyObject_GetAttrString + 72
(lots more of this, snipped)
    frame #279490: 0x00000001036acccc _multiarray_umath.cpython-311-darwin.so`PyArray_DescrFromScalar + 488
    frame #279491: 0x000000010009ef28 python`_PyObject_GenericGetAttrWithDict + 200
    frame #279492: 0x000000010009e7dc python`PyObject_GetAttr + 100
    frame #279493: 0x0000000100138d50 python`_PyEval_EvalFrameDefault + 17896
    frame #279494: 0x00000001001346c8 python`_PyEval_Vector + 200
    frame #279495: 0x000000010004e300 python`object_vacall + 256
    frame #279496: 0x000000010004e564 python`PyObject_CallFunctionObjArgs + 48
    frame #279497: 0x0000000103603278 _multiarray_umath.cpython-311-darwin.so`PyArray_NewFromDescr_int + 1268
    frame #279498: 0x00000001035fa958 _multiarray_umath.cpython-311-darwin.so`PyArray_View + 84
    frame #279499: 0x0000000103688110 _multiarray_umath.cpython-311-darwin.so`array_view + 324
    frame #279500: 0x000000010005867c python`method_vectorcall_FASTCALL_KEYWORDS + 136
    frame #279501: 0x000000010004ccd4 python`PyObject_Vectorcall + 80
    frame #279502: 0x000000010013cf1c python`_PyEval_EvalFrameDefault + 34740
    frame #279503: 0x00000001001346c8 python`_PyEval_Vector + 200
    frame #279504: 0x000000010004c64c python`_PyObject_FastCallDictTstate + 272
    frame #279505: 0x000000010004d394 python`_PyObject_Call_Prepend + 148
    frame #279506: 0x00000001000b6b68 python`slot_tp_new + 88
    frame #279507: 0x00000001000b3550 python`type_call + 84
    frame #279508: 0x000000010004c834 python`_PyObject_MakeTpCall + 344
    frame #279509: 0x000000010013cf1c python`_PyEval_EvalFrameDefault + 34740
    frame #279510: 0x0000000100134598 python`PyEval_EvalCode + 272
    frame #279511: 0x000000010018c5a8 python`PyRun_InteractiveOneObjectEx + 744
    frame #279512: 0x000000010018bad0 python`_PyRun_InteractiveLoopObject + 152
    frame #279513: 0x000000010018b97c python`_PyRun_AnyFileObject + 96
    frame #279514: 0x000000010018c234 python`PyRun_AnyFileExFlags + 68
    frame #279515: 0x00000001001ac1dc python`Py_RunMain + 2196
    frame #279516: 0x00000001001ac424 python`pymain_main + 324
    frame #279517: 0x00000001001ac4c4 python`Py_BytesMain + 40
    frame #279518: 0x000000010057508c dyld`start + 520

Runtime information:

>>> import sys, numpy; print(numpy.__version__); print(sys.version)
1.24.3
3.11.1 (main, Dec 21 2022, 16:19:04) [Clang 14.0.0 (clang-1400.0.29.202)]
>>> print(numpy.show_runtime())
[{'simd_extensions': {'baseline': ['NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD'],
                      'found': ['ASIMDHP', 'ASIMDDP'],
                      'not_found': ['ASIMDFHM']}},
 {'architecture': 'armv8',
  'filepath': '/Users/jelle/py/venvs/py311/lib/python3.11/site-packages/numpy/.dylibs/libopenblas64_.0.dylib',
  'internal_api': 'openblas',
  'num_threads': 8,
  'prefix': 'libopenblas',
  'threading_layer': 'pthreads',
  'user_api': 'blas',
  'version': '0.3.21'}]
None

Context for the issue:

No concrete use case; I found this while trying to instantiate all pairs of buffer classes to look for real-world impact of python/cpython#104297.

[seberg]

Thanks! numpy.flexible is an abstract base class really (or maybe should be one). It should not be valid to subclass it in Python at all.
(I am a bit surprised nothing prevents this, since masked array inherits from ndarray and ndarray and flexible have incompatible layouts.)

[JelleZijlstra]

The memoryview() part in my original report was unnecessary; just instantiating the class is enough:

>>> class X(numpy.flexible, numpy.ma.core.MaskedArray): pass
... 
>>> X()
zsh: segmentation fault  python

[mattip]

@seberg could you expand on "numpy.flexible is an abstract base class" and "It should not be valid to subclass it in Python"? I think abstract base classes are designed to be subclasses, no? Or do you mean numpy.flexible is an internal NumPy class, and should not be exposed to users. In any case, removing the ability to create a subclass of numpy.flexible should break this line, and allowing it seems like a bug in CPython.

numpy/numpy/core/src/multiarray/multiarraymodule.c

Line 464 in 250e147

Reopening for discussion.

[seberg]

I think CPython is fine with allowing it in C presuming you know what you do, but not sure it is intentional or not.

Yes, I suppose you are right. Subclassing it makes in theory sense (honestly, I don't think it does in practice, you cannot have a flexible user dtype in the old machinery anyway).

So yes, it makes a bit more sense to allow it, but move the actual buffer stuff to the concrete subclasses, becasuse there is no reason to have anything related in the superclass. I am not even sure if it is used there, so maybe it can just be deleted from there.