segfault when importing numpy after reinitializing Python

[jcmuel]

I embedded Python into a C program. At certain points in time, the C program resets the interpreter by calling Py_Finalize + Py_Initialize. When the evaluated code imports NumPy before and after the reset, then the second import fails with a segmentation fault.

Reproducing code example:

At first the example C program initializes Python, imports NumPy, and then finalizes Python.
Then it reinitializes Python and imports NumPy again. When importing NumPy
for the second time, the import fails with a segmentation fault.

Tested with Debian 9, Python 3.7.0 valgrind friendly debug build, NumPy debug build of latest
version from github (1.16.0.dev0+e796449). The bug reproduces on Mac (tested with Python 3.5.1,
NumPy 1.11.0) and Windows (Tested with Python 3.7.0 from Python.org, NumPy 1.14.5).

/******************************************************************************
    Test: Initialize NumPy before and after Python reinitialization
******************************************************************************/

#include "Python.h"

int main(int arc, char *argv[]) {
    wchar_t *program = Py_DecodeLocale(argv[0], NULL);
    if (program == NULL) {
        fprintf(stderr, "Fatal error: cannot decode argv[0]\n");
        exit(1);
    }

    Py_SetProgramName(program);

    // Initialize Python, import NumPy, finalize Python. First time.
    Py_Initialize();
    PyRun_SimpleString("import sys; print('Python', sys.version)");
    PyRun_SimpleString("import numpy; print('Numpy', numpy.__version__)");
    Py_Finalize();

    // Initialize Python, import NumPy, finalize Python. Second time.
    Py_Initialize();
    PyRun_SimpleString("import numpy;"); // <-- SEGMENTATION FAULT.
    Py_Finalize();

    PyMem_RawFree(program);
    return 0;
}

Error message:

Segmentation fault on evaluation of "import numpy" after reinitialization of Python. Output of valgrind:

==4346== Memcheck, a memory error detector
==4346== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==4346== Using Valgrind-3.12.0.SVN and LibVEX; rerun with -h for copyright info
==4346== Command: ./main
==4346==
Python 3.7.0 (default, Aug 27 2018, 19:39:47)
[GCC 6.3.0 20170516]
Numpy 1.16.0.dev0+e796449
==4346== Invalid read of size 8
==4346==    at 0x6CEFDC4: PyArray_Item_INCREF (refcount.c:35)
==4346==    by 0x6CF37A8: PyArray_FromScalar (scalarapi.c:335)
==4346==    by 0x6CF5674: gentype_nonzero_number (scalartypes.c.src:349)
==4346==    by 0x4F12AA8: PyObject_IsTrue (object.c:1384)
==4346==    by 0x4FC1995: _PyEval_EvalFrameDefault (ceval.c:2654)
==4346==    by 0x4FB7EE1: PyEval_EvalFrameEx (ceval.c:547)
==4346==    by 0x4EC7C0C: function_code_fastcall (call.c:283)
==4346==    by 0x4EC87CF: _PyFunction_FastCallKeywords (call.c:408)
==4346==    by 0x4FC42FD: call_function (ceval.c:4586)
==4346==    by 0x4FC42FD: _PyEval_EvalFrameDefault (ceval.c:3117)
==4346==    by 0x4FB7EE1: PyEval_EvalFrameEx (ceval.c:547)
==4346==    by 0x4EC7C0C: function_code_fastcall (call.c:283)
==4346==    by 0x4EC846B: _PyFunction_FastCallDict (call.c:322)
==4346==  Address 0x1a is not stack'd, malloc'd or (recently) free'd
==4346==
==4346==
==4346== Process terminating with default action of signal 11 (SIGSEGV)
==4346==  Access not within mapped region at address 0x1A
==4346==    at 0x6CEFDC4: PyArray_Item_INCREF (refcount.c:35)
==4346==    by 0x6CF37A8: PyArray_FromScalar (scalarapi.c:335)
==4346==    by 0x6CF5674: gentype_nonzero_number (scalartypes.c.src:349)
==4346==    by 0x4F12AA8: PyObject_IsTrue (object.c:1384)
==4346==    by 0x4FC1995: _PyEval_EvalFrameDefault (ceval.c:2654)
==4346==    by 0x4FB7EE1: PyEval_EvalFrameEx (ceval.c:547)
==4346==    by 0x4EC7C0C: function_code_fastcall (call.c:283)
==4346==    by 0x4EC87CF: _PyFunction_FastCallKeywords (call.c:408)
==4346==    by 0x4FC42FD: call_function (ceval.c:4586)
==4346==    by 0x4FC42FD: _PyEval_EvalFrameDefault (ceval.c:3117)
==4346==    by 0x4FB7EE1: PyEval_EvalFrameEx (ceval.c:547)
==4346==    by 0x4EC7C0C: function_code_fastcall (call.c:283)
==4346==    by 0x4EC846B: _PyFunction_FastCallDict (call.c:322)
==4346==  If you believe this happened as a result of a stack
==4346==  overflow in your program's main thread (unlikely but
==4346==  possible), you can try to increase the size of the
==4346==  main thread stack using the --main-stacksize= flag.
==4346==  The main thread stack size used in this run was 8388608.
==4346==
==4346== HEAP SUMMARY:
==4346==     in use at exit: 6,190,252 bytes in 34,347 blocks
==4346==   total heap usage: 215,244 allocs, 180,897 frees, 45,494,245 bytes allocated
==4346==
==4346== LEAK SUMMARY:
==4346==    definitely lost: 144 bytes in 2 blocks
==4346==    indirectly lost: 0 bytes in 0 blocks
==4346==      possibly lost: 6,055,624 bytes in 31,510 blocks
==4346==    still reachable: 134,484 bytes in 2,835 blocks
==4346==         suppressed: 0 bytes in 0 blocks
==4346== Rerun with --leak-check=full to see details of leaked memory
==4346==
==4346== For counts of detected and suppressed errors, rerun with: -v
==4346== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Killed

Numpy/Python version information:

Python 3.7.0 (default, Aug 27 2018, 19:39:47) [GCC 6.3.0 20170516]
Numpy 1.16.0.dev0+e796449

[charris]

I believe this is a known problem with Python. @njsmith Comment?

[jcmuel]

It's a know issue that "dynamically loaded extension modules loaded by Python are not unloaded" and " some extensions may not work properly if their initialization routine is called more than once; this can happen if an application calls Py_Initialize() and Py_FinalizeEx() more than once." [Py_FinalizeEx documentation]

It seems like NumPy is one of those extension modules that crashes when the initialization function is called twice. As long as I don't use NumPy, everything works (or seems to work) as expected after resetting the interpreter.

[jcmuel]

There is already a similar bug report regarding reimport of NumPy after reinitialization: #8097

[jcmuel]

Is my understanding correct that the root cause of the problem is that parts of NumPy do not implement the

• module cleanup functions (PEP 3121) and do not implement
• multi-phase extension module initialization (PEP 0489)?

For example the _multiarray_umath module, does not support it, as we can see from the NULLs in the module definition in multiarraymodule.c:

static struct PyModuleDef moduledef = {
        PyModuleDef_HEAD_INIT,
        "_multiarray_umath",
        NULL,
        -1,
        array_module_methods,
        NULL,
        NULL,
        NULL,
        NULL
};

Are there plans to use multi-phase initialization in the future?