Crash when importing numpy from the Python C-API after calling Py_Finilize()

[phelippeneveu]

Hi,

We are getting a crash in the numpy code when importing numpy after Py_Finalize() is called. If we comment out the call to Py_Finalize() the second import will be successful.

#include "python2.7/Python.h"

int RunPythonScript(const char * pszScript)
{
    Py_Initialize();

    int nResult = PyRun_SimpleStringFlags(pszScript, NULL);

    Py_Finalize();

    return nResult;
}

int main(void)
{
    RunPythonScript("import numpy");

    printf("First import of numpy ran successfully!\n");

    RunPythonScript("import numpy");

    printf("Second import of numpy ran successfully!\n");

    return 0;
}

I compiled this example using GCC: gcc -o numpy_crash numpy_crash.cpp -lpython2.7

This simple example should not crash. My guess is that Python will destroy some global objects when Py_Finalize() is called. But these global objects are still referenced in the numpy code. You would need to register an atexit function using Py_AtExit() to reinitialize any global objects to NULL so that they are re-created on subsequent import of numpy.

This bug is most likely the same bug as #656 and #3961. But we are only running a single Python interpreter. We call Py_Finalize() every time a user runs a new Python script so that the variables they defined on the previous runs are no longer available in the Python environment.

Regards,

Phelippe Neveu

[natebazar]

I have encountered the same issue when attempting to embed Python in an application as well running Python 3.5.

[njsmith]

I don't think anyone would object to receiving a fix for this, but it would require a tiresome whole-codebase audit to find all the places numpy caches state, which is unlikely to be a priority for most contributors. So you should probably assume that it's not going to happen unless you do it.

[den-run-ai]

@njsmith just FYI, i spent ~15 minutes looking at this and this issue has been reported many times over past 10 years in various threads with no clear resolution:

pythonnet/pythonnet#595

Perhaps numpy could provide its own finalization/cleanup method so that this can be called before finalizing the Python interpreter?

[pv]

It seems to me nobody has *really* needed a fix for this, since nobody has come forward with work towards it or a patch.

[phelippeneveu]

Our current work around is to not call Py_Finalize() at all. I also added some code that frees attributes from the main module so that the variables defined in a previous run are not available in a future run.

`

PyObject * poMainModule = PyImport_AddModule("__main__");

PyObject * poAttrList = PyObject_Dir(poMainModule);

PyObject * poAttrIter = PyObject_GetIter(poAttrList);

PyObject * poAttrName;

while ((poAttrName = PyIter_Next(poAttrIter)) != NULL)
{
    std::string oAttrName = GetUnicodeString(poAttrName);

    // Make sure we don't delete any private objects.
    if (!StartsWith(oAttrName, "__") || !EndsWith(oAttrName, "__"))
    {
        PyObject * poAttr = PyObject_GetAttr(poMainModule, poAttrName);

        // Make sure we don't delete any module objects.
        if (poAttr && poAttr->ob_type != poMainModule->ob_type)
            PyObject_SetAttr(poMainModule, poAttrName, NULL);

        Py_DecRef(poAttr);
    }

    Py_DecRef(poAttrName);
}

Py_DecRef(poAttrIter);

Py_DecRef(poAttrList);

`

Regards,

Phelippe

[gyp33]

Hi,
I experience exactly the same behavior as Phelippe. I embed execution of python scripts into a C application. If numpy is imported in the script, the application crashes the second time the script is executed.
Each time a script is run, pyInitialize and pyFinalize are called.
The only way to make it work is to stop and restart our application before running the script a second time which is not the way it should work.
It would be nice to have a fix.
Regards,
Alain

[AndreaLazz]

I have same problem as Phelippe.
I need to unload/reload python scripts (python version 2.7.14) using numpy library (it's a mathematical simulation application).
If Phelippe can explain better his workaround I can try to use it in the meantime:

PyObject_Dir(poMainModule) => need to get poMainModule before
GetUnicodeString,StartsWith,EndsWith => are customized functions or defined in some Windows or python library?

[Takishima]

Phelippe's workaround basically lists all the objects accessible in the main module and deletes them one by one provided that they:

• do not start or end with '__'
• are not of module type

One way of getting the main module would be:

 PyObject* poMainModule = PyImport_ImportModule("__main__");
 if (poMainModule != NULL) {
         // Rest of code here...
 }

The GetUnicodeString, StartsWith and EndsWith are probably custom functions as I couldn't find any Windows or Python documentation on them (except in the .NET framework doc which should not be relevant here).

EDIT:

You might want to take a look at the PyString_AsString function from the Python C API

[phelippeneveu]

One of the reason why we wanted to call Py_Finalize() was to make sure that if the user defined variables in a previous run, that they were not available in subsequent runs. The only way I found to do this is to loop through all of the main module's attributes and delete them all except the module objects and the private objects that starts or ends with the "__" string.

If we don't do this, then the second run can use a variable defined from the first run.

Example:

PyRun_SimpleStringFlags("print test", NULL); // Error: test variable not defined.
...
PyRun_SimpleStringFlags("test = 12", NULL); // Defines the test variable.
...
PyRun_SimpleStringFlags("print test", NULL); // Prints 12 since the variable is still defined from the previous run because we haven't called Py_Finalize().

Sorry if my work around example contains non-included functions. The StartsWith() and EndsWith() functions simply checks if the string starts with __ or ends with __. The boost::starts_with() and boost::ends_with() functions could be used instead. The GetUnicodeString() function is simply a function that returns a UTF8 string, it uses PyUnicode_AsWideChar() or PyString_AsString() depending on the object type.

To create the main module, I use:
PyObject * poMainModule = PyImport_AddModule("__main__");
Regards,

Phelippe

[AndreaLazz]

Thank you for your information, Phelippe.
I hope that numpy developers finally will take a look and solve this issue.
In the meantime a good workaround like this can be tried and applied.

[phelippeneveu]

One of the problem with the work around is that if a module is imported on the first run, it will not be imported on subsequent runs. So if the user imports a in-house script, but changes it to fix an issue, he would need to close the your application and start it again for his changes to take effect.

[Archloy]

@phelippeneveu Yes, but with importlib.relaod() he can do the job (with some hack in main script ?)

EDIT : error in function name, sorry =)

[loopless]

I am struggling with the same issue. It does not just apply to numpy...
One issue is that Python itself does not unload the module DLLs on PyFinalize(). Unlike , say Matlab, where 'clear mex' will actually unload any loaded MEX DLL's. PyFinalize should unload all the .pyd (DLLs). That would truly set the modules back to initial state when they are loaded again.
Just to clarify.
Assume your DLL (.pyd) has some 'C' code with a variable
static int myImportantInitialCounter=0;
And inside the DLL that variable is incremented and used.
After a PyFinalize(), that value will remain at it's incremented value, even after a reload of the python module as the DLL module is still loaded in memory.
If PyFinalize properly unloaded all loaded DLL modules then, after a PyInitialize, the next time the python module was loaded the variable would be back to 0 when the DLL module is loaded and initialized.

[njsmith]

@loopless If you want to propose changes to how Python handles extension modules then you should talk to the Python team, not us :-)