segfault when destructing class instance containing loaded py::module and py::scoped_interpreter

[tmplt]

Issue description

Example code segfault with the following backtrace instead of exiting normally.

#0  0x00007ffff79a2d98 in dict_dealloc ()
   from /nix/store/lp7x6ziq5b2zihgad0i8a28xxvz5vnrr-python3-3.6.4/lib/libpython3.6m.so.1.0
#1  0x00007ffff79b0ea9 in module_dealloc ()
   from /nix/store/lp7x6ziq5b2zihgad0i8a28xxvz5vnrr-python3-3.6.4/lib/libpython3.6m.so.1.0
#2  0x0000000000404620 in main () at /home/tmplt/code/bookwyrm/unit-test/main.cpp:40

Omitting the call to plugin_handler::load_plugins() makes the example quit normally.
Code was compiled with GCC 7.3.0, using pybind11 fd9bc8f (current master HEAD).

Reproducible example code

#include <iostream>
#include <pybind11/embed.h>
namespace py = pybind11;

PYBIND11_EMBEDDED_MODULE(fast_calc, m) {
    // `m` is a `py::module` which is used to bind functions and classes
    m.def("add", [](int i, int j) {
        return i + j;
    });
}

class __attribute__ ((visibility("hidden"))) plugin_handler {
public:
    void load_plugins()
    {
        plugin = py::module::import("fast_calc");
        std::cout << "plugin loaded.\n";
    }

    ~plugin_handler()
    {
        std::cout << "destructing...\n";
    }

private:
    py::module plugin;
    py::scoped_interpreter interp;
};

int main()
{
    {
        /* This works without issue. */
        py::scoped_interpreter interp;
        py::module plugin = py::module::import("fast_calc");
    }

    /* But this causes a segfault, even when above block is omitted. */
    plugin_handler ph;
    ph.load_plugins();

    return 0;
}

[yeganer]

Does switching the order of the private definitions change the behavior?

private:
    py::scoped_interpreter interp;
    py::module plugin;

[jagerman]

Try swapping the order of these two members:

    py::module plugin;
    py::scoped_interpreter interp;

Destruction occurs in reverse order of declaration, so as currently written the py::module destruction happens after the interp destruction.

[tmplt]

Destruction occurs in reverse order of declaration

Ah, this explains it (and fixes the issue). It does bring another issue back that I thought "fixed" by getting a segfault instead. With the plugin system I'm building in the repo referenced above, I want to fire off some worker threads and just detach them when I no longer need them. Detaching these threads (running Python scripts via pybind11) and destructing the py::scoped_interpreter eventually leads to a scoped_acquire::dec_ref(): thread state must be current!; I'll have to create another issue and minimal example for that.

Any case, this particular issue can be closed.

[jagerman]

That sounds like a similar issue: a py::object (or related) instance outliving the interpreter.

[tmplt]

Closing in favour of #1360.