Gil locking protocol questions

[bitbugprime]

What is the state of the GIL (locked or unlocked) after pybind11::initialize_interpreter()? The docs don't say.

It is not entirely clear when you must acquire and release the GIL when there are multiple C++ threads calling python functions. In my case, a C++ thread wants to call a python function. That python function may end up calling one or more other C++ functions (usually C++ class methods bound to python like str or getitem). Maximum parallelism would of course be nice, what's the best way to treat the GIL to achieve it?

The docs say the GIL is always locked when calling C++ from python. How about the reverse? Must you always first acquire the GIL in C++ before calling a python function from C++ or is it automatic?

It's also not clear what is safe to access or mutate in a C++ function, especially one called from python. There are both pybind11 objects given as arguments to that function (like slices or kwargs) and pybind11 objects being created only in the scope of the C++ function itself, sometimes then returned from it. Can these objects be accessed outside the scope of a GIL lock (for example, when a call_guard attribute is set in the function definition to automatically unlock it on function entry)? How about C++ code not called from python that nevertheless manipulates pybind11 objects, such as might happen during application setup?

[junyer]

I agree, pybind11 has some subtle, but sharp edges here. In my case, I had C++ functions that took a py::bytes and had a py::call_guard<py::gil_scoped_release>. ThreadSanitizer found data races because the GIL guards object reference counts.

[bstaletic]

What is the state of the GIL (locked or unlocked) after pybind11::initialize_interpreter()?

The GIL is held after initializing the interpreter.

It is not entirely clear when you must acquire and release the GIL when there are multiple C++ threads calling python functions.

You have to hold the GIL before touching python.

The docs say the GIL is always locked when calling C++ from python. How about the reverse? Must you always first acquire the GIL in C++ before calling a python function from C++ or is it automatic?

You have to acquire the gil. You can utilize py::gil_scoped_release and/or py::call_guard to automate it.

Pybind11 types

These store a PyObject* objects. Once again, you must not touch python without the gil.

How about C++ code not called from python that nevertheless manipulates pybind11 objects, such as might happen during application setup?

Same as the above.

I agree, pybind11 has some subtle, but sharp edges here. In my case, I had C++ functions that took a py::bytes and had a py::call_guardpy::gil_scoped_release. ThreadSanitizer found data races because the GIL guards object reference counts.

If you're very careful, you can do this safely. For example, you must not examine that py::bytes object if you released the GIL.

[YannickJadoul]

I wish to add to this the argument that pybind11 is not doing anything special here. This is just the way things work in the CPython C API, and all pybind11 does is provide some nice C++ wrappers.