Running multiple GIL in the same process

[llacroix]

Question

Is it possible to run multiple GIL in the same process or there can only be one GIL at a time... Here's the thing.

I'm running Python from within Rust. The goal I'm trying to achieve is to start multiple python interpreter under the same process. As a result I'd be able to spawn multiple threads in which each of them own a GIL. The threads wouldn't need to share data so in theory each interpreter could run in their own thread without issues.

So I did something like this:

• spawn multiple threads
  • each threads acquire a lock
  • load the application config
  • execute a few requests using wsgi
• join all threads

What I'm noticing thought is that the configuration file is loaded only once and all of the other threads seems to share the same python environment. What I was expecting to see is have the config file being loaded N times where N is the amount of threads.

What I'm trying to achieve is a multithreaded wsgi server. Each request could be dispatched to any of the thread without having to worry about the GIL.
The end goal would be to dispatch asynchronously jobs so each GIL can handle multiple requests while the GIL is freed internally... And mutliple GIL could run in parallel.

[llacroix]

Looking at the Python C API... I don't think what I'm trying to achieve is possible. While it's possible to start different interpreters with https://docs.python.org/3/c-api/init.html#c.Py_NewInterpreter
When looking at https://docs.python.org/3/c-api/init.html#c.PyInterpreterState
We can read this:

Threads belonging to different interpreters initially share nothing, except process state like available memory, open file descriptors and such. The global interpreter lock is also shared by all threads, regardless of to which interpreter they belong.

So from the look of it, even if it was possible to create a new Interpreter context, in a different thread... The GIL is shared against all threads and all python interpreters. 😞

So as it seems, the only way to get multiple Python interpreters with multiple GILs is to have multiple processes. Well that's kinda sad they went for such a design. Having multiple runtime could allow cool things like sending objects in a message queue shared by multiple runtimes or in my case having simply many runtime working in parallel without blocking on the GIL of other threads...

[birkenfeld]

The GIL is shared against all threads and all python interpreters.

Currently yes, but it is being discussed, see for example this recent thread:

https://mail.python.org/archives/list/python-dev@python.org/thread/ZSE2G37E24YYLNMQKOQSBM46F7KLAOZF/

[davidhewitt]

👍 thanks for that link. I think if the final design of sub-interpreters does allow each to have an individual GIL then it would be very interesting to support this in pyo3.

[llacroix]

Thanks, may be there's hope after all.. I read the PEP554 and I wouldn't expect a GIL per interpreter just yet... I guess the point is to have PEP554 ready to make multi interpreters more available. Then having multiple GILs could become implemented in the future in order to have a usable API and give native package the time to adapt their codebase for the changes required to be multi interpreter safe.

That said, I might not send my wsgi toy server to the bin yet.

[davidhewitt]

Cool. For the moment I'd be wary of trying sub-interpreters with pyO3. It's such a big change to some of the ways core things like type objects should behave, so I suspect we don't always play by the rules. We probably won't get around to proper support until PEP 554 is finalized.

[davidhewitt]

Closing as duplicate of #576