Idea: acquire the GIL in all API functions

[cjdoris]

The Problem

Currently none of our API touches the GIL, except for the explicit GIL handling in PythonCall.GIL. This means that users must ensure that all API functions are only called on the thread holding the GIL, which is normally the main thread.

However this makes it hard to do anything in a multithreaded setting. You can try to use GIL.@lock etc to ensure a thread holds the GIL but this is fiddly to get right. Especially as Julia can migrate tasks between threads, so the threadid may change between acquiring and releasing the lock.

Sometimes you don't know that your code will be run in a multithreaded setting. An annoying case of this is that on Julia 1.12+, it will by default launch 1 interactive and 1 default thread. The REPL runs in the interactive thread but auto-completions runs in the default thread. This means that as soon as you type julia> np. it will crash because propertynames(::Py) is called on the wrong thread.

The Proposal

Have all of the API functions explicitly acquire the GIL and release it again when they are done. This will require releasing the GIL at the end of __init__.

This will mean you can call any PythonCall API function from any thread and things should "just work", with the caveat that the GIL will force any parallel Python calls to be performed serially.

Considerations

• How much of a performance impact will this have? Hopefully PyGILState_Ensure is fast if the GIL is already held, so that any larger blocks of code inside a GIL.@lock are OK.
• Can we really ensure that the threadid doesn't change between acquiring and releasing the GIL? We can at least record the threadid when we lock and check it when we unlock, or before any other calls into CPython. We should have the lock as tight as possible around the corresponding CPython function.
• We should introduce a re-entrant lock that works in tandem with the GIL, otherwise we will quickly deadlock a thread. This will have the downside that if the GIL is released Python-side, this other lock will not be released, so PythonCall functions will stay waiting.
• We'll need to make everything else thread-safe too - e.g. all the globals should have locks.
• We can make unsafe versions of the API functions which don't handle the GIL, which can be used to compose higher-level functions which do handle the GIL.
• In particular @py could acquire the GIL once and use these unsafe functions.
• Consider changing GIL.@unlock to not do anything if the GIL is already unlocked. Currently it assumes the GIL is held.
• Is this breaking? Should it wait for v1?