From e488387d5df7c37ace4e053b683a2c4e6a8eff21 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Sun, 12 Oct 2025 11:18:48 -0400 Subject: [PATCH 1/8] Rename to PyInterpreterGuard. --- peps/pep-0788.rst | 189 +++++++++++++++++++++++----------------------- 1 file changed, 95 insertions(+), 94 deletions(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index cf96c1928dd..290a163f53c 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -24,13 +24,13 @@ interpreter. For example: static int thread_function(PyInterpreterView view) { - PyInterpreterLock lock = PyInterpreterLock_FromView(view); - if (lock == 0) { + PyInterpreterGuard guard = PyInterpreterGuard_FromView(view); + if (guard == 0) { return -1; } - PyThreadView thread_view = PyThreadState_Ensure(lock); + PyThreadView thread_view = PyThreadState_Ensure(guard); if (thread_view == 0) { - PyInterpreterLock_Release(lock); + PyInterpreterGuard_Release(guard); return -1; } @@ -38,7 +38,7 @@ interpreter. For example: finalization. */ PyThreadState_Release(thread_view); - PyInterpreterLock_Release(); + PyInterpreterGuard_Release(guard); return 0; } @@ -252,18 +252,18 @@ Rationale Preventing Interpreter Shutdown ------------------------------- -This PEP takes an approach where an interpreter comes with a locking API -that prevents it from shutting down. Holding an interpreter lock will make it +This PEP takes an approach where an interpreter comes with a guarding API +that prevents it from shutting down. Holding an interpreter guard will make it safe to call the C API without worrying about the thread being hung. This means that interfacing Python (for example, in a C++ library) will need -a lock to the interpreter in order to safely call the object, which is more +a guard to the interpreter in order to safely call the object, which is more inconvenient than assuming the main interpreter is the right choice, but there's not really another option. This proposal also comes with "views" to an interpreter that can be used to safely poke at an interpreter that may be dead or alive. Using a view, users -can acquire an interpreter lock at any point during its lifecycle, and +can acquire an interpreter guard at any point during its lifecycle, and will safely fail if the interpreter can no longer support calling Python code. Compatibility Shim for ``PyGILState_Ensure`` @@ -271,50 +271,50 @@ Compatibility Shim for ``PyGILState_Ensure`` This proposal comes with :c:func:`PyUnstable_InterpreterView_FromDefault` as a compatibility hack for some users of :c:func:`PyGILState_Ensure`. It is a -thread-safe way to acquire a lock to the main (or "default") +thread-safe way to acquire a guard for the main (or "default") interpreter. The main drawback to porting new code to :c:func:`PyThreadState_Ensure` is that it isn't a drop-in replacement for :c:func:`!PyGILState_Ensure`, as it needs -an interpreter lock argument. In some large applications, refactoring to -use a :c:type:`PyInterpreterLock` everywhere might be tricky; so, this function +an interpreter guard argument. In some large applications, refactoring to +use a :c:type:`PyInterpreterGuard` everywhere might be tricky; so, this function acts as a last resort for users who explicitly want to disallow support for subinterpreters. Specification ============= -Interpreter Locks +Interpreter Guards ----------------- -.. c:type:: PyInterpreterLock +.. c:type:: PyInterpreterGuard - An opaque interpreter lock. + An opaque interpreter guard. - By holding an interpreter lock, the caller can know that the interpreter + By holding an interpreter guard, the caller can know that the interpreter will be in a state where it can safely execute Python code. - This is a special type of "readers-writers" lock; threads may hold an - interpreter's lock concurrently, and the interpreter will have to wait - until all threads have released the lock until it can enter finalization. + This is similar to a "readers-writers" lock; threads may hold an + interpreter's guard concurrently, and the interpreter will have to wait + until all threads have released the guard until it can enter finalization. This type is guaranteed to be pointer-sized. -.. c:function:: PyInterpreterLock PyInterpreterLock_FromCurrent(void) +.. c:function:: PyInterpreterGuard PyInterpreterGuard_FromCurrent(void) - Acquire a lock for the current interpreter. + Acquire a guard for the current interpreter. - On success, this function locks the interpreter and returns an opaque - reference to the lock, or returns ``0`` with an exception set on failure. + On success, this function guards the interpreter and returns an opaque + reference to the guard, or returns ``0`` with an exception set on failure. The caller must hold an :term:`attached thread state`. -.. c:function:: PyInterpreterLock PyInterpreterLock_FromView(PyInterpreterView view) +.. c:function:: PyInterpreterGuard PyInterpreterGuard_FromView(PyInterpreterView view) - Acquire a lock to an interpreter through a view. + Acquire a guard to an interpreter through a view. - On success, this function returns a lock to the interpreter + On success, this function returns a guard to the interpreter denoted by *view*. The view is still valid after calling this function. @@ -324,25 +324,26 @@ Interpreter Locks The caller does not need to hold an :term:`attached thread state`. -.. c:function:: PyInterpreterState *PyInterpreterLock_GetInterpreter(PyInterpreterLock lock) +.. c:function:: PyInterpreterState *PyInterpreterGuard_GetInterpreter(PyInterpreterGuard guard) - Return the :c:type:`PyInterpreterState` pointer denoted by *lock*. + Return the :c:type:`PyInterpreterState` pointer denoted by *guard*. This function cannot fail, and the caller doesn't need to hold an :term:`attached thread state`. -.. c:function:: PyInterpreterLock PyInterpreterLock_Copy(PyInterpreterLock lock) +.. c:function:: PyInterpreterGuard PyInterpreterGuard_Copy(PyInterpreterGuard guard) - Duplicate a lock to an interpreter. + Duplicate an interpreter guard. - On success, this function returns a lock to the interpreter - denoted by *lock*, and returns ``0`` without an exception set on failure. + On success, this function returns a copy of *guard*, and returns ``0`` + without an exception set on failure. The caller does not need to hold an :term:`attached thread state`. -.. c:function:: void PyInterpreterLock_Release(PyInterpreterLock lock) +.. c:function:: void PyInterpreterGuard_Release(PyInterpreterGuard guard) - Release an interpreter's lock, possibly allowing it to shut down. + Release an interpreter guard, possibly allowing the interpreter to shut + down if no guards remain. This function cannot fail, and the caller doesn't need to hold an :term:`attached thread state`. @@ -364,7 +365,7 @@ Interpreter Views Create a view to the current interpreter. This function is generally meant to be used in tandem with - :c:func:`PyInterpreterLock_FromView`. + :c:func:`PyInterpreterGuard_FromView`. On success, this function returns a view to the current interpreter, and returns ``0`` with an exception set on failure. @@ -418,10 +419,10 @@ replace :c:func:`PyGILState_Ensure` and :c:func:`PyGILState_Release`. This type is guaranteed to be pointer-sized. -.. c:function:: PyThreadView PyThreadState_Ensure(PyInterpreterLock lock) +.. c:function:: PyThreadView PyThreadState_Ensure(PyInterpreterGuard guard) Ensure that the thread has an :term:`attached thread state` for the - interpreter denoted by *lock*, and thus can safely invoke that + interpreter denoted by *guard*, and thus can safely invoke that interpreter. It is OK to call this function if the thread already has an attached thread state, as long as there is a subsequent call to :c:func:`PyThreadState_Release` that matches this one. @@ -429,16 +430,16 @@ replace :c:func:`PyGILState_Ensure` and :c:func:`PyGILState_Release`. Nested calls to this function will only sometimes create a new :term:`thread state`. If there is no attached thread state, then this function will check for the most recent attached thread - state used by this thread. If none exists or it doesn't match *lock*, - a new thread state is created. If it does match *lock*, it is reattached. + state used by this thread. If none exists or it doesn't match *guard*, + a new thread state is created. If it does match *guard*, it is reattached. If there is an attached thread state, then a similar check occurs; - if the interpreter matches *lock*, it is attached, and otherwise a new + if the interpreter matches *guard*, it is attached, and otherwise a new thread state is created. Return a non-zero thread view of the old thread state on success, and ``0`` on failure. -.. c:function:: void PyThreadState_Release(PyThreadView lock) +.. c:function:: void PyThreadState_Release(PyThreadView view) Release a :c:func:`PyThreadState_Ensure` call. @@ -508,15 +509,15 @@ With this PEP, you would implement it like this: PyObject *file, PyObject *text) { - PyInterpreterLock lock = PyInterpreterLock_FromView(view); - if (lock == 0) { + PyInterpreterGuard guard = PyInterpreterGuard_FromView(view); + if (guard == 0) { /* Python interpreter has shut down */ return -1; } - PyThreadView thread_view = PyThreadState_Ensure(lock); + PyThreadView thread_view = PyThreadState_Ensure(guard); if (thread_view == 0) { - PyInterpreterLock_Release(lock); + PyInterpreterGuard_Release(guard); fputs("Cannot call Python.\n", stderr); return -1; } @@ -527,7 +528,7 @@ With this PEP, you would implement it like this: // print out the exception ourself. PyErr_Print(); PyThreadState_Release(thread_view); - PyInterpreterLock_Release(lock); + PyInterpreterGuard_Release(guard); return -1; } int res = PyFile_WriteString(to_write, file); @@ -537,7 +538,7 @@ With this PEP, you would implement it like this: } PyThreadState_Release(thread_view); - PyInterpreterLock_Release(lock); + PyInterpreterGuard_Release(guard); return res < 0; } @@ -556,8 +557,8 @@ held. Any future finalizer that attempted to acquire the lock would be deadlocke my_critical_operation(PyObject *self, PyObject *Py_UNUSED(args)) { assert(PyThreadState_GetUnchecked() != NULL); - PyInterpreterLock lock = PyInterpreterLock_FromCurrent(); - if (lock == 0) { + PyInterpreterGuard guard = PyInterpreterGuard_FromCurrent(); + if (guard == 0) { /* Python interpreter has shut down */ return NULL; } @@ -571,7 +572,7 @@ held. Any future finalizer that attempted to acquire the lock would be deadlocke release_some_lock(); Py_END_ALLOW_THREADS; - PyInterpreterLock_Release(lock); + PyInterpreterGuard_Release(guard); Py_RETURN_NONE; } @@ -619,17 +620,17 @@ This is the same code, rewritten to use the new functions: static int thread_func(void *arg) { - PyInterpreterLock interp = (PyInterpreterLock)arg; - PyThreadView thread_view = PyThreadState_Ensure(interp); + PyInterpreterGuard guard = (PyInterpreterGuard)arg; + PyThreadView thread_view = PyThreadState_Ensure(guard); if (thread_view == 0) { - PyInterpreterLock_Release(interp); + PyInterpreterGuard_Release(guard); return -1; } if (PyRun_SimpleString("print(42)") < 0) { PyErr_Print(); } PyThreadState_Release(thread_view); - PyInterpreterLock_Release(interp); + PyInterpreterGuard_Release(guard); return 0; } @@ -639,13 +640,13 @@ This is the same code, rewritten to use the new functions: PyThread_handle_t handle; PyThead_indent_t indent; - PyInterpreterLock lock = PyInterpreterLock_FromCurrent(); - if (lock == 0) { + PyInterpreterGuard guard = PyInterpreterGuard_FromCurrent(); + if (guard == 0) { return NULL; } - if (PyThread_start_joinable_thread(thread_func, (void *)lock, &ident, &handle) < 0) { - PyInterpreterLock_Release(lock); + if (PyThread_start_joinable_thread(thread_func, (void *)guard, &ident, &handle) < 0) { + PyInterpreterGuard_Release(guard); return NULL; } Py_BEGIN_ALLOW_THREADS @@ -660,7 +661,7 @@ Example: A Daemon Thread With this PEP, daemon threads are very similar to how non-Python threads work in the C API today. After calling :c:func:`PyThreadState_Ensure`, simply -release the interpreter lock to allow the interpreter to shut down (and +release the interpreter guard to allow the interpreter to shut down (and hang the current thread forever). .. code-block:: c @@ -668,15 +669,15 @@ hang the current thread forever). static int thread_func(void *arg) { - PyInterpreterLock lock = (PyInterpreterLock)arg; - PyThreadView thread_view = PyThreadState_Ensure(lock); + PyInterpreterGuard guard = (PyInterpreterGuard)arg; + PyThreadView thread_view = PyThreadState_Ensure(guard); if (thread_view == 0) { - PyInterpreterLock_Release(lock); + PyInterpreterGuard_Release(guard); return -1; } - /* Release the interpreter lock, allowing it to + /* Release the interpreter guard, allowing it to finalize. This means that print(42) can hang this thread. */ - PyInterpreterLock_Release(lock); + PyInterpreterGuard_Release(guard); if (PyRun_SimpleString("print(42)") < 0) { PyErr_Print(); } @@ -690,13 +691,13 @@ hang the current thread forever). PyThread_handle_t handle; PyThead_indent_t indent; - PyInterpreterLock lock = PyInterpreterLock_FromCurrent(); - if (lock == 0) { + PyInterpreterGuard guard = PyInterpreterGuard_FromCurrent(); + if (guard == 0) { return NULL; } - if (PyThread_start_joinable_thread(thread_func, (void *)lock, &ident, &handle) < 0) { - PyInterpreterLock_Release(lock); + if (PyThread_start_joinable_thread(thread_func, (void *)guard, &ident, &handle) < 0) { + PyInterpreterGuard_Release(guard); return NULL; } Py_RETURN_NONE; @@ -716,22 +717,22 @@ Example: An Asynchronous Callback { ThreadData *tdata = (ThreadData *)arg; PyInterpreterView view = tdata->view; - PyInterpreterLock lock = PyInterpreterLock_FromView(view); - if (lock == 0) { + PyInterpreterGuard guard = PyInterpreterGuard_FromView(view); + if (guard == 0) { fputs("Python has shut down!\n", stderr); return -1; } - PyThreadView thread_view = PyThreadState_Ensure(lock); + PyThreadView thread_view = PyThreadState_Ensure(guard); if (thread_view == 0) { - PyInterpreterLock_Release(lock); + PyInterpreterGuard_Release(guard); return -1; } if (PyRun_SimpleString("print(42)") < 0) { PyErr_Print(); } PyThreadState_Release(thread_view); - PyInterpreterLock_Release(lock); + PyInterpreterGuard_Release(guard); PyInterpreterView_Close(view); PyMem_RawFree(tdata); return 0; @@ -762,8 +763,8 @@ Example: Calling Python Without a Callback Parameter **************************************************** There are a few cases where callback functions don't take a callback parameter -(``void *arg``), so it's difficult to acquire a lock to any specific -interpreter. The solution to this problem is to acquire a lock to the main +(``void *arg``), so it's difficult to acquire a guard to any specific +interpreter. The solution to this problem is to acquire a guard for the main interpreter through :c:func:`PyUnstable_InterpreterView_FromDefault`. .. code-block:: c @@ -772,20 +773,20 @@ interpreter through :c:func:`PyUnstable_InterpreterView_FromDefault`. call_python(void) { PyInterpreterView view = PyUnstable_InterpreterView_FromDefault(); - if (lock == 0) { + if (guard == 0) { fputs("Python has shut down.", stderr); return; } - PyInterpreterLock lock = PyInterpreterLock_FromView(view); - if (lock == 0) { + PyInterpreterGuard guard = PyInterpreterGuard_FromView(view); + if (guard == 0) { fputs("Python has shut down.", stderr); return; } - PyThreadView thread_view = PyThreadState_Ensure(lock); + PyThreadView thread_view = PyThreadState_Ensure(guard); if (thread_view == 0) { - PyInterpreterLock_Release(lock); + PyInterpreterGuard_Release(guard); PyInterpreterView_Close(view); return -1; } @@ -793,7 +794,7 @@ interpreter through :c:func:`PyUnstable_InterpreterView_FromDefault`. PyErr_Print(); } PyThreadState_Release(thread_view); - PyInterpreterLock_Release(lock); + PyInterpreterGuard_Release(guard); PyInterpreterView_Close(view); return 0; } @@ -810,13 +811,13 @@ Open Issues How Should the APIs Fail? ------------------------- -There is a bit of disagreement on how the ``PyInterpreter[Lock|View]`` APIs +There is a bit of disagreement on how the ``PyInterpreter[Guard|View]`` APIs should indicate a failure to the caller. There are two competing ideas: 1. Return -1 to indicate failure, and 0 to indicate success. On success, - functions will assign to a ``PyInterpreter[Lock|View]`` pointer passed as an + functions will assign to a ``PyInterpreter[Guard|View]`` pointer passed as an argument. -2. Directly return a ``PyInterpreter[Lock|View]``, which a value of 0 being +2. Directly return a ``PyInterpreter[Guard|View]``, which a value of 0 being equivalent to ``NULL``, indicating failure. Currently, the PEP spells the latter. @@ -845,7 +846,7 @@ areas of their code required/used a strong reference. In response to that pushback, this PEP specified ``PyInterpreterRef`` APIs that would also mimic reference counting, but in a more explicit manner that made it easier upon developers. ``PyInterpreterRef`` was analogous to -:c:type:`PyInterpreterLock` in this PEP. Similarly, the older revision included +:c:type:`PyInterpreterGuard` in this PEP. Similarly, the older revision included ``PyInterpreterWeakRef``, which was analogous to :c:type:`PyInterpreterView`. Eventually, the notion of reference counting was completely abandonded from @@ -866,13 +867,13 @@ this proposal for a few reasons: Non-daemon Thread States ------------------------ -In earlier revisions of this PEP, interpreter locks were a property of +In earlier revisions of this PEP, interpreter guards were a property of a thread state rather than a property of an interpreter. This meant that -:c:func:`PyThreadState_Ensure` kept an interpreter lock held, and +:c:func:`PyThreadState_Ensure` kept an interpreter guard held, and it was released upon calling :c:func:`PyThreadState_Release`. A thread state -that held a lock to an interpreter was known as a "non-daemon thread +that held a guard to an interpreter was known as a "non-daemon thread state." At first, this seemed like an improvement, because it shifted management -of a lock's lifetime to the thread instead of the user, which eliminated +of a guard's lifetime to the thread instead of the user, which eliminated some boilerplate. However, this ended up making the proposal significantly more complex and @@ -882,12 +883,12 @@ hurt the proposal's goals: threads as the problem, which hurt the clarity of the PEP. Additionally, the phrase "non-daemon" added extra confusion, because non-daemon Python threads are explicitly joined, whereas a non-daemon C thread is only waited on - until it releases its lock. -- In many cases, an interpreter lock should outlive a singular thread - state. Stealing the interpreter lock in :c:func:`PyThreadState_Ensure` + until it releases its guard. +- In many cases, an interpreter guard should outlive a singular thread + state. Stealing the interpreter guard in :c:func:`PyThreadState_Ensure` was particularly troublesome for these cases. If :c:func:`PyThreadState_Ensure` - didn't steal a lock with non-daemon thread states, it would muddy the - ownership story of the interpreter lock, leading to a more confusing API. + didn't steal a guard with non-daemon thread states, it would muddy the + ownership story of the interpreter guard, leading to a more confusing API. .. _pep-788-activate-deactivate-instead: From 33720311d2cdd0704792aadc97384381c6bd6ad0 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Sun, 12 Oct 2025 11:29:49 -0400 Subject: [PATCH 2/8] Avoid locking terminology. --- peps/pep-0788.rst | 60 +++++++++++++++++++++++------------------------ 1 file changed, 30 insertions(+), 30 deletions(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index 290a163f53c..e815c42414d 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -30,7 +30,7 @@ interpreter. For example: } PyThreadView thread_view = PyThreadState_Ensure(guard); if (thread_view == 0) { - PyInterpreterGuard_Release(guard); + PyInterpreterGuard_Close(guard); return -1; } @@ -38,7 +38,7 @@ interpreter. For example: finalization. */ PyThreadState_Release(thread_view); - PyInterpreterGuard_Release(guard); + PyInterpreterGuard_Close(guard); return 0; } @@ -263,7 +263,7 @@ there's not really another option. This proposal also comes with "views" to an interpreter that can be used to safely poke at an interpreter that may be dead or alive. Using a view, users -can acquire an interpreter guard at any point during its lifecycle, and +can create an interpreter guard at any point during its lifecycle, and will safely fail if the interpreter can no longer support calling Python code. Compatibility Shim for ``PyGILState_Ensure`` @@ -271,7 +271,7 @@ Compatibility Shim for ``PyGILState_Ensure`` This proposal comes with :c:func:`PyUnstable_InterpreterView_FromDefault` as a compatibility hack for some users of :c:func:`PyGILState_Ensure`. It is a -thread-safe way to acquire a guard for the main (or "default") +thread-safe way to create a guard for the main (or "default") interpreter. The main drawback to porting new code to :c:func:`PyThreadState_Ensure` is that @@ -285,7 +285,7 @@ Specification ============= Interpreter Guards ------------------ +------------------ .. c:type:: PyInterpreterGuard @@ -296,13 +296,13 @@ Interpreter Guards This is similar to a "readers-writers" lock; threads may hold an interpreter's guard concurrently, and the interpreter will have to wait - until all threads have released the guard until it can enter finalization. + until all threads have destroyed their guards until it can enter finalization. This type is guaranteed to be pointer-sized. .. c:function:: PyInterpreterGuard PyInterpreterGuard_FromCurrent(void) - Acquire a guard for the current interpreter. + Create a guard for the current interpreter. On success, this function guards the interpreter and returns an opaque reference to the guard, or returns ``0`` with an exception set on failure. @@ -312,7 +312,7 @@ Interpreter Guards .. c:function:: PyInterpreterGuard PyInterpreterGuard_FromView(PyInterpreterView view) - Acquire a guard to an interpreter through a view. + Create a guard to an interpreter through a view. On success, this function returns a guard to the interpreter denoted by *view*. The view is still valid after calling this @@ -340,9 +340,9 @@ Interpreter Guards The caller does not need to hold an :term:`attached thread state`. -.. c:function:: void PyInterpreterGuard_Release(PyInterpreterGuard guard) +.. c:function:: void PyInterpreterGuard_Close(PyInterpreterGuard guard) - Release an interpreter guard, possibly allowing the interpreter to shut + Destroy an interpreter guard, possibly allowing the interpreter to shut down if no guards remain. This function cannot fail, and the caller doesn't need to hold an @@ -517,7 +517,7 @@ With this PEP, you would implement it like this: PyThreadView thread_view = PyThreadState_Ensure(guard); if (thread_view == 0) { - PyInterpreterGuard_Release(guard); + PyInterpreterGuard_Close(guard); fputs("Cannot call Python.\n", stderr); return -1; } @@ -528,7 +528,7 @@ With this PEP, you would implement it like this: // print out the exception ourself. PyErr_Print(); PyThreadState_Release(thread_view); - PyInterpreterGuard_Release(guard); + PyInterpreterGuard_Close(guard); return -1; } int res = PyFile_WriteString(to_write, file); @@ -538,7 +538,7 @@ With this PEP, you would implement it like this: } PyThreadState_Release(thread_view); - PyInterpreterGuard_Release(guard); + PyInterpreterGuard_Close(guard); return res < 0; } @@ -572,7 +572,7 @@ held. Any future finalizer that attempted to acquire the lock would be deadlocke release_some_lock(); Py_END_ALLOW_THREADS; - PyInterpreterGuard_Release(guard); + PyInterpreterGuard_Close(guard); Py_RETURN_NONE; } @@ -623,14 +623,14 @@ This is the same code, rewritten to use the new functions: PyInterpreterGuard guard = (PyInterpreterGuard)arg; PyThreadView thread_view = PyThreadState_Ensure(guard); if (thread_view == 0) { - PyInterpreterGuard_Release(guard); + PyInterpreterGuard_Close(guard); return -1; } if (PyRun_SimpleString("print(42)") < 0) { PyErr_Print(); } PyThreadState_Release(thread_view); - PyInterpreterGuard_Release(guard); + PyInterpreterGuard_Close(guard); return 0; } @@ -646,7 +646,7 @@ This is the same code, rewritten to use the new functions: } if (PyThread_start_joinable_thread(thread_func, (void *)guard, &ident, &handle) < 0) { - PyInterpreterGuard_Release(guard); + PyInterpreterGuard_Close(guard); return NULL; } Py_BEGIN_ALLOW_THREADS @@ -661,7 +661,7 @@ Example: A Daemon Thread With this PEP, daemon threads are very similar to how non-Python threads work in the C API today. After calling :c:func:`PyThreadState_Ensure`, simply -release the interpreter guard to allow the interpreter to shut down (and +close the interpreter guard to allow the interpreter to shut down (and hang the current thread forever). .. code-block:: c @@ -672,12 +672,12 @@ hang the current thread forever). PyInterpreterGuard guard = (PyInterpreterGuard)arg; PyThreadView thread_view = PyThreadState_Ensure(guard); if (thread_view == 0) { - PyInterpreterGuard_Release(guard); + PyInterpreterGuard_Close(guard); return -1; } - /* Release the interpreter guard, allowing it to + /* Close the interpreter guard, allowing it to finalize. This means that print(42) can hang this thread. */ - PyInterpreterGuard_Release(guard); + PyInterpreterGuard_Close(guard); if (PyRun_SimpleString("print(42)") < 0) { PyErr_Print(); } @@ -697,7 +697,7 @@ hang the current thread forever). } if (PyThread_start_joinable_thread(thread_func, (void *)guard, &ident, &handle) < 0) { - PyInterpreterGuard_Release(guard); + PyInterpreterGuard_Close(guard); return NULL; } Py_RETURN_NONE; @@ -725,14 +725,14 @@ Example: An Asynchronous Callback PyThreadView thread_view = PyThreadState_Ensure(guard); if (thread_view == 0) { - PyInterpreterGuard_Release(guard); + PyInterpreterGuard_Close(guard); return -1; } if (PyRun_SimpleString("print(42)") < 0) { PyErr_Print(); } PyThreadState_Release(thread_view); - PyInterpreterGuard_Release(guard); + PyInterpreterGuard_Close(guard); PyInterpreterView_Close(view); PyMem_RawFree(tdata); return 0; @@ -763,8 +763,8 @@ Example: Calling Python Without a Callback Parameter **************************************************** There are a few cases where callback functions don't take a callback parameter -(``void *arg``), so it's difficult to acquire a guard to any specific -interpreter. The solution to this problem is to acquire a guard for the main +(``void *arg``), so it's difficult to create a guard to any specific +interpreter. The solution to this problem is to create a guard for the main interpreter through :c:func:`PyUnstable_InterpreterView_FromDefault`. .. code-block:: c @@ -786,7 +786,7 @@ interpreter through :c:func:`PyUnstable_InterpreterView_FromDefault`. PyThreadView thread_view = PyThreadState_Ensure(guard); if (thread_view == 0) { - PyInterpreterGuard_Release(guard); + PyInterpreterGuard_Close(guard); PyInterpreterView_Close(view); return -1; } @@ -794,7 +794,7 @@ interpreter through :c:func:`PyUnstable_InterpreterView_FromDefault`. PyErr_Print(); } PyThreadState_Release(thread_view); - PyInterpreterGuard_Release(guard); + PyInterpreterGuard_Close(guard); PyInterpreterView_Close(view); return 0; } @@ -870,7 +870,7 @@ Non-daemon Thread States In earlier revisions of this PEP, interpreter guards were a property of a thread state rather than a property of an interpreter. This meant that :c:func:`PyThreadState_Ensure` kept an interpreter guard held, and -it was released upon calling :c:func:`PyThreadState_Release`. A thread state +it was closed upon calling :c:func:`PyThreadState_Release`. A thread state that held a guard to an interpreter was known as a "non-daemon thread state." At first, this seemed like an improvement, because it shifted management of a guard's lifetime to the thread instead of the user, which eliminated @@ -883,7 +883,7 @@ hurt the proposal's goals: threads as the problem, which hurt the clarity of the PEP. Additionally, the phrase "non-daemon" added extra confusion, because non-daemon Python threads are explicitly joined, whereas a non-daemon C thread is only waited on - until it releases its guard. + until it destroys its guard. - In many cases, an interpreter guard should outlive a singular thread state. Stealing the interpreter guard in :c:func:`PyThreadState_Ensure` was particularly troublesome for these cases. If :c:func:`PyThreadState_Ensure` From 7bf55078fe610286f4b843083e989f0bddf8f8ef Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Sun, 12 Oct 2025 11:34:33 -0400 Subject: [PATCH 3/8] Some clarity fixes. --- peps/pep-0788.rst | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index e815c42414d..d2a59702562 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -315,7 +315,7 @@ Interpreter Guards Create a guard to an interpreter through a view. On success, this function returns a guard to the interpreter - denoted by *view*. The view is still valid after calling this + represented by *view*. The view is still valid after calling this function. If the interpreter no longer exists or can no longer support calling Python @@ -326,7 +326,7 @@ Interpreter Guards .. c:function:: PyInterpreterState *PyInterpreterGuard_GetInterpreter(PyInterpreterGuard guard) - Return the :c:type:`PyInterpreterState` pointer denoted by *guard*. + Return the :c:type:`PyInterpreterState` pointer protected by *guard*. This function cannot fail, and the caller doesn't need to hold an :term:`attached thread state`. @@ -376,9 +376,8 @@ Interpreter Views Duplicate a view to an interpreter. - On success, this function returns a non-zero view to the - interpreter denoted by *view*, and returns ``0`` without an exception set - on failure. + On success, this function returns a non-zero copy of *view*, and returns + ``0`` without an exception set on failure. This function cannot fail, and the caller doesn't need to hold an :term:`attached thread state`. @@ -422,7 +421,7 @@ replace :c:func:`PyGILState_Ensure` and :c:func:`PyGILState_Release`. .. c:function:: PyThreadView PyThreadState_Ensure(PyInterpreterGuard guard) Ensure that the thread has an :term:`attached thread state` for the - interpreter denoted by *guard*, and thus can safely invoke that + interpreter protected by *guard*, and thus can safely invoke that interpreter. It is OK to call this function if the thread already has an attached thread state, as long as there is a subsequent call to :c:func:`PyThreadState_Release` that matches this one. From 6cb140a084520e1dd614d91a0d88ff89b8631033 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 13 Oct 2025 12:29:18 -0400 Subject: [PATCH 4/8] Some copyediting. --- peps/pep-0788.rst | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index d2a59702562..f2b818942fa 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -17,26 +17,33 @@ Abstract ======== This PEP introduces a suite of functions in the C API to safely attach to an -interpreter. For example: +interpreter by preventing finalization. For example: .. code-block:: c static int thread_function(PyInterpreterView view) { + // Prevent the interpreter from finalizing PyInterpreterGuard guard = PyInterpreterGuard_FromView(view); if (guard == 0) { return -1; } + + // Analogous to PyGILState_Ensure(), but this is thread-safe. PyThreadView thread_view = PyThreadState_Ensure(guard); if (thread_view == 0) { PyInterpreterGuard_Close(guard); return -1; } - /* Call Python code, without worrying about the thread hanging due to - finalization. */ + // Now we can call Python code, without worrying about the thread + // hanging due to finalization. + if (PyRun_SimpleString("print('My hovercraft is full of eels') < 0) { + PyErr_Print(); + } + // Destroy the thread state and allow the interpreter to finalize. PyThreadState_Release(thread_view); PyInterpreterGuard_Close(guard); return 0; @@ -72,9 +79,10 @@ want to execute Python code alongside some other native code. In addition, a common pattern among users creating non-Python threads is to use :c:func:`PyGILState_Ensure`, which was introduced in :pep:`311`. This has been very unfortunate for subinterpreters, because :c:func:`PyGILState_Ensure` -tends to choose to create a thread state for the main interpreter instead of the current interpreter. This leads -to thread-safety issues when extensions create threads that interact with the -Python interpreter, because assumptions about the GIL are incorrect. +tends to choose to create a thread state for the main interpreter instead of +the current interpreter. This leads to thread-safety issues when extensions +create threads that interact with the Python interpreter, because assumptions +about the GIL are incorrect. Motivation ========== @@ -191,7 +199,7 @@ The Term "GIL" Is Tricky for Free-threading A large issue with the term "GIL" in the C API is that it is semantically misleading. This was noted in `python/cpython#127989 `_, -created by the authors of this PEP: +created by the author of this PEP: The biggest issue is that for free-threading, there is no GIL, so users erroneously call the C API inside ``Py_BEGIN_ALLOW_THREADS`` blocks or @@ -292,7 +300,7 @@ Interpreter Guards An opaque interpreter guard. By holding an interpreter guard, the caller can know that the interpreter - will be in a state where it can safely execute Python code. + will not finalize until the guard has been destroyed. This is similar to a "readers-writers" lock; threads may hold an interpreter's guard concurrently, and the interpreter will have to wait @@ -548,7 +556,7 @@ This example shows acquiring a C lock in a Python method. If this were to be called from a daemon thread, then the interpreter could hang the thread while reattaching the thread state, leaving us with the lock -held. Any future finalizer that attempted to acquire the lock would be deadlocked. +held. Any future finalizer that attempts to acquire the lock would be deadlocked. .. code-block:: c From d7bf283bb6459a85ef8985040e7c21b9ce3f197b Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 13 Oct 2025 12:58:12 -0400 Subject: [PATCH 5/8] Even more copyediting. --- peps/pep-0788.rst | 184 +++++++++++++++++++++++----------------------- 1 file changed, 94 insertions(+), 90 deletions(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index f2b818942fa..2f47d6aa457 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -55,14 +55,14 @@ proposal. Background ========== -In the C API, threads are able to interact with an interpreter by holding an +In the C API, threads can interact with an interpreter by holding an :term:`attached thread state` for the current thread. This can get complicated when it comes to creating and attaching :term:`thread states ` in a safe manner, because any non-Python thread (one not created via the :mod:`threading` module) is considered to be "daemon", meaning that the interpreter won't wait on that thread before shutting down. Instead, the interpreter will hang the -thread when it goes to attach a thread state, making the thread unusable past that -point. +thread when it attempts to attach a thread state, making the thread unusable +thereafter. Attaching a thread state can happen at any point when invoking Python, such as in-between bytecode instructions (to yield the :term:`GIL` to a different thread), @@ -71,18 +71,18 @@ guarding against whether the interpreter is finalizing isn't enough to safely call Python code. (Note that hanging the thread is a relatively new behavior; in older versions, the thread would exit, but the issue is the same.) -Currently, the C API doesn't have any way to ensure that an interpreter -is in a state where it won't hang a thread when trying to attach. -This can be a frustrating issue to deal with in large applications that -want to execute Python code alongside some other native code. +Currently, the C API doesn’t provide any way to ensure that an interpreter is +in a state that won’t cause a thread to hang when trying to attach. This can +be a frustrating issue in large applications that need to execute Python code +alongside other native code. -In addition, a common pattern among users creating non-Python threads is to +In addition, a typical pattern among users creating non-Python threads is to use :c:func:`PyGILState_Ensure`, which was introduced in :pep:`311`. This has been very unfortunate for subinterpreters, because :c:func:`PyGILState_Ensure` -tends to choose to create a thread state for the main interpreter instead of -the current interpreter. This leads to thread-safety issues when extensions -create threads that interact with the Python interpreter, because assumptions -about the GIL are incorrect. +tends to create a thread state for the main interpreter rather than the +urrent interpreter. This leads to thread-safety issues when extensions create +threads that interact with the Python interpreter, because assumptions about +the GIL are incorrect. Motivation ========== @@ -90,7 +90,7 @@ Motivation Non-Python Threads Always Hang During Finalization -------------------------------------------------- -Many large libraries might need to call Python code in highly-asynchronous +Many large libraries might need to call Python code in highly asynchronous situations where the desired interpreter could be finalizing or deleted, but want to continue running code after invoking the interpreter. This desire has been `brought up by users `_. @@ -98,8 +98,8 @@ For example, a callback that wants to call Python code might be invoked when: - A kernel has finished running on a GPU. - A network packet was received. -- A thread has quit, and a native library is executing static finalizers of - thread local storage. +- A thread has quit, and a native library is executing static finalizers for + thread-local storage. Generally, this pattern would look something like this: @@ -119,7 +119,7 @@ Generally, this pattern would look something like this: } This means that any non-Python thread may be terminated at any point, which -is severely limiting for users who want to do more than just execute Python +severely limits users who want to do more than just execute Python code in their stream of calls. ``Py_IsFinalizing`` Is Not Atomic @@ -138,7 +138,7 @@ the thread: Unfortunately, this doesn't work reliably, because of time-of-call to time-of-use issues; the interpreter might not be finalizing during the call to :c:func:`Py_IsFinalizing`, but it might start finalizing immediately -afterwards, which would cause the attachment of a thread state to hang the +afterward, which would cause the attachment of a thread state to hang the thread. Users have `expressed a desire `_ for an @@ -156,10 +156,10 @@ be hung. For example: deadlocks. 2. The main thread begins finalization and tells all thread states to hang upon attachment. -3. The thread acquires the lock it was waiting on, but is then hung by attempting +3. The thread acquires the lock it was waiting on, but then hangs while attempting to reattach its thread state via :c:macro:`Py_END_ALLOW_THREADS`. 4. The main thread can no longer acquire the lock, because the thread holding it - has been hung. + has hung. This affects CPython itself, and there's not much that can be done to fix it with the current API. For example, @@ -196,7 +196,7 @@ works during finalization, because it would break existing code. The Term "GIL" Is Tricky for Free-threading ------------------------------------------- -A large issue with the term "GIL" in the C API is that it is semantically +A significant issue with the term "GIL" in the C API is that it is semantically misleading. This was noted in `python/cpython#127989 `_, created by the author of this PEP: @@ -236,9 +236,9 @@ For example, if the thread had access to object A, which belongs to a subinterpreter, but then called :c:func:`PyGILState_Ensure`, the thread would have an :term:`attached thread state` pointing to the main interpreter, not the subinterpreter. This means that any GIL assumptions about the -object are wrong, because there isn't any synchronization between the two GILs. +object are wrong, because there is no synchronization between the two GILs. -There's not any great way to solve this, other than introducing a new API that +There's no great way to solve this, other than introducing a new API that explicitly takes an interpreter from the caller. Subinterpreters Can Concurrently Deallocate @@ -260,18 +260,18 @@ Rationale Preventing Interpreter Shutdown ------------------------------- -This PEP takes an approach where an interpreter comes with a guarding API -that prevents it from shutting down. Holding an interpreter guard will make it -safe to call the C API without worrying about the thread being hung. +This PEP takes an approach in which an interpreter includes a guarding API +that prevents it from shutting down. Holding an interpreter guard ensures it is +safe to call the C API without worrying about the thread being hung by finalization. -This means that interfacing Python (for example, in a C++ library) will need +This means that interfacing with Python (for example, in a C++ library) will need a guard to the interpreter in order to safely call the object, which is more inconvenient than assuming the main interpreter is the right choice, but there's not really another option. This proposal also comes with "views" to an interpreter that can be used to safely poke at an interpreter that may be dead or alive. Using a view, users -can create an interpreter guard at any point during its lifecycle, and +can create an interpreter guard at any point during its lifecycle, and it will safely fail if the interpreter can no longer support calling Python code. Compatibility Shim for ``PyGILState_Ensure`` @@ -285,8 +285,8 @@ interpreter. The main drawback to porting new code to :c:func:`PyThreadState_Ensure` is that it isn't a drop-in replacement for :c:func:`!PyGILState_Ensure`, as it needs an interpreter guard argument. In some large applications, refactoring to -use a :c:type:`PyInterpreterGuard` everywhere might be tricky; so, this function -acts as a last resort for users who explicitly want to disallow support for +use a :c:type:`PyInterpreterGuard` everywhere might be tricky, so this function +serves as a last resort for users who explicitly want to disallow support for subinterpreters. Specification @@ -299,35 +299,36 @@ Interpreter Guards An opaque interpreter guard. - By holding an interpreter guard, the caller can know that the interpreter - will not finalize until the guard has been destroyed. + By holding an interpreter guard, the caller can ensure that the interpreter + will not finalize until the guard is destroyed. This is similar to a "readers-writers" lock; threads may hold an interpreter's guard concurrently, and the interpreter will have to wait - until all threads have destroyed their guards until it can enter finalization. + until all threads have destroyed their guards before it can enter finalization. This type is guaranteed to be pointer-sized. + .. c:function:: PyInterpreterGuard PyInterpreterGuard_FromCurrent(void) - Create a guard for the current interpreter. + Create a finalization guard for the current interpreter. On success, this function guards the interpreter and returns an opaque - reference to the guard, or returns ``0`` with an exception set on failure. + reference to the guard; on failure, it returns ``0`` with an exception set. The caller must hold an :term:`attached thread state`. .. c:function:: PyInterpreterGuard PyInterpreterGuard_FromView(PyInterpreterView view) - Create a guard to an interpreter through a view. + Create a finalization guard for an interpreter through a view. On success, this function returns a guard to the interpreter represented by *view*. The view is still valid after calling this function. - If the interpreter no longer exists or can no longer support calling Python - code safely, then this function returns ``0`` without an exception set. + If the interpreter no longer exists or cannot safely run Python code, + this function returns 0 without setting an exception. The caller does not need to hold an :term:`attached thread state`. @@ -339,19 +340,21 @@ Interpreter Guards This function cannot fail, and the caller doesn't need to hold an :term:`attached thread state`. + .. c:function:: PyInterpreterGuard PyInterpreterGuard_Copy(PyInterpreterGuard guard) Duplicate an interpreter guard. - On success, this function returns a copy of *guard*, and returns ``0`` - without an exception set on failure. + On success, this function returns a copy of *guard*; on failure, it returns + ``0`` without an exception set. The caller does not need to hold an :term:`attached thread state`. + .. c:function:: void PyInterpreterGuard_Close(PyInterpreterGuard guard) - Destroy an interpreter guard, possibly allowing the interpreter to shut - down if no guards remain. + Destroy an interpreter guard, allowing the interpreter to enter + finalization if no other guards remain. This function cannot fail, and the caller doesn't need to hold an :term:`attached thread state`. @@ -375,8 +378,8 @@ Interpreter Views This function is generally meant to be used in tandem with :c:func:`PyInterpreterGuard_FromView`. - On success, this function returns a view to the current - interpreter, and returns ``0`` with an exception set on failure. + On success, this function returns a view to the current interpreter; on + failure, it returns ``0`` with an exception set. The caller must hold an :term:`attached thread state`. @@ -384,8 +387,8 @@ Interpreter Views Duplicate a view to an interpreter. - On success, this function returns a non-zero copy of *view*, and returns - ``0`` without an exception set on failure. + On success, this function returns a non-zero copy of *view*; on failure, + it returns ``0`` without an exception set. This function cannot fail, and the caller doesn't need to hold an :term:`attached thread state`. @@ -401,11 +404,11 @@ Interpreter Views Create a view for an arbitrary "main" interpreter. - This function only exists for special cases where a specific interpreter + This function only exists for exceptional cases where a specific interpreter can't be saved. On success, this function returns a view to the main - interpreter, and returns ``0`` without an exception set on failure. + interpreter; on failure, it returns ``0`` without an exception set. The caller does not need to hold an :term:`attached thread state`. @@ -420,9 +423,9 @@ replace :c:func:`PyGILState_Ensure` and :c:func:`PyGILState_Release`. An opaque view of a :term:`thread state`. - In this PEP, a thread view comes with no additional properties over a - :c:expr:`PyThreadState *` pointer. APIs for ``PyThreadView`` may be added - in the future. + In this PEP, a thread view provides no additional properties beyond a + :c:expr:`PyThreadState *` pointer. However, APIs for ``PyThreadView`` may + be added in the future. This type is guaranteed to be pointer-sized. @@ -450,10 +453,11 @@ replace :c:func:`PyGILState_Ensure` and :c:func:`PyGILState_Release`. Release a :c:func:`PyThreadState_Ensure` call. - The :term:`attached thread state` prior to the corresponding + The :term:`attached thread state` before the corresponding :c:func:`PyThreadState_Ensure` call is guaranteed to be restored upon - returning. The cached thread state as used by :c:func:`PyThreadState_Ensure` - and :c:func:`PyGILState_Ensure` will also be restored. + returning. The cached thread state as used (the "GIL-state"), by + :c:func:`PyThreadState_Ensure` and :c:func:`PyGILState_Ensure`, will also + be restored. This function cannot fail. @@ -490,7 +494,7 @@ How to Teach This ================= As with all C API functions, all the new APIs in this PEP will be documented -in the C API documentation, ideally under the :ref:`python:gilstate` section. +in the C API documentation, ideally under the ":ref:`python:gilstate`" section. The existing ``PyGILState`` documentation should be updated accordingly to point to the new APIs. @@ -498,7 +502,7 @@ Examples -------- These examples are here to help understand the APIs described in this PEP. -Ideally, they could be reused in the documentation. +They could be reused in the documentation. Example: A Library Interface **************************** @@ -532,7 +536,7 @@ With this PEP, you would implement it like this: const char *to_write = PyUnicode_AsUTF8(text); if (to_write == NULL) { // Since the exception may be destroyed upon calling PyThreadState_Release(), - // print out the exception ourself. + // print out the exception ourselves. PyErr_Print(); PyThreadState_Release(thread_view); PyInterpreterGuard_Close(guard); @@ -552,11 +556,11 @@ With this PEP, you would implement it like this: Example: A Single-threaded Ensure ********************************* -This example shows acquiring a C lock in a Python method. +This example shows how to acquire a C lock in a Python method defined from C. -If this were to be called from a daemon thread, then the interpreter could -hang the thread while reattaching the thread state, leaving us with the lock -held. Any future finalizer that attempts to acquire the lock would be deadlocked. +If this were called from a daemon thread, the interpreter could hang the +thread while reattaching its thread state, leaving us with the lock held. Any +future finalizer that attempts to acquire the lock would be deadlocked. .. code-block:: c @@ -770,7 +774,7 @@ Example: Calling Python Without a Callback Parameter **************************************************** There are a few cases where callback functions don't take a callback parameter -(``void *arg``), so it's difficult to create a guard to any specific +(``void *arg``), so it's difficult to create a guard for any specific interpreter. The solution to this problem is to create a guard for the main interpreter through :c:func:`PyUnstable_InterpreterView_FromDefault`. @@ -818,13 +822,13 @@ Open Issues How Should the APIs Fail? ------------------------- -There is a bit of disagreement on how the ``PyInterpreter[Guard|View]`` APIs +There is some disagreement over how the ``PyInterpreter[Guard|View]`` APIs should indicate a failure to the caller. There are two competing ideas: 1. Return -1 to indicate failure, and 0 to indicate success. On success, functions will assign to a ``PyInterpreter[Guard|View]`` pointer passed as an argument. -2. Directly return a ``PyInterpreter[Guard|View]``, which a value of 0 being +2. Directly return a ``PyInterpreter[Guard|View]``, with a value of 0 being equivalent to ``NULL``, indicating failure. Currently, the PEP spells the latter. @@ -835,9 +839,9 @@ Rejected Ideas Interpreter Reference Counting ------------------------------ -There were two iterations of this proposal that both specified an interpreter to -have a reference count, and the interpreter would wait for that reference count -to hit zero before shutting down. +There were two iterations of this proposal that both specified that an +interpreter maintain a reference count and would wait for that count to reach +zero before shutting down. The first iteration of this idea did this by adding implicit reference counting to ``PyInterpreterState *`` pointers. A function known as ``PyInterpreterState_Hold`` @@ -846,26 +850,26 @@ would increment the reference count (making it a "strong reference"), and standalone ``int64_t``) was used as a form of weak reference, which could be used to look up an interpreter state and atomically increment its reference count. These ideas were ultimately rejected because they seemed to make things -very confusing -- all existing uses of ``PyInterpreterState *`` would be -borrowed, which would make it difficult for developers to understand which -areas of their code required/used a strong reference. +very confusing. All existing uses of ``PyInterpreterState *`` would be +borrowed, making it difficult for developers to understand which +parts of their code require or use a strong reference. In response to that pushback, this PEP specified ``PyInterpreterRef`` APIs that would also mimic reference counting, but in a more explicit manner that -made it easier upon developers. ``PyInterpreterRef`` was analogous to +made it easier for developers. ``PyInterpreterRef`` was analogous to :c:type:`PyInterpreterGuard` in this PEP. Similarly, the older revision included ``PyInterpreterWeakRef``, which was analogous to :c:type:`PyInterpreterView`. -Eventually, the notion of reference counting was completely abandonded from +Eventually, the notion of reference counting was completely abandoned from this proposal for a few reasons: -1. There was contention about overcomplication in the API design; the reference - counting design looked very similar to that of HPy, which had no precedent - in CPython. There was fear that this proposal was being overcomplicated to - look more like HPy. -2. Unlike traditional reference counting APIs, acquiring a strong reference to - an interpreter could arbitrarily fail, and an interpreter would not - immediately deallocate when its reference count reached zero. +1. There was contention over overcomplication in the API design; the + reference-counting design looked very similar to HPy's, which had no + precedent in CPython. There was fear that this proposal was being + overcomplicated to look more like HPy. +2. Unlike traditional reference-counting APIs, acquiring a strong reference to + an interpreter could fail at any time, and an interpreter would not + be deallocated immediately when its reference count reached zero. 3. There was prior discussion about adding "true" reference counting to interpreters (which would deallocate upon reaching zero), which would have been very confusing if there was an existing API in CPython titled @@ -878,19 +882,19 @@ In earlier revisions of this PEP, interpreter guards were a property of a thread state rather than a property of an interpreter. This meant that :c:func:`PyThreadState_Ensure` kept an interpreter guard held, and it was closed upon calling :c:func:`PyThreadState_Release`. A thread state -that held a guard to an interpreter was known as a "non-daemon thread -state." At first, this seemed like an improvement, because it shifted management -of a guard's lifetime to the thread instead of the user, which eliminated -some boilerplate. +that had a guard to an interpreter was known as a "non-daemon thread +state." At first, this seemed like an improvement because it shifted the +management of a guard's lifetime to the thread rather than the user, which +eliminated some boilerplate. However, this ended up making the proposal significantly more complex and hurt the proposal's goals: -- Most importantly, non-daemon thread states put too much emphasis on daemon - threads as the problem, which hurt the clarity of the PEP. Additionally, the - phrase "non-daemon" added extra confusion, because non-daemon Python threads - are explicitly joined, whereas a non-daemon C thread is only waited on - until it destroys its guard. +- Most importantly, non-daemon thread states place too much emphasis on daemon + threads as the problem, which undermines the PEP's clarity. Additionally, + the phrase “non-daemon” added extra confusion, because non-daemon Python + threads are explicitly joined. In contrast, a non-daemon C thread is only + waited on until it destroys its guard. - In many cases, an interpreter guard should outlive a singular thread state. Stealing the interpreter guard in :c:func:`PyThreadState_Ensure` was particularly troublesome for these cases. If :c:func:`PyThreadState_Ensure` @@ -905,7 +909,7 @@ Exposing an ``Activate``/``Deactivate`` API Instead of ``Ensure``/``Clear`` In prior discussions of this API, it was `suggested `_ to provide actual :c:type:`PyThreadState` pointers in the API in an attempt to -make the ownership and lifetime of the thread state clearer: +make the ownership and lifetime of the thread state more straightforward: More importantly though, I think this makes it clearer who owns the thread state - a manually created one is controlled by the code that created it, @@ -930,8 +934,8 @@ benefit of providing an error message. This was rejected because it's `not clear `_ that an error message would be all that useful; all the conceived use-cases for this API wouldn't really care about a message indicating why Python -can't be invoked. As such, the API would only be needlessly harder to use, -which in turn would hurt the transition from :c:func:`PyGILState_Ensure`. +can't be invoked. As such, the API would only be needlessly more complex to +use, which in turn would hurt the transition from :c:func:`PyGILState_Ensure`. In addition, :c:type:`PyStatus` isn't commonly used in the C API. A few functions related to interpreter initialization use it (simply because they From e324111e10084308d23a251fbebda92a6e8868cc Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 13 Oct 2025 12:58:51 -0400 Subject: [PATCH 6/8] Improve some minor phrasing. --- peps/pep-0788.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index 2f47d6aa457..fcb7d5bd4ab 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -891,7 +891,7 @@ However, this ended up making the proposal significantly more complex and hurt the proposal's goals: - Most importantly, non-daemon thread states place too much emphasis on daemon - threads as the problem, which undermines the PEP's clarity. Additionally, + threads as the problem, which made the PEP confusing. Additionally, the phrase “non-daemon” added extra confusion, because non-daemon Python threads are explicitly joined. In contrast, a non-daemon C thread is only waited on until it destroys its guard. From 92add04c164f22086192433a24440b542aab527d Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Mon, 13 Oct 2025 13:00:23 -0400 Subject: [PATCH 7/8] Remove distracting hyperlinks. --- peps/pep-0788.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index fcb7d5bd4ab..e1d5eea43dd 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -205,11 +205,11 @@ created by the author of this PEP: erroneously call the C API inside ``Py_BEGIN_ALLOW_THREADS`` blocks or omit ``PyGILState_Ensure`` in fresh threads. -Again, :c:func:`PyGILState_Ensure` gets an :term:`attached thread state` -for the thread on both with-GIL and free-threaded builds. -An attached thread state is always needed to call the C API, so -:c:func:`PyGILState_Ensure` still needs to be called on free-threaded builds, -but with a name like "ensure GIL", it's not immediately clear that that's true. +Again, :c:func:`PyGILState_Ensure` gets an attached thread state for the +thread on both with-GIL and free-threaded builds. An attached thread state is +always needed to call the C API, so :c:func:`PyGILState_Ensure` still needs +to be called on free-threaded builds, but with a name like "ensure GIL", it's +not immediately clear that that's true. .. _pep-788-subinterpreters-gilstate: @@ -234,7 +234,7 @@ used on objects shared between the threads. For example, if the thread had access to object A, which belongs to a subinterpreter, but then called :c:func:`PyGILState_Ensure`, the thread would -have an :term:`attached thread state` pointing to the main interpreter, +have an attached thread state pointing to the main interpreter, not the subinterpreter. This means that any GIL assumptions about the object are wrong, because there is no synchronization between the two GILs. From 47434e7ac7a582df0cd04866433b320cab5cef29 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Tue, 14 Oct 2025 07:20:59 -0400 Subject: [PATCH 8/8] Apply suggestions from code review Co-authored-by: Victor Stinner --- peps/pep-0788.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0788.rst b/peps/pep-0788.rst index e1d5eea43dd..b45e4f3324a 100644 --- a/peps/pep-0788.rst +++ b/peps/pep-0788.rst @@ -80,7 +80,7 @@ In addition, a typical pattern among users creating non-Python threads is to use :c:func:`PyGILState_Ensure`, which was introduced in :pep:`311`. This has been very unfortunate for subinterpreters, because :c:func:`PyGILState_Ensure` tends to create a thread state for the main interpreter rather than the -urrent interpreter. This leads to thread-safety issues when extensions create +current interpreter. This leads to thread-safety issues when extensions create threads that interact with the Python interpreter, because assumptions about the GIL are incorrect. @@ -328,7 +328,7 @@ Interpreter Guards function. If the interpreter no longer exists or cannot safely run Python code, - this function returns 0 without setting an exception. + this function returns ``0`` without setting an exception. The caller does not need to hold an :term:`attached thread state`.