Make Py_BEGIN_CRITICAL_SECTION() and Py_END_CRITICAL_SECTION() public in the non-limited C API

[colesbury]

Feature or enhancement

The critical section API is useful for making C API extensions thread-safe when the GIL is disabled. Compared to plain mutexes, the API makes it easier to avoid deadlocks, especially when interacting with the Python C API, where calls like Py_DECREF() may invoke destructors that themselves acquire locks.

A more detailed description and motivation is in PEP 703: https://peps.python.org/pep-0703/#python-critical-sections

The underlying implementation is hooked into Python's PyThreadState management, so it would not be practical to implement this API outside of CPython.

C-API WG Issue

• Make critical section API public as part of non-limited C API capi-workgroup/decisions#26

Linked PRs

• gh-119344: Make critical section API public #119353
• [3.13] gh-119344: Make critical section API public (GH-119353) #120856

[da-woods]

I've been trying to use these in Cython (by picking them out of the internal headers in the short term). One comment:

In a few places we end up wanting to have an error path and a non-error path out of the critical section (with both paths unlocking the critical section). That doesn't end up easy to do because the macros add brackets, so you can't have 1 BEGIN and 2 ENDs.

For the similar Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS there are equivalent bracket-less macros that make that easier to do (Py_UNBLOCK_THREADS and Py_BLOCK_THREADS).

Would it be possible to have something similar here?

(Edit: I can work around this though, but it might be nicer not to have to do so)

[colesbury]

We can add another pair of macros like Py_[UN]BLOCK_THREADS, but the naming starts to get confusing. In CPython, we've generally factored things so as not to need that, either by moving things to separate functions (most commonly) or using goto or similar (less commonly).