Add internal-only _PyThreadStateImpl "wrapper" for PyThreadState

[colesbury]

Feature or enhancement

The PyThreadState struct definitions (i.e., struct _ts) is visible in Python's public C API. Our documentation lists only the "interp" field as public. In practice, all the fields are visible, and some extensions (like Cython) make use of some of those fields.

We will want to add some private fields on a per-PyThreadState basis for the --disable-gil builds, but we also do not want to expose them publicly.

The "solution" used in nogil-3.12 was to add a new struct _PyThreadStateImpl that contains the PyThreadState as well as the private fields. Every PyThreadState is actually a _PyThreadStateImpl -- you can cast pointers from one to the other safely. Only the code that allocates & frees PyThreadStates (in pystate.c) and code that accesses those private fields needs to know about _PyThreadStateImpl. Everything else can keep using PyThreadState* pointers.

Here is an example definition:

typedef struct {
    PyThreadState base;
    // private fields go here
#ifdef Py_GIL_DISABLED
    struct _Py_float_state float_state; // e.g., float free-list
#endif
    ...
} _PyThreadStateImpl;

Some of the private fields include free lists, state for biased reference counting, and mimalloc heaps. Free lists are currently stored per-interpreter (in PyInterpreterState), but we will want them per-PyThreadState in --disable-gil builds.

Alternative

We could instead add an opaque pointer from PyThreadState to private data. For example, something like:

typedef struct _PyThreadStatePrivate _PyThreadStatePrivate;  // opaque in the public API

struct _ts {
    ...
    _PyThreadStatePrivate *private;
};

This requires an extra pointer-dereference to access private data, and many of the use cases for these private fields are performance sensitive (e.g., free lists, memory allocation).

cc @ericsnowcurrently, @vstinner

Linked PRs

• gh-112538: Add internal-only _PyThreadStateImpl "wrapper" for PyThreadState #112560