Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable

[vstinner]

BPO	36710
Nosy	@nascheme, @ncoghlan, @vstinner, @phsilva, @markshannon, @ericsnowcurrently, @jdemeyer, @zooba
PRs	bpo-36710: Add 'ceval' local variable to ceval.c #12934 bpo-36710: Add runtime parameter to _PyThreadState_Init() #12935 bpo-36710: PyOS_AfterFork_Child() pass runtime parameter #12936 bpo-36710: Add runtime variable to Py_FinalizeEx() #12937 bpo-36710: Add runtime variable to Py_InitializeEx() #12939 bpo-36710: Add runtime variable in pystate.c #12956 bpo-36710: Add runtime parameter in gcmodule.c #12958 bpo-36710: Fix compiler warning on PyThreadState_Delete() #12962 bpo-36710: add tstate parameter in errors.c #13540 bpo-36710: Add tstate parameter in ceval.c #13547 bpo-36710: Pass explicitly tstate in sysmodule.c #14060 bpo-36710: Add tstate parameter in import.c #14218 bpo-36710: Remove PyImport_Cleanup() function #14221 bpo-36710: Use tstate in pylifecycle.c #14249 bpo-36710: Add tstate parameter in _warnings.c #14250 bpo-36710: Pass tstate parameter to GC collect() #17267 bpo-36710: Add PyInterpreterState.runtime field #17270 bpo-36710: Pass tstate explicitly in abstract.c #21075

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-01-13.16:08:27.587>
created_at = <Date 2019-04-24.12:58:00.763>
labels = ['interpreter-core', '3.8']
title = 'Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable'
updated_at = <Date 2020-06-23.13:55:15.412>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2020-06-23.13:55:15.412>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-01-13.16:08:27.587>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2019-04-24.12:58:00.763>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 36710
keywords = ['patch']
message_count = 40.0
messages = ['340772', '340775', '340779', '340784', '340789', '340842', '340845', '340860', '340871', '340873', '340880', '340915', '340930', '340941', '340942', '340945', '340946', '340971', '340986', '340996', '341305', '341500', '342138', '343380', '343393', '345537', '345538', '346016', '346030', '346088', '356496', '357013', '357017', '359915', '360210', '364434', '364438', '364440', '364444', '372170']
nosy_count = 8.0
nosy_names = ['nascheme', 'ncoghlan', 'vstinner', 'phsilva', 'Mark.Shannon', 'eric.snow', 'jdemeyer', 'steve.dower']
pr_nums = ['12934', '12935', '12936', '12937', '12939', '12956', '12958', '12962', '13540', '13547', '14060', '14218', '14221', '14249', '14250', '17267', '17270', '21075']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue36710'
versions = ['Python 3.8']

[vstinner]

Eric Snow moved global variables into a _PyRuntimeState structure which is made of sub-structures. There is a single instance of _PyRuntimeState: the _PyRuntime global variable.

I would like to add "_PyRuntimeState *" parameters to functions to avoid relying directly on _PyRuntime global variable. The long term goal is to have "stateless" code: don't rely on global variables, only on input parameters. In practice, we will continue to use thread local storage (TLS) to get the "current context" like the current interpreter and the current Python thread state.

[vstinner]

New changeset 8bb3230 by Victor Stinner in branch 'master':
bpo-36710: Add runtime parameter to _PyThreadState_Init() (GH-12935)
8bb3230

[vstinner]

New changeset b930a2d by Victor Stinner in branch 'master':
bpo-36710: PyOS_AfterFork_Child() pass runtime parameter (GH-12936)
b930a2d

[vstinner]

New changeset 8e91c24 by Victor Stinner in branch 'master':
bpo-36710: Add runtime variable to Py_FinalizeEx() (GH-12937)
8e91c24

[vstinner]

New changeset 4312522 by Victor Stinner in branch 'master':
bpo-36710: Add runtime variable to Py_InitializeEx() (GH-12939)
4312522

[jdemeyer]

I don't really understand the rationale for these changes. What's wrong with the global variable _PyRuntime?

What's the long-term goal for _PyRuntime? If you can't get rid of all occurrences of _PyRuntime, how does it help to get rid of *some* occurences?

[vstinner]

I don't really understand the rationale for these changes. What's wrong with the global variable _PyRuntime?
What's the long-term goal for _PyRuntime? If you can't get rid of all occurrences of _PyRuntime, how does it help to get rid of *some* occurences?

The long term goal is to support multiple interpreter instances per process:

Eric Snow's PEP-554 "Multiple Interpreters in the Stdlib"
https://www.python.org/dev/peps/pep-0554/

Right now, there is a single instance of _PyRuntimeState: _PyRuntime. I would be interested to have one _PyRuntimeState per interpreter.

Maybe _PyRuntimeState should be reworked in the meanwhile. Right now, it's a strange beast: it contains things which are set before Python initialization and things which are set after. It contains C types and Python objects. Maybe some parts should be moved into PyInterpreterState or even PyThreadState. I don't know at this point. It takes time to look at each individual structure field...

Anyway, more generally, IMHO it's a bad practice to rely on a global variable. Python runtime should be "stateless".

The current implementation of CPython leaks dozens of *Python* objects at exit. For example, I started to work on this issue while working on https://bugzilla.redhat.com/show_bug.cgi?id=1696322 : Python doesn't clear 2 warnings variables at exit. When I looked into this issue, I also noticed that _PyRuntime.gc.garbage remains *alive* after Py_Finalize().

That's plain wrong: *all* Python objects must be cleared by Py_Finalize(). Two interpreters must *not* share any Python object.

Well, the PEP should better explain the rationale than me :-)

--

When I wrote PR 12934, I noticed that even getting the current thread state rely on _PyRuntime:

#define _PyThreadState_GET() \
    ((PyThreadState*)_Py_atomic_load_relaxed(&_PyRuntime.gilstate.tstate_current))

That's wrong in the case of ceval.c: it should be possible to run _PyEval_EvalFrameDefault() twice at the same time in two threads using two "isolated" interpreters.

Well, PR 12934 doesn't fix all issues. It's just one small step towards "stateless" runtime and the even more long term of having one GIL per *interpeter*, rather than a single GIL per *process*.

[vstinner]

I created bpo-36724: Clear _PyRuntime at exit.

[vstinner]

New changeset 10c8e6a by Victor Stinner in branch 'master':
bpo-36710: Add runtime variable in pystate.c (GH-12956)
10c8e6a

[vstinner]

New changeset 9db0324 by Victor Stinner in branch 'master':
bpo-36710: Add runtime parameter in gcmodule.c (GH-12958)
9db0324

[vstinner]

New changeset 99e69d4 by Victor Stinner in branch 'master':
bpo-36710: Fix compiler warning on PyThreadState_Delete() (GH-12962)
99e69d4

[jdemeyer]

The long term goal is to support multiple interpreter instances per process:
Eric Snow's PEP-554 "Multiple Interpreters in the Stdlib"
https://www.python.org/dev/peps/pep-0554/

Sorry, but I don't see the relation between this issue and PEP-554. It seems to me that the PEP is about making subinterpreters available from pure Python (instead of only at the C level). It doesn't say anything about the *implementation* of subinterpreters, which is what this issue is about.

So I'm still missing the bigger picture where this issue fits in.

The current implementation of CPython leaks dozens of *Python* objects at exit.

That may be an issue to be fixed, but again I don't see the relation with this issue.

[vstinner]

Jeroen Demeyer:

Sorry, but I don't see the relation between this issue and PEP-554.

The long term plan for PEP-554 is to support having one GIL per interpreter for best performances. The GIL lives in _PyRuntime.

It's not just about the GIL. Currently, the gc module stores its state into _PyRuntime. It's wrong to share a single gc state between two interpreters: each interpreter should have its own "namespace" completely isolated from the other namespaces. For example, _PyRuntime.gc.garbage is a Python list: each interpreter should have its own list.

My PR 12934 is only a first step to prepare ceval.c for that.

Said differently, if I understood correctly, each interpreter must have its own _PyRuntime instance.

Maybe tomorrow, we will keep a single _PyRuntime instance, *but* my work is needed to identify the relationship between the current implementation of Python and _PyRuntime.

[jdemeyer]

So what's the relation between _PyRuntime and PyInterpreterState? If the latter is a structure per interpreter, what's the point of also making the former per interpreter? It would be better to move data from _PyRuntime to PyInterpreterState.

[jdemeyer]

It's wrong to share a single gc state between two interpreters

And what's your solution for that? I'm not asking for a complete ready-to-implement answer, but at least a basic idea. Otherwise it's impossible for me to judge whether your PR 12934 helps with that or not.

Basically I would like to see something like PEP-579 but for this problem.

[ericsnowcurrently]

I don't think this change is the right way to go (yet), but something related might be. First, let's be clear on the status quo for CPython. (This has gotten long, but I want to be clear.)

Status Quo
============

For simplicity sake, let's say nearly all the code operates relative to the 3 levels of runtime state:

• global - _PyRuntimeState
• interpreter - PyInterpreterState
• thread - PyThreadState

Furthermore, there are 3 groups of functions in the C-API:

• context-sensitive - operate relative to the current Python thread
• runtime-dependent - operate relative to some part of the runtime state, regardless of thread
• runtime-independent - have nothing to do with CPython's runtime state

Most of the C-API is context-sensitive. A small portion is runtime-dependent. A handful of functions are runtime-independent (effectively otherwise stateless helper functions that only happen to be part of the C-API).

Each context-sensitive function relies on there being a "runtime context" it can use relative to the current OS thread. That context consists of the current (i.e. active) PyThreadState, the corresponding PyInterpreterState, and the global _PyRuntimeState. That context is derived from data in TSS (see caveats below). This group includes most of the C-API.

Each runtime-dependent function operates against one or more runtime state target, regardless of the current thread context (or even if there isn't one). The target state (e.g. PyInterpreterState) is always passed explicitly. Again, this is only a small portion of the C-API.

Caveats:

• for context-sensitive functions, we get the global runtime state from the global C variable (_PyRuntime) rather than via the implicit thread context
• for some of the runtime-dependent functions that target _PyRuntimeState, we rely on the global C variable

All of this is the pattern we use currently. Using TSS to identify the implicit runtime context has certain benefits and costs:

benefits:

• sticking with the status quo means no backward incompatibility for existing C-extension code
• easier to distinguish the context-sensitive functions from the runtime-dependent ones
• (debatable) callers don't have to track, nor pass through, an extra argument

costs:

• extra complexity in keeping TSS correct
• makes the C-API bigger (extra macros, etc.)

Alternative
=============

For every context-sensitive function we could add a new first parameter, "context", that provides the runtime context to use. That would be something like this:

struct {
    PyThreadState *tstate;
    ...
} PyRuntimeContext;

The interpreter state and global runtime state would still be accessible via the same indirection we have now.

Taking this alternative would eliminate the previous costs. Having a consistent "PyRuntimeContext *context" first parameter would maintain the easy distinction from runtime-dependent functions. Asking callers to pass in the context explicitly is probably better regardless. As to backward compatibility, we could maintain a shim to bridge between the old way and the new.

About the C-global _PyRuntime
==============================

Currently the global runtime state (_PyRuntimeState) is stored in a static global C variable, _PyRuntime. I added it at the time I consolidated many of the existing C globals into a single struct. Having a C global makes it easy to do the wrong thing, so it may be good to do something else.

That would mean allocating a _PyRuntimeState on the heap early in startup and pass that around where needed. I expect that would not have any meaningful performance penalty. It would probably also simplify some of the code we currently use to manage _PyRuntime correctly.

As a bonus, this would be important if we decided that multiple-runtimes-per-process were a desirable thing. That's a neat idea, though I don't see a need currently. So on its own it's not really a justification for dropping a static _PyRuntime. :) However, I think the other reasons are enough.

Conclusions
====================

This issue has a specific objective that I think is premature. We have an existing pattern and we should stick with that until we decide to change to a new pattern. That said, a few things should get corrected and we should investigate alternative patterns for the context-sensitive C-API.

As to getting rid of the _PyRuntime global variable in favor of putting it on the heap, I'm not opposed. However, doing so should probably be handled in a separate issue.

Here are my thoughts on actionable items:

1. look for a better pattern for the context-sensitive C-API
2. clearly document which of the 3 groups each C-API function belongs to
3. add a "runtime" field to the PyInterpreterState pointing to the parent _PyRuntimeState
4. (maybe) add a _PyRuntimeState_GET() macro, a la PyThreadState_GET()
5. for context-sensitive C-API that uses the global runtime state, get it from the current PyInterpreterState
6. for runtime-dependent C-API that targets the global runtime state, ensure the _PyRuntimeState is always an explicit parameter
7. (maybe) drop _PyRuntime and create a _PyRuntimeState on the heap during startup to pass around

[ericsnowcurrently]

FWIW, PEP-554 is part of a larger project that I've been working on (slowly) for several years now. [1] The concrete objective is to leverage subinterpreters as the mechanism by which we can achieve multi-core parallelism in Python code. Moving the GIL (and some other parts of _PyRuntimeState, as Victor indicated) down to per-interpreter state is essential to that.

However, I don't thing making _PyRuntime a per-interpreter thing is right. The runtime holds the set of interpreters, as well as any state state shared by the interpreters.

Also, to be clear, the status quo is not a problem for me, so make sure I'm not used as the justification for the change (thoughtful as that is of Victor). :)

[1] https://github.com/ericsnowcurrently/multi-core-python

[jdemeyer]

Changing *every* C API function to include a state parameter looks very cumbersome. Another alternative would be to store the interpreter state in every Python object (or every class, that would be sufficient). That way, you would only need to pass context to C API functions which do not take a Python object as argument.

[zooba]

Changing every API to take the context parameter would bring us into alignment with the JavaScript VMs.

I'm working on a project that embeds a few of these, as well as Python, and our thread management is much worse than their context parameter. Though I'm of course very sympathetic to the compatibility argument (but then the shims would just load the context from TSS and pass it around, so they're not too bad).

Eric's breakdown of context scopes seems spot on, and it means that we only really need the thread state to be passed around. The few places that would be satisfied by runtime state now (GIL, GC) should become interpreter state, which is most easily found from a thread state anyway.

Runtime state should eventually probably become runtime configuration (those settings we need to create interpreters) and a minimum amount of state to track live interpreters. I see no reason to pass it around anywhere other than interpreter creation, and as a transitional step toward that goal it should be accessible through the active interpreter state.

[ericsnowcurrently]

FWIW, I don't mean to side-track this issue. If we want to have any further discussion about broader solutions then let's take this to capi-sig. In fact, I've started a thread there. I'd post the link, but I think it got stuck in moderation. :)

[nascheme]

I think there are two questions to answer. First, do we want to support multiple runtimes per process? Second, if we do, what is the best way to do that? Some people would argue that multiple runtimes are not needed or are too hard to do. Maybe they are correct, I'm not sure. We should try to get a consensus on that first question.

If we do decide to do it, then we need to answer the second question. Passing a "context" argument around seems the best solution. That is how the Java JNI does it. It sounds like that's how Javascript VMs do it too. We don't need to get creative. Look at what other VMs do and copy the best idea.

If we do decide to do it, evolving the codebase and all extension modules is going to be a massive task. I would imagine that we can have a backwards compatible API layer that uses TSS. The layer that passes context explicitly would still have to maintain the TSS. There could be a build option that turns that backwards compatibility on or off. If off, you would gain some performance advantage because TSS does not have to be kept up-to-date.

My feeling right now that even though this is a massive job, it is the correct thing to do. CPUs continue to gain cores. Improving CPython's ability to do multi-threading and multi-processing should be a priority for CPython core developers.

[zooba]

I think Neil is right, though I believe we'll have a clear enough internal boundary that we should only rarely have to maintain TSS for the sake of legacy callers. The build option should just turn off the entire legacy API, which would also make it easier to remove one day.

[vstinner]

New changeset 09532fe by Victor Stinner in branch 'master':
bpo-36710: Add 'ceval' local variable to ceval.c (GH-12934)
09532fe

[vstinner]

New changeset b4bdecd by Victor Stinner in branch 'master':
bpo-36710: Add tstate parameter in errors.c (GH-13540)
b4bdecd

[vstinner]

New changeset 438a12d by Victor Stinner in branch 'master':
bpo-36710: Add tstate parameter in ceval.c (GH-13547)
438a12d

[vstinner]

New changeset 838f264 by Victor Stinner in branch 'master':
bpo-36710: Pass explicitly tstate in sysmodule.c (GH-14060)
838f264

[vstinner]

I wrote my notes on this issue there:
https://pythoncapi.readthedocs.io/runtime.html

[vstinner]

New changeset 0a28f8d by Victor Stinner in branch 'master':
bpo-36710: Add tstate parameter in import.c (GH-14218)
0a28f8d

[vstinner]

New changeset 987a0dc by Victor Stinner in branch 'master':
bpo-36710: Remove PyImport_Cleanup() function (GH-14221)
987a0dc

[vstinner]

New changeset b45d259 by Victor Stinner in branch 'master':
bpo-36710: Use tstate in pylifecycle.c (GH-14249)
b45d259

[vstinner]

I started a thread on python-dev about this issue:
"Pass the Python thread state to internal C functions"
https://mail.python.org/archives/list/python-dev@python.org/thread/PQBGECVGVYFTVDLBYURLCXA3T7IPEHHO/

[vstinner]

New changeset 2e96906 by Victor Stinner in branch 'master':
bpo-36710: Pass tstate parameter to GC collect() (GH-17267)
2e96906

[vstinner]

New changeset 01b1cc1 by Victor Stinner in branch 'master':
bpo-36710: Add PyInterpreterState.runtime field (GH-17270)
01b1cc1

[vstinner]

I continued this work by passing tstate to internal C functions: bpo-38644.

I also added PyInterpreterState.runtime field, so it's now possible to retrieve the runtime from tstate:

    runtime = tstate->interp->runtime;

I wrote an article on passing tstate to internal C functions:

https://vstinner.github.io/cpython-pass-tstate.html

I consider that this issue is now done. I close the issue.

[ericsnowcurrently]

Thanks, Victor!

[markshannon]

Instead of passing _PyRuntimeState around everywhere, why not just let it disappear in time.

Currently _PyRuntimeState manages "global" state, mainly the GIL and some config.
Once the GIL has been migrated to the sub-interpreters, the config part can be factored out and _PyRuntimeState can just disappear.

[zooba]

Instead of passing _PyRuntimeState around everywhere, why not just let it disappear in time.

Agreed. It's valuable to pass the thread state, but the runtime state should only be needed to create a new thread state (and arguably not even then).

[vstinner]

Instead of passing _PyRuntimeState around everywhere, why not just let it disappear in time.

Passing runtime (_PyRuntimeState) is a temporary move until more and more fields are moved from _PyRuntimeState into PyInterpreterState. I just created bpo-39984 "Move some ceval fields from _PyRuntime.ceval to PyInterpreterState.ceval" yesterday ;-)

Once we will manage to move the GIL into PyInterpreterState, we would still have to pass PyInterpreterState or PyThreadState to function which require to access the GIL. Passing explicitly runtime is a first step to prepare to migration to PyInterpreterState or PyThreadState. My intent is to show that many functions rely on "global variables": pass these variables instead.

If you are thinking about getting the current Python thread state using a thread local storage, that's a different topic and I'm not aware of an open issue to track this idea.

Currently _PyRuntimeState manages "global" state, mainly the GIL and some config. Once the GIL has been migrated to the sub-interpreters, the config part can be factored out and _PyRuntimeState can just disappear.

I don't think that we will be able to fully remove _PyRuntimeState. It seems like the PEP-554 "Multiple Interpreters in the Stdlib" requires a registry of interpreters and it currently lives in _PyRuntimeState.

[markshannon]

Even if _PyRuntime ends up as just a list of interpreters and doesn't disappear completely, it won't be used anything like as much as it is now.

Many of the functions that it getting passed to will no longer need it, so why bother passing it now?

[vstinner]

New changeset 61b6492 by Victor Stinner in branch 'master':
bpo-36710: Pass tstate explicitly in abstract.c (GH-21075)
61b6492