Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable #80891

Closed
vstinner opened this issue Apr 24, 2019 · 40 comments
Labels
3.8 (EOL) end of life interpreter-core (Objects, Python, Grammar, and Parser dirs)

Comments

@vstinner
Copy link
Member

BPO 36710
Nosy @nascheme, @ncoghlan, @vstinner, @phsilva, @markshannon, @ericsnowcurrently, @jdemeyer, @zooba
PRs
  • bpo-36710: Add 'ceval' local variable to ceval.c #12934
  • bpo-36710: Add runtime parameter to _PyThreadState_Init() #12935
  • bpo-36710: PyOS_AfterFork_Child() pass runtime parameter #12936
  • bpo-36710: Add runtime variable to Py_FinalizeEx() #12937
  • bpo-36710: Add runtime variable to Py_InitializeEx() #12939
  • bpo-36710: Add runtime variable in pystate.c #12956
  • bpo-36710: Add runtime parameter in gcmodule.c #12958
  • bpo-36710: Fix compiler warning on PyThreadState_Delete() #12962
  • bpo-36710: add tstate parameter in errors.c #13540
  • bpo-36710: Add tstate parameter in ceval.c #13547
  • bpo-36710: Pass explicitly tstate in sysmodule.c #14060
  • bpo-36710: Add tstate parameter in import.c #14218
  • bpo-36710: Remove PyImport_Cleanup() function #14221
  • bpo-36710: Use tstate in pylifecycle.c #14249
  • bpo-36710: Add tstate parameter in _warnings.c #14250
  • bpo-36710: Pass tstate parameter to GC collect() #17267
  • bpo-36710: Add PyInterpreterState.runtime field #17270
  • bpo-36710: Pass tstate explicitly in abstract.c #21075
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2020-01-13.16:08:27.587>
    created_at = <Date 2019-04-24.12:58:00.763>
    labels = ['interpreter-core', '3.8']
    title = 'Pass _PyRuntimeState as an argument rather than using the _PyRuntime global variable'
    updated_at = <Date 2020-06-23.13:55:15.412>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2020-06-23.13:55:15.412>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2020-01-13.16:08:27.587>
    closer = 'vstinner'
    components = ['Interpreter Core']
    creation = <Date 2019-04-24.12:58:00.763>
    creator = 'vstinner'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 36710
    keywords = ['patch']
    message_count = 40.0
    messages = ['340772', '340775', '340779', '340784', '340789', '340842', '340845', '340860', '340871', '340873', '340880', '340915', '340930', '340941', '340942', '340945', '340946', '340971', '340986', '340996', '341305', '341500', '342138', '343380', '343393', '345537', '345538', '346016', '346030', '346088', '356496', '357013', '357017', '359915', '360210', '364434', '364438', '364440', '364444', '372170']
    nosy_count = 8.0
    nosy_names = ['nascheme', 'ncoghlan', 'vstinner', 'phsilva', 'Mark.Shannon', 'eric.snow', 'jdemeyer', 'steve.dower']
    pr_nums = ['12934', '12935', '12936', '12937', '12939', '12956', '12958', '12962', '13540', '13547', '14060', '14218', '14221', '14249', '14250', '17267', '17270', '21075']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue36710'
    versions = ['Python 3.8']

    @vstinner
    Copy link
    Member Author

    Eric Snow moved global variables into a _PyRuntimeState structure which is made of sub-structures. There is a single instance of _PyRuntimeState: the _PyRuntime global variable.

    I would like to add "_PyRuntimeState *" parameters to functions to avoid relying directly on _PyRuntime global variable. The long term goal is to have "stateless" code: don't rely on global variables, only on input parameters. In practice, we will continue to use thread local storage (TLS) to get the "current context" like the current interpreter and the current Python thread state.

    @vstinner vstinner added 3.8 (EOL) end of life interpreter-core (Objects, Python, Grammar, and Parser dirs) labels Apr 24, 2019
    @vstinner
    Copy link
    Member Author

    New changeset 8bb3230 by Victor Stinner in branch 'master':
    bpo-36710: Add runtime parameter to _PyThreadState_Init() (GH-12935)
    8bb3230

    @vstinner
    Copy link
    Member Author

    New changeset b930a2d by Victor Stinner in branch 'master':
    bpo-36710: PyOS_AfterFork_Child() pass runtime parameter (GH-12936)
    b930a2d

    @vstinner
    Copy link
    Member Author

    New changeset 8e91c24 by Victor Stinner in branch 'master':
    bpo-36710: Add runtime variable to Py_FinalizeEx() (GH-12937)
    8e91c24

    @vstinner
    Copy link
    Member Author

    New changeset 4312522 by Victor Stinner in branch 'master':
    bpo-36710: Add runtime variable to Py_InitializeEx() (GH-12939)
    4312522

    @jdemeyer
    Copy link
    Contributor

    I don't really understand the rationale for these changes. What's wrong with the global variable _PyRuntime?

    What's the long-term goal for _PyRuntime? If you can't get rid of all occurrences of _PyRuntime, how does it help to get rid of *some* occurences?

    @vstinner
    Copy link
    Member Author

    I don't really understand the rationale for these changes. What's wrong with the global variable _PyRuntime?
    What's the long-term goal for _PyRuntime? If you can't get rid of all occurrences of _PyRuntime, how does it help to get rid of *some* occurences?

    The long term goal is to support multiple interpreter instances per process:

    Eric Snow's PEP-554 "Multiple Interpreters in the Stdlib"
    https://www.python.org/dev/peps/pep-0554/

    Right now, there is a single instance of _PyRuntimeState: _PyRuntime. I would be interested to have one _PyRuntimeState per interpreter.

    Maybe _PyRuntimeState should be reworked in the meanwhile. Right now, it's a strange beast: it contains things which are set before Python initialization and things which are set after. It contains C types and Python objects. Maybe some parts should be moved into PyInterpreterState or even PyThreadState. I don't know at this point. It takes time to look at each individual structure field...

    Anyway, more generally, IMHO it's a bad practice to rely on a global variable. Python runtime should be "stateless".

    The current implementation of CPython leaks dozens of *Python* objects at exit. For example, I started to work on this issue while working on https://bugzilla.redhat.com/show_bug.cgi?id=1696322 : Python doesn't clear 2 warnings variables at exit. When I looked into this issue, I also noticed that _PyRuntime.gc.garbage remains *alive* after Py_Finalize().

    That's plain wrong: *all* Python objects must be cleared by Py_Finalize(). Two interpreters must *not* share any Python object.

    Well, the PEP should better explain the rationale than me :-)

    --

    When I wrote PR 12934, I noticed that even getting the current thread state rely on _PyRuntime:

    #define _PyThreadState_GET() \
        ((PyThreadState*)_Py_atomic_load_relaxed(&_PyRuntime.gilstate.tstate_current))

    That's wrong in the case of ceval.c: it should be possible to run _PyEval_EvalFrameDefault() twice at the same time in two threads using two "isolated" interpreters.

    Well, PR 12934 doesn't fix all issues. It's just one small step towards "stateless" runtime and the even more long term of having one GIL per *interpeter*, rather than a single GIL per *process*.

    @vstinner
    Copy link
    Member Author

    I created bpo-36724: Clear _PyRuntime at exit.

    @vstinner
    Copy link
    Member Author

    New changeset 10c8e6a by Victor Stinner in branch 'master':
    bpo-36710: Add runtime variable in pystate.c (GH-12956)
    10c8e6a

    @vstinner
    Copy link
    Member Author

    New changeset 9db0324 by Victor Stinner in branch 'master':
    bpo-36710: Add runtime parameter in gcmodule.c (GH-12958)
    9db0324

    @vstinner
    Copy link
    Member Author

    New changeset 99e69d4 by Victor Stinner in branch 'master':
    bpo-36710: Fix compiler warning on PyThreadState_Delete() (GH-12962)
    99e69d4

    @jdemeyer
    Copy link
    Contributor

    The long term goal is to support multiple interpreter instances per process:
    Eric Snow's PEP-554 "Multiple Interpreters in the Stdlib"
    https://www.python.org/dev/peps/pep-0554/

    Sorry, but I don't see the relation between this issue and PEP-554. It seems to me that the PEP is about making subinterpreters available from pure Python (instead of only at the C level). It doesn't say anything about the *implementation* of subinterpreters, which is what this issue is about.

    So I'm still missing the bigger picture where this issue fits in.

    The current implementation of CPython leaks dozens of *Python* objects at exit.

    That may be an issue to be fixed, but again I don't see the relation with this issue.

    @vstinner
    Copy link
    Member Author

    Jeroen Demeyer:

    Sorry, but I don't see the relation between this issue and PEP-554.

    The long term plan for PEP-554 is to support having one GIL per interpreter for best performances. The GIL lives in _PyRuntime.

    It's not just about the GIL. Currently, the gc module stores its state into _PyRuntime. It's wrong to share a single gc state between two interpreters: each interpreter should have its own "namespace" completely isolated from the other namespaces. For example, _PyRuntime.gc.garbage is a Python list: each interpreter should have its own list.

    My PR 12934 is only a first step to prepare ceval.c for that.

    Said differently, if I understood correctly, each interpreter must have its own _PyRuntime instance.

    Maybe tomorrow, we will keep a single _PyRuntime instance, *but* my work is needed to identify the relationship between the current implementation of Python and _PyRuntime.

    @jdemeyer
    Copy link
    Contributor

    So what's the relation between _PyRuntime and PyInterpreterState? If the latter is a structure per interpreter, what's the point of also making the former per interpreter? It would be better to move data from _PyRuntime to PyInterpreterState.

    @jdemeyer
    Copy link
    Contributor

    It's wrong to share a single gc state between two interpreters

    And what's your solution for that? I'm not asking for a complete ready-to-implement answer, but at least a basic idea. Otherwise it's impossible for me to judge whether your PR 12934 helps with that or not.

    Basically I would like to see something like PEP-579 but for this problem.

    @ericsnowcurrently
    Copy link
    Member

    I don't think this change is the right way to go (yet), but something related might be. First, let's be clear on the status quo for CPython. (This has gotten long, but I want to be clear.)

    Status Quo
    ============

    For simplicity sake, let's say nearly all the code operates relative to the 3 levels of runtime state:

    • global - _PyRuntimeState
    • interpreter - PyInterpreterState
    • thread - PyThreadState

    Furthermore, there are 3 groups of functions in the C-API:

    • context-sensitive - operate relative to the current Python thread
    • runtime-dependent - operate relative to some part of the runtime state, regardless of thread
    • runtime-independent - have nothing to do with CPython's runtime state

    Most of the C-API is context-sensitive. A small portion is runtime-dependent. A handful of functions are runtime-independent (effectively otherwise stateless helper functions that only happen to be part of the C-API).

    Each context-sensitive function relies on there being a "runtime context" it can use relative to the current OS thread. That context consists of the current (i.e. active) PyThreadState, the corresponding PyInterpreterState, and the global _PyRuntimeState. That context is derived from data in TSS (see caveats below). This group includes most of the C-API.

    Each runtime-dependent function operates against one or more runtime state target, regardless of the current thread context (or even if there isn't one). The target state (e.g. PyInterpreterState) is always passed explicitly. Again, this is only a small portion of the C-API.

    Caveats:

    • for context-sensitive functions, we get the global runtime state from the global C variable (_PyRuntime) rather than via the implicit thread context
    • for some of the runtime-dependent functions that target _PyRuntimeState, we rely on the global C variable

    All of this is the pattern we use currently. Using TSS to identify the implicit runtime context has certain benefits and costs:

    benefits:

    • sticking with the status quo means no backward incompatibility for existing C-extension code
    • easier to distinguish the context-sensitive functions from the runtime-dependent ones
    • (debatable) callers don't have to track, nor pass through, an extra argument

    costs:

    • extra complexity in keeping TSS correct
    • makes the C-API bigger (extra macros, etc.)

    Alternative
    =============

    For every context-sensitive function we could add a new first parameter, "context", that provides the runtime context to use. That would be something like this:

    struct {
        PyThreadState *tstate;
        ...
    } PyRuntimeContext;

    The interpreter state and global runtime state would still be accessible via the same indirection we have now.

    Taking this alternative would eliminate the previous costs. Having a consistent "PyRuntimeContext *context" first parameter would maintain the easy distinction from runtime-dependent functions. Asking callers to pass in the context explicitly is probably better regardless. As to backward compatibility, we could maintain a shim to bridge between the old way and the new.

    About the C-global _PyRuntime
    ==============================

    Currently the global runtime state (_PyRuntimeState) is stored in a static global C variable, _PyRuntime. I added it at the time I consolidated many of the existing C globals into a single struct. Having a C global makes it easy to do the wrong thing, so it may be good to do something else.

    That would mean allocating a _PyRuntimeState on the heap early in startup and pass that around where needed. I expect that would not have any meaningful performance penalty. It would probably also simplify some of the code we currently use to manage _PyRuntime correctly.

    As a bonus, this would be important if we decided that multiple-runtimes-per-process were a desirable thing. That's a neat idea, though I don't see a need currently. So on its own it's not really a justification for dropping a static _PyRuntime. :) However, I think the other reasons are enough.

    Conclusions
    ====================

    This issue has a specific objective that I think is premature. We have an existing pattern and we should stick with that until we decide to change to a new pattern. That said, a few things should get corrected and we should investigate alternative patterns for the context-sensitive C-API.

    As to getting rid of the _PyRuntime global variable in favor of putting it on the heap, I'm not opposed. However, doing so should probably be handled in a separate issue.

    Here are my thoughts on actionable items:

    1. look for a better pattern for the context-sensitive C-API
    2. clearly document which of the 3 groups each C-API function belongs to
    3. add a "runtime" field to the PyInterpreterState pointing to the parent _PyRuntimeState
    4. (maybe) add a _PyRuntimeState_GET() macro, a la PyThreadState_GET()
    5. for context-sensitive C-API that uses the global runtime state, get it from the current PyInterpreterState
    6. for runtime-dependent C-API that targets the global runtime state, ensure the _PyRuntimeState is always an explicit parameter
    7. (maybe) drop _PyRuntime and create a _PyRuntimeState on the heap during startup to pass around

    @ericsnowcurrently
    Copy link
    Member

    FWIW, PEP-554 is part of a larger project that I've been working on (slowly) for several years now. [1] The concrete objective is to leverage subinterpreters as the mechanism by which we can achieve multi-core parallelism in Python code. Moving the GIL (and some other parts of _PyRuntimeState, as Victor indicated) down to per-interpreter state is essential to that.

    However, I don't thing making _PyRuntime a per-interpreter thing is right. The runtime holds the set of interpreters, as well as any state state shared by the interpreters.

    Also, to be clear, the status quo is not a problem for me, so make sure I'm not used as the justification for the change (thoughtful as that is of Victor). :)

    [1] https://github.com/ericsnowcurrently/multi-core-python

    @jdemeyer
    Copy link
    Contributor

    Changing *every* C API function to include a state parameter looks very cumbersome. Another alternative would be to store the interpreter state in every Python object (or every class, that would be sufficient). That way, you would only need to pass context to C API functions which do not take a Python object as argument.

    @zooba
    Copy link
    Member

    zooba commented Apr 27, 2019

    Changing every API to take the context parameter would bring us into alignment with the JavaScript VMs.

    I'm working on a project that embeds a few of these, as well as Python, and our thread management is much worse than their context parameter. Though I'm of course very sympathetic to the compatibility argument (but then the shims would just load the context from TSS and pass it around, so they're not too bad).

    Eric's breakdown of context scopes seems spot on, and it means that we only really need the thread state to be passed around. The few places that would be satisfied by runtime state now (GIL, GC) should become interpreter state, which is most easily found from a thread state anyway.

    Runtime state should eventually probably become runtime configuration (those settings we need to create interpreters) and a minimum amount of state to track live interpreters. I see no reason to pass it around anywhere other than interpreter creation, and as a transitional step toward that goal it should be accessible through the active interpreter state.

    @ericsnowcurrently
    Copy link
    Member

    FWIW, I don't mean to side-track this issue. If we want to have any further discussion about broader solutions then let's take this to capi-sig. In fact, I've started a thread there. I'd post the link, but I think it got stuck in moderation. :)

    @nascheme
    Copy link
    Member

    nascheme commented May 2, 2019

    I think there are two questions to answer. First, do we want to support multiple runtimes per process? Second, if we do, what is the best way to do that? Some people would argue that multiple runtimes are not needed or are too hard to do. Maybe they are correct, I'm not sure. We should try to get a consensus on that first question.

    If we do decide to do it, then we need to answer the second question. Passing a "context" argument around seems the best solution. That is how the Java JNI does it. It sounds like that's how Javascript VMs do it too. We don't need to get creative. Look at what other VMs do and copy the best idea.

    If we do decide to do it, evolving the codebase and all extension modules is going to be a massive task. I would imagine that we can have a backwards compatible API layer that uses TSS. The layer that passes context explicitly would still have to maintain the TSS. There could be a build option that turns that backwards compatibility on or off. If off, you would gain some performance advantage because TSS does not have to be kept up-to-date.

    My feeling right now that even though this is a massive job, it is the correct thing to do. CPUs continue to gain cores. Improving CPython's ability to do multi-threading and multi-processing should be a priority for CPython core developers.

    @zooba
    Copy link
    Member

    zooba commented May 6, 2019

    I think Neil is right, though I believe we'll have a clear enough internal boundary that we should only rarely have to maintain TSS for the sake of legacy callers. The build option should just turn off the entire legacy API, which would also make it easier to remove one day.

    @vstinner
    Copy link
    Member Author

    New changeset 09532fe by Victor Stinner in branch 'master':
    bpo-36710: Add 'ceval' local variable to ceval.c (GH-12934)
    09532fe

    @vstinner
    Copy link
    Member Author

    New changeset b4bdecd by Victor Stinner in branch 'master':
    bpo-36710: Add tstate parameter in errors.c (GH-13540)
    b4bdecd

    @vstinner
    Copy link
    Member Author

    New changeset 438a12d by Victor Stinner in branch 'master':
    bpo-36710: Add tstate parameter in ceval.c (GH-13547)
    438a12d

    @vstinner
    Copy link
    Member Author

    New changeset 838f264 by Victor Stinner in branch 'master':
    bpo-36710: Pass explicitly tstate in sysmodule.c (GH-14060)
    838f264

    @vstinner
    Copy link
    Member Author

    I wrote my notes on this issue there:
    https://pythoncapi.readthedocs.io/runtime.html

    @vstinner
    Copy link
    Member Author

    New changeset 0a28f8d by Victor Stinner in branch 'master':
    bpo-36710: Add tstate parameter in import.c (GH-14218)
    0a28f8d

    @vstinner
    Copy link
    Member Author

    New changeset 987a0dc by Victor Stinner in branch 'master':
    bpo-36710: Remove PyImport_Cleanup() function (GH-14221)
    987a0dc

    @vstinner
    Copy link
    Member Author

    New changeset b45d259 by Victor Stinner in branch 'master':
    bpo-36710: Use tstate in pylifecycle.c (GH-14249)
    b45d259

    @vstinner
    Copy link
    Member Author

    I started a thread on python-dev about this issue:
    "Pass the Python thread state to internal C functions"
    https://mail.python.org/archives/list/python-dev@python.org/thread/PQBGECVGVYFTVDLBYURLCXA3T7IPEHHO/

    @vstinner
    Copy link
    Member Author

    New changeset 2e96906 by Victor Stinner in branch 'master':
    bpo-36710: Pass tstate parameter to GC collect() (GH-17267)
    2e96906

    @vstinner
    Copy link
    Member Author

    New changeset 01b1cc1 by Victor Stinner in branch 'master':
    bpo-36710: Add PyInterpreterState.runtime field (GH-17270)
    01b1cc1

    @vstinner
    Copy link
    Member Author

    I continued this work by passing tstate to internal C functions: bpo-38644.

    I also added PyInterpreterState.runtime field, so it's now possible to retrieve the runtime from tstate:

        runtime = tstate->interp->runtime;

    I wrote an article on passing tstate to internal C functions:

    https://vstinner.github.io/cpython-pass-tstate.html

    I consider that this issue is now done. I close the issue.

    @ericsnowcurrently
    Copy link
    Member

    Thanks, Victor!

    @markshannon
    Copy link
    Member

    Instead of passing _PyRuntimeState around everywhere, why not just let it disappear in time.

    Currently _PyRuntimeState manages "global" state, mainly the GIL and some config.
    Once the GIL has been migrated to the sub-interpreters, the config part can be factored out and _PyRuntimeState can just disappear.

    @zooba
    Copy link
    Member

    zooba commented Mar 17, 2020

    Instead of passing _PyRuntimeState around everywhere, why not just let it disappear in time.

    Agreed. It's valuable to pass the thread state, but the runtime state should only be needed to create a new thread state (and arguably not even then).

    @vstinner
    Copy link
    Member Author

    Instead of passing _PyRuntimeState around everywhere, why not just let it disappear in time.

    Passing runtime (_PyRuntimeState) is a temporary move until more and more fields are moved from _PyRuntimeState into PyInterpreterState. I just created bpo-39984 "Move some ceval fields from _PyRuntime.ceval to PyInterpreterState.ceval" yesterday ;-)

    Once we will manage to move the GIL into PyInterpreterState, we would still have to pass PyInterpreterState or PyThreadState to function which require to access the GIL. Passing explicitly runtime is a first step to prepare to migration to PyInterpreterState or PyThreadState. My intent is to show that many functions rely on "global variables": pass these variables instead.

    If you are thinking about getting the current Python thread state using a thread local storage, that's a different topic and I'm not aware of an open issue to track this idea.

    Currently _PyRuntimeState manages "global" state, mainly the GIL and some config. Once the GIL has been migrated to the sub-interpreters, the config part can be factored out and _PyRuntimeState can just disappear.

    I don't think that we will be able to fully remove _PyRuntimeState. It seems like the PEP-554 "Multiple Interpreters in the Stdlib" requires a registry of interpreters and it currently lives in _PyRuntimeState.

    @markshannon
    Copy link
    Member

    Even if _PyRuntime ends up as just a list of interpreters and doesn't disappear completely, it won't be used anything like as much as it is now.

    Many of the functions that it getting passed to will no longer need it, so why bother passing it now?

    @vstinner
    Copy link
    Member Author

    New changeset 61b6492 by Victor Stinner in branch 'master':
    bpo-36710: Pass tstate explicitly in abstract.c (GH-21075)
    61b6492

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 (EOL) end of life interpreter-core (Objects, Python, Grammar, and Parser dirs)
    Projects
    None yet
    Development

    No branches or pull requests

    6 participants