Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C API] Make PyInterpreterState opaque: move it into the internal C API (internal/pycore_pystate.h) #80067

Closed
ericsnowcurrently opened this issue Feb 2, 2019 · 25 comments
Assignees
Labels
3.8 only security fixes expert-C-API

Comments

@ericsnowcurrently
Copy link
Member

BPO 35886
Nosy @arigo, @vsajip, @ncoghlan, @vstinner, @njsmith, @ericsnowcurrently, @matrixise
PRs
  • bpo-35886: Make PyInterpreterState an opaque type in the public API. #11731
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/ericsnowcurrently'
    closed_at = <Date 2020-04-22.17:44:38.443>
    created_at = <Date 2019-02-02.01:07:08.851>
    labels = ['expert-C-API', '3.8']
    title = '[C API] Make PyInterpreterState opaque: move it into the internal C API (internal/pycore_pystate.h)'
    updated_at = <Date 2020-04-22.17:44:38.441>
    user = 'https://github.com/ericsnowcurrently'

    bugs.python.org fields:

    activity = <Date 2020-04-22.17:44:38.441>
    actor = 'vstinner'
    assignee = 'eric.snow'
    closed = True
    closed_date = <Date 2020-04-22.17:44:38.443>
    closer = 'vstinner'
    components = ['C API']
    creation = <Date 2019-02-02.01:07:08.851>
    creator = 'eric.snow'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 35886
    keywords = ['patch', 'patch', 'patch']
    message_count = 25.0
    messages = ['334733', '334734', '335120', '335464', '335657', '336397', '336500', '336501', '336506', '336510', '336590', '336591', '336659', '336660', '336661', '336664', '336671', '336676', '336679', '336682', '353093', '353094', '353370', '367029', '367034']
    nosy_count = 7.0
    nosy_names = ['arigo', 'vinay.sajip', 'ncoghlan', 'vstinner', 'njs', 'eric.snow', 'matrixise']
    pr_nums = ['11731']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue35886'
    versions = ['Python 3.8']

    @ericsnowcurrently
    Copy link
    Member Author

    In November Victor created the Include/cpython directory and moved a decent amount of public (but not limited) API there. This included the PyInterpreterState struct. I'd like to move it into the "internal" headers since it is somewhat coupled to the internal runtime implementation. The alternative is extra complexity. I ran into this while working on another issue.

    Note that the docs indicate that all of the struct's fields are private (and I am not aware of any macros leaking fields into the stable ABI). So moving it should not break anything (yeah, right!), and certainly not the stable ABI (Victor's move would have done that already).

    @ericsnowcurrently ericsnowcurrently added the 3.8 only security fixes label Feb 2, 2019
    @ericsnowcurrently ericsnowcurrently self-assigned this Feb 2, 2019
    @ericsnowcurrently
    Copy link
    Member Author

    FWIW, I was hoping to the same for PyThreadState but it looks like 6 of its fields are exposed in "stable" header files via the following macros:

    # Include/object.h
    Py_TRASHCAN_SAFE_BEGIN
    Py_TRASHCAN_SAFE_END
    Include.ceval.h
    Py_EnterRecursiveCall
    Py_LeaveRecursiveCall
    _Py_MakeRecCheck
    Py_ALLOW_RECURSION
    Py_END_ALLOW_RECURSION

    I'm not sure how that factors into the stable ABI (PyThreadState wasn't ever guarded by Py_LIMITED_API). However, I didn't need to deal with it right now so I'm not going to go there. :)

    @ericsnowcurrently
    Copy link
    Member Author

    @victor, do you see any problems with doing this? It will help simplify other changes I'm working on.

    @vstinner
    Copy link
    Member

    @victor, do you see any problems with doing this? It will help simplify other changes I'm working on.

    I'm quite sure that they are users of the PyInterpreterState structure outside CPython internals and stdlib, but I expect that the number is quite low.

    Since internal headers are now installed (I modified "make install" for that) (but need to define Py_BUILD_CORE), it might be acceptable to force users of this structure to opt-in for internal headers.

    Just make sure that we properly communicate on such backward incompatible changes:

    The bpo-35810 also proposes a subtle backward incompatible change which makes me unhappy, but Stefan Behnel seems less scared than me, so maybe it will be fine.

    Maybe we need to organize a collective effort to better communicate on our backward incompatible C API changes. The capi-sig mailing list may be a good channel for that. I asked to test some popular C extensions to check that they are not broken. If it's the case, we should help them to be prepared for the new C API.

    @ericsnowcurrently
    Copy link
    Member Author

    Thanks, Victor!

    python-dev: https://mail.python.org/pipermail/python-dev/2019-February/156344.html

    Also, my PR already has a What's New entry in the porting section. :)

    @ericsnowcurrently
    Copy link
    Member Author

    New changeset be3b295 by Eric Snow in branch 'master':
    bpo-35886: Make PyInterpreterState an opaque type in the public API. (GH-11731)
    be3b295

    @njsmith
    Copy link
    Contributor

    njsmith commented Feb 25, 2019

    This broke cffi:

    gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/opt/python/3.8-dev/include/python3.8m -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.8/c/_cffi_backend.o
    In file included from c/cffi1_module.c:20:0,
                     from c/_cffi_backend.c:7552:
    c/call_python.c: In function ‘_get_interpstate_dict’:
    c/call_python.c:20:30: error: dereferencing pointer to incomplete type ‘PyInterpreterState {aka struct _is}’
         builtins = tstate->interp->builtins;
                                  ^
    error: command 'gcc' failed with exit status 1
    

    I haven't investigated further but heads up.

    @arigo
    Copy link
    Mannequin

    arigo mannequin commented Feb 25, 2019

    Just so you know, when we look at the changes to CPython, the easiest fix is to add these lines in cffi:

        #if PY_VERSION_HEX >= 0x03080000
        # define Py_BUILD_CORE
        #  include "internal/pycore_pystate.h"
        # undef Py_BUILD_CORE
        #endif

    But if we're looking for a cleaner way to fix this, then cffi's needs could be covered by adding a general-purpose-storage dict to PyInterpreterState (like PyThreadState->dict, but per sub-interpreter instead of per thread), and an API function to get it.

    @vstinner
    Copy link
    Member

    This broke cffi:

    Well, that's not a surprise :-) It's a deliberate backward incompatible change.

    But that's where I suggested to test "popular C extensions" with a C API change before merging it. It's ok if issues are discovered later, but the author of backward incompatible C API changes should collaborate with popular C extension maintainers to help to deal with these issues.

    Can someone report the issue to cffi and post back the url to the bug here?

    @njsmith
    Copy link
    Contributor

    njsmith commented Feb 25, 2019

    The cffi issue is: https://bitbucket.org/cffi/cffi/issues/403/build-fails-on-38-dev-pyinterpreterstate

    Armin already went ahead and committed the change to make cffi include internal/pycore_pystate.h. I guess this might not be the ideal final outcome though :-). Without some fix though it's very difficult to test things on 3.8-dev, because cffi shows up in the dependency stacks of *large* portions of the ecosystem.

    @ericsnowcurrently
    Copy link
    Member Author

    @armin, thanks for fixing things on your end so quickly. :)

    Adding PyInterpreterState.dict and a public getter function (a la PyThreadState) is probably fine. I'm just not sure how that relates to the problem with cffi's use of the "builtins" field. If it would be useful in general then feel free to open a separate issue. I doubt there'd be much opposition if you presented a decent use case, since the cost would be low.

    In the case of PyInterpreterState.builtins specifically, if you need it we could add a simple PyInterpreterState_GetBuiltins(). However, couldn't you already use existing public API, e.g. PyEval_GetBuiltins() or PyImport_GetModule("builtins")? Regardless, we certainly don't want to put anyone in a position that they must define Py_BUILD_CORE just for access to PyInterpreterState fields.

    @ericsnowcurrently
    Copy link
    Member Author

    On Mon, Feb 25, 2019 at 3:02 AM STINNER Victor <report@bugs.python.org> wrote:

    But that's where I suggested to test "popular C extensions" with a C API change before merging it. It's ok if issues are discovered later, but the author of backward incompatible C API changes should collaborate with popular C extension maintainers to help to deal with these issues.

    Yeah, that was my bad. I jumped the gun a bit.

    @ncoghlan
    Copy link
    Contributor

    Next incompatibility: python-hyper/brotlicffi#147 (which indirectly broke httpbin)

    @ncoghlan
    Copy link
    Contributor

    (On closer inspection, that's actually be the same breakage as already mentioned above)

    However, what I'm not clear on is how this would affect projects that had *already* generated their cffi code, and include that in their sdist. Are all those sdists going to fail to build on Python 3.8 now?

    It's OK to require that extension modules be rebuilt for a new release, but breaking compatibility with a *code generator* that means a broad selection of projects are all going to fail to build is a much bigger problem.

    @matrixise
    Copy link
    Member

    @nick which indirectly broke httpbin and this one is used by python-requests and we can't execute the tests of requests.

    @arigo
    Copy link
    Mannequin

    arigo mannequin commented Feb 26, 2019

    @nick the C sources produced by cffi don't change. When they are compiled, they use Py_LIMITED_API so you can continue using a single compiled module version for any recent-enough Python 3.x. The required fix is only inside the cffi module itself.

    @ncoghlan
    Copy link
    Contributor

    Oh, cool (both the fact the issue here is only with building cffi itself, and that cffi creates extension modules that build with PY_LIMITED_API).

    I've filed https://bugs.python.org/issue36124 to follow up on the PyInterpreter_GetDict API idea.

    @arigo
    Copy link
    Mannequin

    arigo mannequin commented Feb 26, 2019

    Cool. Also, no bugfix release of cffi was planned, but I can make one if you think it might help for testing the current pre-release of CPython 3.8.

    @matrixise
    Copy link
    Member

    @arigo

    Yep, I am interested because I would like to execute the tests of the major projects/libraries (django, flasks, pandas, requests, ...) and create issues for the maintainer.

    the sooner we get feedback, the sooner we can fix the bugs.

    @arigo
    Copy link
    Mannequin

    arigo mannequin commented Feb 26, 2019

    Done.

    @vstinner
    Copy link
    Member

    This change broke at least two projects:

    • Blender: access "interp->modules" but also "interp->builtins"
    • FreeCAD: access "interp->modules"

    It would be nice to document replacement solutions at:
    https://docs.python.org/dev/whatsnew/3.8.html#porting-to-python-3-8

    • Replace "interp->modules" with PyImport_GetModuleDict()
    • Replace "interp->builtins" with PyEval_GetBuiltins()

    @vstinner vstinner reopened this Sep 24, 2019
    @vstinner
    Copy link
    Member

    Note: IMHO the "The PyInterpreterState struct has been moved into the “internal” header files (...)" entry should be documented in the "Changes in the C API" section, rather than in the "Changes in the Python API" section of:
    https://docs.python.org/dev/whatsnew/3.8.html#porting-to-python-3-8

    @ericsnowcurrently
    Copy link
    Member Author

    +1 to the suggested "whatsnew" updates. If no one beats me to it I'll work on a PR soon.

    @vstinner
    Copy link
    Member

    FWIW, I was hoping to the same for PyThreadState but it looks like 6 of its fields are exposed in "stable" header files via the following macros: (...)

    I created bpo-39947: "[C API] Make the PyThreadState structure opaque (move it to the internal C API)".

    @vstinner
    Copy link
    Member

    Python 3.8 was released in October 2019 with this change (PyInterpreterState structure is now opaque). Fedora 32 is going to be released soon with Python 3.8 as /usr/bin/python. The change only broke very few projets: cffi (which indirectly broke brotlipy and httpbin), Blender, FreeBSD. Fixes are trivial:

    • Replace "interp->modules" with PyImport_GetModuleDict()
    • Replace "interp->builtins" with PyEval_GetBuiltins()

    I close the issue. Well done Eric!

    @vstinner vstinner changed the title Move PyInterpreterState into Include/internal/pycore_pystate.h [C API] Make PyInterpreterState opaque: move it into the internal C API (internal/pycore_pystate.h) Apr 22, 2020
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 only security fixes expert-C-API
    Projects
    None yet
    Development

    No branches or pull requests

    5 participants