Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make release and debug ABI compatible #80646

Closed
vstinner opened this issue Mar 28, 2019 · 9 comments
Closed

Make release and debug ABI compatible #80646

vstinner opened this issue Mar 28, 2019 · 9 comments
Labels
3.8 build

Comments

@vstinner
Copy link
Member

@vstinner vstinner commented Mar 28, 2019

BPO 36465
Nosy @pitrou, @vstinner
PRs
  • #12614
  • #12615
  • #12945
  • #18520
  • #25131
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2019-04-24.22:58:10.873>
    created_at = <Date 2019-03-28.23:01:42.956>
    labels = ['build', '3.8']
    title = 'Make release and debug ABI compatible'
    updated_at = <Date 2021-04-02.13:45:45.078>
    user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2021-04-02.13:45:45.078>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2019-04-24.22:58:10.873>
    closer = 'vstinner'
    components = ['Build']
    creation = <Date 2019-03-28.23:01:42.956>
    creator = 'vstinner'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 36465
    keywords = ['patch']
    message_count = 9.0
    messages = ['339079', '339080', '339081', '340746', '340750', '340812', '340813', '362126', '390064']
    nosy_count = 2.0
    nosy_names = ['pitrou', 'vstinner']
    pr_nums = ['12614', '12615', '12945', '18520', '25131']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = None
    url = 'https://bugs.python.org/issue36465'
    versions = ['Python 3.8']

    @vstinner
    Copy link
    Member Author

    @vstinner vstinner commented Mar 28, 2019

    When Python is built in debug mode, PyObject gets 2 new fields: _ob_prev and _ob_next. These fields change the offset of following fields in the PyObject structure and so breaks the ABI.

    I propose to modify the debug build (Py_DEBUG) to not imply Py_TRACE_REFS anymore. Antoine Pitrou proposed this idea when the C API was discussed to get a stable ABI.

    Another more radical idea is to completely remove Py_TRACE_REFS special build.

    @vstinner vstinner added 3.8 build labels Mar 28, 2019
    @vstinner
    Copy link
    Member Author

    @vstinner vstinner commented Mar 28, 2019

    Another more radical idea is to completely remove Py_TRACE_REFS special build.

    I wrote PR 12614 to implement this idea. I wrote it to see which code depends on it:

    commit 6350949 (HEAD -> remove_trace_refs, origin/remove_trace_refs)
    Author: Victor Stinner <vstinner@redhat.com>
    Date: Thu Mar 28 23:26:58 2019 +0100

    WIP: bpo-36465: Remove Py_TRACE_REFS special build
    
    Remove _ob_prev and _ob_next fields of PyObject when Python is
    compiled in debug mode to make debug ABI closer to the release ABI.
    
    Remove:
    
    * sys.getobjects()
    * PYTHONDUMPREFS environment variable
    * _PyCoreConfig.dump_refs
    * PyObject._ob_prev and PyObject._ob_next fields
    * _PyObject_HEAD_EXTRA and _PyObject_EXTRA_INIT macros
    * _Py_AddToAllObjects()
    * _Py_PrintReferenceAddresses()
    * _Py_PrintReferences()
    * _Py_ForgetReference(op) is replaced with _Py_INC_TPFREES(op)
    

    I never used PYTHONDUMPREFS. I just tried in Python 3.7: Python does crash with this option... so this option doesn't sound popuplar. Otherwise, many users would complain.

    $ PYTHONDUMPREFS=1 python3.7-dbg -c pass
    ...
    0x7f7eae14aa90 [1] 'Thread-local dummy'
    0x7f7eae19b448 [1] (<class '_thread._localdummy'>, <class 'object'>)
    0x7f7eae14aa30 [1] {'__doc__': 'Thread-local dummy'}
    0x7f7eae1356d8 [1] (<class 'object'>,)
    0x7d0940 [2] <class '_thread._localdummy'>
    0x7f7eae120d58 [1] Segmentation fault (core dumped)

    I never used sys.getobjects() neither, but I can imagine that someone might want to use for very specific needs. So maybe it's safer to not immediately remove the feature. At least, a deprecation period would be needed.

    I suggest to reject PR 12614 and not remove Py_TRACE_REFS.

    @vstinner
    Copy link
    Member Author

    @vstinner vstinner commented Mar 28, 2019

    PR 12615 changes Py_DEBUG to no longer imply Py_TRACE_REFS. IMHO it's a more reasonable approach.

    I'm not sure if it's enough to magically get exactly the same ABI than Python built in release mode.

    @vstinner
    Copy link
    Member Author

    @vstinner vstinner commented Apr 23, 2019

    Since my goal here is to support a C extension compiled in release mode on a Python compiled in debug mode, Py_TRACE_REFS should be a new separated ABI since it modify Py_INCREF/Py_DECREF. Maybe a new configure option should be added to opt-in for Py_TRACE_REFS and add "t" (T stands for Trace references) to SOABI.

    @vstinner
    Copy link
    Member Author

    @vstinner vstinner commented Apr 24, 2019

    I started a thread on python-dev to discuss this issue:
    "[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode"
    https://mail.python.org/pipermail/python-dev/2019-April/157178.html

    It's a follow-up of my previous thread:
    "[Python-Dev] No longer enable Py_TRACE_REFS by default in debug build"
    https://mail.python.org/pipermail/python-dev/2019-April/157015.html

    @vstinner vstinner changed the title No longer enable Py_TRACE_REFS by default in debug build Make release and debug ABI compatible Apr 24, 2019
    @vstinner
    Copy link
    Member Author

    @vstinner vstinner commented Apr 24, 2019

    New changeset f4e4703 by Victor Stinner in branch 'master':
    bpo-36465: Make release and debug ABI compatible (GH-12615)
    f4e4703

    @vstinner
    Copy link
    Member Author

    @vstinner vstinner commented Apr 24, 2019

    New changeset 235e7b2 by Victor Stinner in branch 'master':
    bpo-36465: Fix test_regrtest on Windows (GH-12945)
    235e7b2

    @vstinner
    Copy link
    Member Author

    @vstinner vstinner commented Feb 17, 2020

    New changeset a784759 by Hai Shi in branch 'master':
    bpo-36465: Update doc of init_config.rst (GH-18520)
    a784759

    @vstinner
    Copy link
    Member Author

    @vstinner vstinner commented Apr 2, 2021

    New changeset 3359cab by Victor Stinner in branch 'master':
    bpo-43688: Support the limited C API in debug mode (GH-25131)
    3359cab

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 build
    Projects
    None yet
    Development

    No branches or pull requests

    1 participant