Make release and debug ABI compatible

[vstinner]

BPO	36465
Nosy	@pitrou, @vstinner
PRs	WIP: bpo-36465: Remove Py_TRACE_REFS special build #12614 bpo-36465: Make release and debug ABI compatible #12615 bpo-36465: Fix test_regrtest on Windows #12945 bpo-39647: Update doc of init_config.rst #18520 bpo-43688: Support the limited C API in debug mode #25131

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-04-24.22:58:10.873>
created_at = <Date 2019-03-28.23:01:42.956>
labels = ['build', '3.8']
title = 'Make release and debug ABI compatible'
updated_at = <Date 2021-04-02.13:45:45.078>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-04-02.13:45:45.078>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-04-24.22:58:10.873>
closer = 'vstinner'
components = ['Build']
creation = <Date 2019-03-28.23:01:42.956>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 36465
keywords = ['patch']
message_count = 9.0
messages = ['339079', '339080', '339081', '340746', '340750', '340812', '340813', '362126', '390064']
nosy_count = 2.0
nosy_names = ['pitrou', 'vstinner']
pr_nums = ['12614', '12615', '12945', '18520', '25131']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue36465'
versions = ['Python 3.8']

[vstinner]

When Python is built in debug mode, PyObject gets 2 new fields: _ob_prev and _ob_next. These fields change the offset of following fields in the PyObject structure and so breaks the ABI.

I propose to modify the debug build (Py_DEBUG) to not imply Py_TRACE_REFS anymore. Antoine Pitrou proposed this idea when the C API was discussed to get a stable ABI.

Another more radical idea is to completely remove Py_TRACE_REFS special build.

[vstinner]

Another more radical idea is to completely remove Py_TRACE_REFS special build.

I wrote PR 12614 to implement this idea. I wrote it to see which code depends on it:

commit 6350949 (HEAD -> remove_trace_refs, origin/remove_trace_refs)
Author: Victor Stinner <vstinner@redhat.com>
Date: Thu Mar 28 23:26:58 2019 +0100

WIP: bpo-36465: Remove Py_TRACE_REFS special build

Remove _ob_prev and _ob_next fields of PyObject when Python is
compiled in debug mode to make debug ABI closer to the release ABI.

Remove:

* sys.getobjects()
* PYTHONDUMPREFS environment variable
* _PyCoreConfig.dump_refs
* PyObject._ob_prev and PyObject._ob_next fields
* _PyObject_HEAD_EXTRA and _PyObject_EXTRA_INIT macros
* _Py_AddToAllObjects()
* _Py_PrintReferenceAddresses()
* _Py_PrintReferences()
* _Py_ForgetReference(op) is replaced with _Py_INC_TPFREES(op)

I never used PYTHONDUMPREFS. I just tried in Python 3.7: Python does crash with this option... so this option doesn't sound popuplar. Otherwise, many users would complain.

$ PYTHONDUMPREFS=1 python3.7-dbg -c pass
...
0x7f7eae14aa90 [1] 'Thread-local dummy'
0x7f7eae19b448 [1] (<class '_thread._localdummy'>, <class 'object'>)
0x7f7eae14aa30 [1] {'__doc__': 'Thread-local dummy'}
0x7f7eae1356d8 [1] (<class 'object'>,)
0x7d0940 [2] <class '_thread._localdummy'>
0x7f7eae120d58 [1] Segmentation fault (core dumped)

I never used sys.getobjects() neither, but I can imagine that someone might want to use for very specific needs. So maybe it's safer to not immediately remove the feature. At least, a deprecation period would be needed.

I suggest to reject PR 12614 and not remove Py_TRACE_REFS.

[vstinner]

PR 12615 changes Py_DEBUG to no longer imply Py_TRACE_REFS. IMHO it's a more reasonable approach.

I'm not sure if it's enough to magically get exactly the same ABI than Python built in release mode.

[vstinner]

Since my goal here is to support a C extension compiled in release mode on a Python compiled in debug mode, Py_TRACE_REFS should be a new separated ABI since it modify Py_INCREF/Py_DECREF. Maybe a new configure option should be added to opt-in for Py_TRACE_REFS and add "t" (T stands for Trace references) to SOABI.

[vstinner]

I started a thread on python-dev to discuss this issue:
"[Python-Dev] Use C extensions compiled in release mode on a Python compiled in debug mode"
https://mail.python.org/pipermail/python-dev/2019-April/157178.html

It's a follow-up of my previous thread:
"[Python-Dev] No longer enable Py_TRACE_REFS by default in debug build"
https://mail.python.org/pipermail/python-dev/2019-April/157015.html

[vstinner]

New changeset f4e4703 by Victor Stinner in branch 'master':
bpo-36465: Make release and debug ABI compatible (GH-12615)
f4e4703

[vstinner]

New changeset 235e7b2 by Victor Stinner in branch 'master':
bpo-36465: Fix test_regrtest on Windows (GH-12945)
235e7b2

[vstinner]

New changeset a784759 by Hai Shi in branch 'master':
bpo-36465: Update doc of init_config.rst (GH-18520)
a784759

[vstinner]

New changeset 3359cab by Victor Stinner in branch 'master':
bpo-43688: Support the limited C API in debug mode (GH-25131)
3359cab