Convert PyTuple_GET_ITEM() macro to a function call with additional checks in debug mode

[vstinner]

BPO	35199
Nosy	@rhettinger, @vstinner
PRs	bpo-35199: Add an internal _PyTuple_ITEMS() macro #10434 [WIP] bpo-35199: PyTuple_GET_ITEM() becomes a function in debug mode #10435

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-11-10.00:36:49.655>
created_at = <Date 2018-11-09.15:32:02.819>
labels = ['interpreter-core', '3.8']
title = 'Convert PyTuple_GET_ITEM() macro to a function call with additional checks in debug mode'
updated_at = <Date 2018-11-10.01:43:18.485>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2018-11-10.01:43:18.485>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2018-11-10.00:36:49.655>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2018-11-09.15:32:02.819>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 35199
keywords = ['patch']
message_count = 9.0
messages = ['329523', '329525', '329529', '329530', '329536', '329570', '329572', '329577', '329585']
nosy_count = 2.0
nosy_names = ['rhettinger', 'vstinner']
pr_nums = ['10434', '10435']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue35199'
versions = ['Python 3.8']

[vstinner]

Currently, even when Python is compiled in debug mode, PyTuple_GET_ITEM() doesn't check that the first argument is a tuple objet and that the second argument is valid index. It can lead to a crash and Python doesn't help debugging.

I propose to convert the macro to a function call and use regular assertions to abort Python if the C API is misused.

I propose to use a function call rather than abusing the preprocessor syntax like (assert(...),expr) syntax used in unicodeobject.h, because I the preprocessor causes complex bugs (difficult to understand and to fix/work around) and because later I would like to experiment to be able to compile C extensions to always use function calls, but get a different implementation depending on the "Python runtime".

I elaborated this idea on this website:

• https://pythoncapi.readthedocs.io/runtimes.html#debug-build
• https://pythoncapi.readthedocs.io/

I am working on an implementation.

[vstinner]

Currently, PyTuple_GET_ITEM() can be used with &PyTuple_GET_ITEM(). If we convert PyTuple_GET_ITEM() to a function returning PyObject*, it's no longer possible. My PR 10434 prepares the code for that.

[vstinner]

New changeset d17a693 by Victor Stinner in branch 'master':
bpo-35199: Add an internal _PyTuple_ITEMS() macro (GH-10434)
d17a693

[vstinner]

Copy of my comment:
#10435 (comment)

This change breaks the backward compatibility when a C extension is compiled with Py_DEBUG. So I'm not sure that it should be merged into Python 3.8. Maybe we should add a new opt-in experimental option to enable it, instead of Py_DEBUG?

[rhettinger]

I don't think this should be merged. A lot of code of uses PyTuple_GET_ITEM(). Also, the "problem" your solving doesn't seem to exist in practice. Further, I worry that a compiler may choose not to inline on occasion, leading to performance regressions in code that has already been finely tuned.

[vstinner]

I don't think this should be merged. A lot of code of uses PyTuple_GET_ITEM(). Also, the "problem" your solving doesn't seem to exist in practice.

Sorry my comment on the PR, a friend contacted me because (...) he misused PyTuple_GET_ITEM() :-)

Further, I worry that a compiler may choose not to inline on occasion, leading to performance regressions in code that has already been finely tuned.

Oh, maybe I didn't explain properly my change. I don't want to change the "default" API nor the "default" implementation. My change is enabled enabled if you compile Python in debug mode, if Py_DEBUG is defined.

In fact, it's an early attempt to see how we can introduce a new C API in Python. It seems like abusing Py_DEBUG was a bad idea, so I close my PR and I will close this issue as well. I will start a discussion on python-dev later, when I will have a plan.

See also the discussion on the PR:
#10435 (comment)

[rhettinger]

Thanks. I don't really know enough about compilers and required-vs-optional C99 semantics to know whether these changes are safe. These are very old APIs and are used throughout the entire Python ecosystem. It would pay to be somewhat cautious.

The whole C-API project is predicated on giving a 2x speed-up in exchange for destabilizing changes. These cross-module inline function-for-macro swaps aren't aligned with that goal. At the very best, the swaps will be performance neutral. At worst, they will invisibly degrade a huge swath of code, someone of which is finely-tuned (people generally only use PyTuple_GET_ITEM() in places where they want better performance than the existing function based alternative)

[vstinner]

Oops, I forgot to close the issue.

This change was an early attempt (tagged as WIP/DO-NOT-MERGE) to expriment changing the implementation without touching the API. But I was wrong, it does change the API (&PyTuple_GET_ITEM(ob, i) causes a compilation error) and so I closed my PR.

Instead, I opened a thread on python-dev to better explain the whole context and my intent:
https://mail.python.org/pipermail/python-dev/2018-November/155702.html

I invite everyone to discuss there instead :-)

[vstinner]

I created bpo-35206 "Add a new experimental _Py_CAPI2 API".