Remove cross-version binary compatibility

[pitrou]

BPO	32388
Nosy	@warsaw, @pfmoore, @ncoghlan, @pitrou, @scoder, @vstinner, @tjguk, @ned-deily, @njsmith, @zware, @serhiy-storchaka, @zooba
PRs	bpo-32388: Remove cross-version binary compatibility requirement in tp_flags #4944

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-05-29.20:13:09.708>
created_at = <Date 2017-12-20.17:27:58.959>
labels = ['interpreter-core', 'type-bug', '3.8']
title = 'Remove cross-version binary compatibility'
updated_at = <Date 2019-06-12.13:11:44.251>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2019-06-12.13:11:44.251>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-05-29.20:13:09.708>
closer = 'pitrou'
components = ['Interpreter Core']
creation = <Date 2017-12-20.17:27:58.959>
creator = 'pitrou'
dependencies = []
files = []
hgrepos = []
issue_num = 32388
keywords = ['patch']
message_count = 12.0
messages = ['308753', '308881', '308882', '308884', '313182', '313351', '319944', '319947', '341560', '343918', '343919', '345339']
nosy_count = 12.0
nosy_names = ['barry', 'paul.moore', 'ncoghlan', 'pitrou', 'scoder', 'vstinner', 'tim.golden', 'ned.deily', 'njs', 'zach.ware', 'serhiy.storchaka', 'steve.dower']
pr_nums = ['4944']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32388'
versions = ['Python 3.8']

[pitrou]

See original discussion on python-dev:
https://mail.python.org/pipermail/python-dev/2017-December/151328.html

Summary: binary compatibility for C extensions which don't use the stable ABI is currently "best effort" and uses distinguished flags in the tp_flags field of type objects to indicate whether a field is physically present or not. Unfortunately, tp_flags is 32 bits and therefore a scarce resource. Also, binary compatibility is 1) not guaranteed anyway 2) of less use nowadays.

So we propose to remove the binary compatibility requirement when creating static type objects (i.e. not using the stable ABI).

[zooba]

Totally okay with this in theory, but wanted to clarify that the linked PR is really just taking advantage of removing the guarantee?

That is, the change isn't necessary to remove the requirement.

[pitrou]

That's basically exact. Actually, there is no ABI guarantee at all (when not using the stable ABI), just a best effort for advanced users who know what they're doing.

Actually, this PR stems from Nathaniel's message here:
https://mail.python.org/pipermail/python-dev/2017-December/151345.html

[zooba]

Good to know. As I said on the other issue, Windows won't load extension modules built for a different version anyway unless they use the stable ABI, so there's no real reason these flags need to be binary compatible between 3.x versions.

[scoder]

I'm ok with "removing" this "guarantee" (although it seems too late to apply this specific change to 3.7 now).

While Cython users do ask for a way to build cross-version binaries from time to time, it's neither supported nor planned, and the underlying problem is IMHO solved better via tooling like the manylinux wheel build.

I also generally consider it good to let user code take advantage of new features in new CPython releases, even if it means that they need to retranslate their Cython code from time to time. The 1.5 years release cycle of CPython should exceed that of most "still in maintenance" extension modules out there, and it's long enough to not impose a real burden on their maintainers.

[pitrou]

Yes, let's retarget to 3.8.

[vstinner]

In which version of Python tp_finalize slot has been added? Does it mean that Python 3.8 wil crash on loading C extensions compiled on Python older than this version?

[pitrou]

Does it mean that Python 3.8 wil crash on loading C extensions compiled on Python older than this version?

What makes you think that it would be otherwise be able to load and execute such C extensions without any bugs?

There is no ABI except the stable ABI, and tp_flags isn't part of the stable ABI.

PS : tp_finalize is in Python 3.4, but that doesn't really matter.

[pitrou]

I updated the PR (sorry to let this sleep).

[pitrou]

New changeset ada319b by Antoine Pitrou in branch 'master':
bpo-32388: Remove cross-version binary compatibility requirement in tp_flags (GH-4944)
ada319b

[pitrou]

PR is merged now!

[vstinner]

About ABI "foward compatibility", Python 3.8 introduced another backward incompatible change: bpo-37250 "C files generated by Cython set tp_print to NULL: PyTypeObject.tp_print removed, replaced with tp_vectorcall_offset".