Macro to hide deprecated functions

[encukou]

(from capi-workgroup/problems#54)

New meta-API idea:

Define a macro like Py_NO_LEGACY=0x30c0000 to hide definitions of all API that's deprecated and soft-deprecated in that version.
This is strictly optional, meant for people who tend to like linters.
Users can to set it for new projects to e.g. hide the unnecessary definitions from editor autocompletion.

If and when we add "modern" alternatives to everything, it can make it easier to review things like refcounting.

What's a good name for it?

[encukou]

Victor proposed fine-grained flags like PyCAPI_NO_BORROW_REF, PyCAPI_NO_COMPAT_ALIAS, PyCAPI_NO_PYOBJECT_CAST but IMO that's too much detail :)

[zooba]

Is there any value in specifying a version, and not just having a flag to totally hide anything that would trigger a deprecation warning in the current headers (if we did those reliably in C)?

My initial thought was that of course we should allow a version, but I'd prefer it to be semantically more like the limited API one (so set Py_CURRENT_API=0x030C0000 to only allow undeprecated APIs as of that version). But then I tried to work through a bigger example and couldn't come up with a feasible case where I'd expect to set the version lower than the current version and still have things work (obviously we can't set the version higher than the current one, because deprecations don't get backported).

So maybe just Py_NO_DEPRECATED_API=1 is enough?

[encukou]

I imagine users will set this when they start a new project, and then they won't update it. In a few years when we declare there's now a slightly better way of doing something, they won't be bothered with warnings – but new projects won't see the old API.

If we need to be more forceful – for API that's unsafe, dangerous in most uses, or blocking development – we have regular deprecation warnings.

[zooba]

Ah I see, so we hide any definitions that were soft-deprecated before the version, and hide any warnings for soft deprecations after the version.

In that case, maybe something like Py_TARGET_VERSION=? Or Py_WRITTEN_FOR_VERSION=? And clear documentation that the intent is for it to be a hard coded constant rather than something that automatically tracks the current build version.

[encukou]

Py_TARGET_VERSION looks like minimum supported version (Py_LIMITED_API).
Py_WRITTEN_FOR_VERSION looks like it won't compile with other versions.

Rust's edition is similar but different – it changes behaviour.

In my sprint slides I used epoch ("an event or a time marked by an event that begins a new period or development" or "an instant of time or a date selected as a point of reference").

---

and hide any warnings for soft deprecations after the version.

Soft deprecations are already only mentioned in docs. (And picked up by linters, hopefully, soon.)

[zooba]

"Epoch" is already used in packaging as an optional first part of a version number, which leads some to consider the "3" in 3.x to be the epoch.

Py_TARGET_VERSION looks like minimum supported version

What value would there be in setting it to a later version that the earliest that you support?

Soft deprecations are already only mentioned in docs.

Then are we suppressing warnings for hard deprecations with this?

[encukou]

What value would there be in setting it to a later version that the earliest that you support?

You'd do that if you want to use new features, but don't want to rewrite all of your code.

Then are we suppressing warnings for hard deprecations with this?

No. We are hiding (not exposing) deprecated and soft-deprecated API.

[zooba]

I think I might need a worked example to better illustrate how this would work over the period of a few CPython releases.

You'd do that if you want to use new features, but don't want to rewrite all of your code.

We've already said that the limited API needs to be compiled with the earliest version you want to support, so I guess we're not talking about that here.

In the regular API, we require compiling with the headers from the version that build is going to support. So if I'm compiling with the 3.13 headers, but set the flag to 3.14, what does that achieve? We don't have metadata in the 3.13 headers relating to 3.14 deprecations, soft or otherwise. Are we going to start backporting it? If not (and I hope not), what value is there?

If you want to use 3.14 features, you need to drop 3.13 (or do your own detection). At that point, your minimum version matches the flag again. I don't understand how it helps to hide 3.14 deprecations when building for 3.13, unless we're going to backport them.

[encukou]

Example

I'm building a new extension for 3.20+.
For API compatibility with existing extensions, 3.20 still exposes the venerable PyDict_GetItem. It still works, especially in single-threaded application, but it's discouraged.
I set Py_NO_LEGACY to 3.20, so I don't see PyDict_GetItem and its ilk. And since I know that 3.18 soft-deprecated all API that returns borrowed references, my refcounting reviews are much easier!

Some time later

I've written the extension for 3.20.
Now I want to update it to require 3.21, which adds a cool feature I can't live without.
As it happens, 3.21 also added a glorious new API for converting numbers to native types, and finally soft-deprecated PyLong_AsLong. But I use PyLong_AsLong all the time, do the weird error-checking dance it requires, and I really don't want to rewrite all my existing code.
I keep Py_NO_LEGACY set to 3.20, but bump Py_MIN_TARGET_VERSION to 3.21 to get the cool feature.

[zooba]

Thanks. Having two independent variables makes sense.

So with that, setting Py_MIN_TARGET_VERSION=3.20 and Py_NO_LEGACY=3.21 simultaneously means that when youbuild on 3.21 you can't use newly deprecated functions, and therefore when you build on 3.20 (still with Py_NO_LEGACY=3.21) even though you can still see the not-yet-deprecated functions, you know you're not using them.

[vstinner]

I proposed a concrete PEP to implement this idea: PEP 743 "Add Py_COMPAT_API_VERSION to the Python C API". You can join the discussion at: https://discuss.python.org/t/pep-743-add-py-compat-api-version-to-the-python-c-api/48243