Opt-in macro for a new clean API? Subset of functions with no known issues

[vstinner]

The Python C API has a long history. Many functions known to be ackwards are kept because of "backward compatibility". The cost of removing them is higher than the benefits.

My use case is: I would like to write a new C extension with only APIs with no known issues. For example, I would like to avoid borrowed references, ambiguous return values, etc.

Can we define a subset of the Python C API for that?

It can be a practical implementation of issue #53: make the C API smaller. Such subset would be smaller by design.

In such subset, we can consider to have different a different deprecation policy: more aggressive. Developers who opt-in for such API are aware of the ongoing effort to fix the C API and so are more eagger to update their code than unmaintained projects which continue to work only because the C API didn't break so far.

In my first C API hack in 2017, I added multiple macros to opt-out from APIs known to have issues. Examples:

• Py_NEWCAPI: future-proof name, right? :-)
• Py_NEWCAPI_NO_STRUCT
• Py_NEWCAPI_NO_MACRO
• Py_NEWCAPI_BORROWED_REF

I'm not sure if such design is future-proof, C extensions may not be aware that next opt-out options would be added. Maybe it's better to have a single big button: "drop all legacy, give me only good API".

Question: if we design such subset, would it be ok to override names of the regular API? For example, can we rename PyDict_GetItemRef() (regular API) as PyDict_GetItem()? How would it work at the ABI level? Would it be confusing when exchanging code written with the two APIs?

In issue #52, @markshannon proposed using different name prefixes. Rather than using suffixes.

So maybe PyDict_GetItemRef() could become PyNew_Dict_GetItem(), PyNewFinal2CorrectThisThime_Dict_GetItem()? :-)

[markshannon]

Please don't use New. It won't always be new.
E.g. New College is over 700 years old.

[vstinner]

Oh sorry, I wasn't clear. The "Py_NEWCAPI" macro was a temporary name that I chose in 2017 in my experimental project until I could come up with a better name. It was just to show the option of having many macros. I don't know if we need 1 or many of them.

[vstinner]

Windows <windows.h> has something like that: #define WIN32_LEAN_AND_MEAN:

Define WIN32_LEAN_AND_MEAN to exclude APIs such as Cryptography, DDE, RPC, Shell, and Windows Sockets.

https://devblogs.microsoft.com/oldnewthing/20091130-00/?p=15863

As I recall, on a 50MHz 80486 with 8MB of memory, switching to WIN32_LEAN_AND_MEAN shaved three seconds off the compile time of each C file. When your project consists of 20 C files, that’s a whole minute saved right there.

[encukou]

This seems like a solution. AFAIK, the point of this repo is to problems, so we can design solutions that address multiple related problems at once.

IMO, it would be good to keep the actual problems separate: in that case you could take a proposed solution, and check which problems it solves, keeps, or makes worse.

[encukou]

Py_LIMITED_API was already used for this, back when the set of "known problems" was much smaller. But nowadays, deprecating things from it is difficult. It “upper” bound: Py_LIMITED_API=X basically means “nothing newer than X”. The “lower” bound -- “avoid API considered problematic in X” -- is stuck on Python 3.2.
How about adding such a “lower” bound, reusing Py_LIMITED_API's versioning scheme?

Something like Py_NO_LEGACY=0x30c0000?

[vstinner]

If a single opt-in macro causes too many changes, maybe it would be better to have a smoother migration plan by having multiple macros to only exclude a small set of functions. Examples:

• PyCAPI_NO_BORROW_REF: Remove functions returning borrowed references, like PyWeakref_GetObject()
• PyCAPI_NO_COMPAT_ALIAS: Remove aliases kept for backward compatibility, like _PyUnicode_AsString (alias to PyUnicode_AsUTF8)
• PyCAPI_NO_PYOBJECT_CAST: No macro casting function arguments to PyObject. Maybe this one is too annoying, having to call Py_INCREF() only with PyObject* is annoying :-)

And maybe we can also have a "superset" which combines all other opt-in macros.

Something like Py_NO_LEGACY=0x30c0000?

Yeah, I think that it's important that a C extension can do the work once for a specific API version and let expect to have no new error/warning until the maintainers decide to upgrade to the next Py_NO_LEGACY version.

API versionning sounds like a good idea.

The C API is big and it's hard to keep track of each C API changes.

[vstinner]

See also issue #62: How can an user access old removed functions? Can a 3rd party project provide them?

[davidhewitt]

I just wanted to note that with these proposed macros to remove functions, it might be easier for other language bindings if the macros do indeed only remove functions (and not redefine them).

A recent example which we had to deal with in PyO3 was the PY_SSIZE_T_CLEAN macro, which had the effect of redefining PyObject_CallFunction as _PyObject_CallFunction_SizeT. Given the PyObject_CallFunction symbol continued to exist in the ABI, it took us some time to notice that we were calling an outdated symbol when functionality changed on Python 3.10.

[vstinner]

When reviewing python/cpython#108768, I would like to have a macro to opt-out for C API known to ignore exceptions silently: PyDict_GetItem(), PyMapping_HasKey(), PyObject_HasAttrString(), etc.

Even if I'm following closely the Python development, I know the C internals of Python, but it's still really hard to me to remind which functions have a clean error management, and which ones have a crappy error management (ignore silently errors).

[scoder]

Py_LIMITED_API was already used for this, back when the set of "known problems" was much smaller. But nowadays, deprecating things from it is difficult. It “upper” bound: Py_LIMITED_API=X basically means “nothing newer than X”. The “lower” bound -- “avoid API considered problematic in X” -- is stuck on Python 3.2.

Does it have to be? If someone sets Py_LIMITED_API to, say, Python 3.15, and Python 3.15 has decided to remove a "stable" C-API function after a deprecation period, then that function can be excluded from the 3.15+ API and only left in #if Py_LIMITED_API < 3.15.

That doesn't solve the problem of having to keep the function available in the ABI, but then, if there's a reason to remove the function, there's probably also a reason to break the ABI for it. But that's the general ABI problem, not a problem with Py_LIMITED_API or any other versioning macro.

[encukou]

Proposed solution issue: capi-workgroup/api-evolution#24

[vstinner]

Proposed solution issue: capi-workgroup/api-evolution#24

I close this issue, let's continue the discussion there.