Set guidelines to add APIs to the limited C API and to the stable ABI

[vstinner]

Current limited C API guidelines: https://devguide.python.org/developer-workflow/c-api/index.html#limited-api

Last years, when functions were added to the C API, it was common to also add them directly to the limited C API, and so to the stable ABI.

Later, Misc/stable_abi.toml and tests were added: tests fail if an API is added to the limited C API without being added to Misc/stable_abi.toml. Well, it's just to keep Misc/stable_abi.toml consistent, it wasn't really a strict rule to make sure that added APIs were designed with the stable ABI in mind.

Adding an API to the limited C API requires to put a version with an #if, such as:

#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000

Sadly, there is no automated test to check if the version is tested or not, only manual review can catch it. Recent issue: Py_MOD_PER_INTERPRETER_GIL_SUPPORTED.

When new functions are added to the Python C API, should them be added immediately to the limited C API and the stable ABI, as done currently? Or should it be done on a case by case basis? If yes, based on which criteria?

I don't think that we can predict future ABI issues. The best that we can do is to avoid patterns known to cause ABI issues. For example, exposing structure members in the ABI is risky: any structure change is likely to break the ABI. Another example is that PEP 384 – Defining a Stable ABI asks to not use the FILE* type in the limited C API (but file object or file descriptors).

[vstinner]

In Python 3.13, 26 functions were added to Python 3.13 stable ABI: see stable_abi.toml.

---

On my PyList_Extend() PR, @encukou wrote:

Please wait until we have a better process for adding to the limited API -- e.g. until the WG is formed and can discuss this.

So I modified my PR to not add the function the limited C API to Python 3.13.

[zooba]

I don't think that we can predict future ABI issues. The best that we can do is to avoid patterns known to cause ABI issues.

We can't predict with reliable accuracy, but we do have decades of software engineering experience between us, which gives us pretty good pattern recognition to lean on.

---

I think one easy guideline is that new API shouldn't be added to the limited API simultaneously (unless it's specifically being added to support the limited API). Give all new API designs a chance to prove themselves before we commit to not changing them for an extended period of time. One extra version won't hurt anyone.

[encukou]

In fact, I'd prefer that new limited API is vetted by the C-API working group -- preferably in batches, so bigger-picture issues are more apparent.

I would like new limited API to be a shining example of best design practices. That's hard to do in PRs that add individual functions, which can be hurried to meet users' needs.
In limited API, even a year of delay usually isn't a big deal. Let's make the most of it.

[vstinner]

One extra version won't hurt anyone.

PyType_FromModuleAndSpec() was added directly to Python 3.12 limited C API. This function was designed with the limited C API in mind, to add a missing feature asked by multiple projects (especially pybind11 if I recall correctly).

Give all new API designs a chance to prove themselves before we commit to not changing them for an extended period of time.

Do we have examples of API which were identifed as being problematic after a Python release and that we had to deprecate and/or change quickly after the release? Right now, I fail to remind such example.

Even when a non-limited API is broken / bad / a misfit, we still have to support it anyway. Even if it's deprecated, we have to support it for 2 to 3 years anyway. The stable ABI has 62 "ABI only" symbols. It's not like it's a new problem to have to keep functions/variables that we don't want forever :-) Obviously, this function should be as small as possible. By the way, it would be nice to have a process to get rid of them "at some point".

It became rare that newly added functions only make sense for non-limited C API. New functions are now designed to avoid implementation details. New functions which rely on implementation details are now added to the internal C API instead.

---

Some new functions are adding "missing features". Examples in Python 3.13:

• Py_IsFinalizing()
• PyLong_AsInt()
• PyObject_VisitManagedDict()
• PyUnicode_EqualToUTF8()
• PyThreadState_GetUnchecked()
• PySys_AuditTuple()
• PyErr_FormatUnraisable()

They are no replacement with the existing API, or very complicated replacements. I would prefer to be able to use these functions in the limited C API as soon as Python 3.13.

Some functions are more enhancements to the existing C API. For these ones, well, it's less important to add them immediately to the limited C API.

One extra version won't hurt anyone.

My plan is to convert most C extensions to the limited C API. If any limited C API addition takes at least 2 years instead of 1, my plan will be 2x slower, and it's already very slow. So well, it hurts me :-)

[zooba]

They are no replacement with the existing API, or very complicated replacements.

Moving to the non-limited C API is not a complicated replacement. Each one of these should prove for its own sake that it's necessary and it is impossible without them to write certain extension modules that ought to be cross-version compatible.

IMHO, as soon as you're critically dependent on some of these, you are probably not writing a module that will be binary and semantically compatible with multiple Python versions and implementations. I can be convinced otherwise, but simply saying "they aren't there" isn't enough. By that logic, we should treat the entire CPython API as the limited API, which is clearly not feasible.

[encukou]

PyType_FromModuleAndSpec() was added directly to Python 3.12 limited C API. This function was designed with the limited C API in mind, to add a missing feature asked by multiple projects (especially pybind11 if I recall correctly).

I know. I wrote the PEP -- I used the most rigorous process we had available.
If an accepted PEP says something should go directly in the limited API, then yes, it should!

---

And I am talking about changing the status quo. Past examples aren't too relevant -- often they're mistakes to learn from.
Again, IMO, limited API should get extra review -- including the API added in 3.13.

My plan is to convert most C extensions to the limited C API. If any limited C API addition takes at least 2 years instead of 1, my plan will be 2x slower, and it's already very slow. So well, it hurts me :-)

Please adjust your plan to work on volunteer timescales. The urgency is entirely artificial.

[vstinner]

Sorry, I'm no longer sure what we are talking about. I'm talking about making the stable ABI usable for non-trivial code base. Today, there are multiple pain points which make the limited C API complicated to use. Look into python/cpython#85283 for examples. I tried to convert some stdlib extensions to the limited C API but I was quickly blocked by the lack of some stupid functions. For example, I'm now working on an API to format a type name in an error message (type(obj).__name__) in C and a Cython dev said that Cython is also impacted by this issue (when Cython targets the limited C API).

@encukou:

The urgency is entirely artificial.

Rust extensions only use the stable ABI (PyO3). The new nanobind project to bind C++ with Python also only uses the stable ABI. And these projects are affected by missing features in the stable ABI. Wenzel Jakob of nanobind is active on discuss.python.org, example: https://discuss.python.org/t/use-the-limited-c-api-for-some-of-our-stdlib-c-extensions/32465/56

I'm not sure why, but these devs don't ask for missing features. Maybe because of the tone of replies when they asked for features in the past? Or just because nobdy replied to them in the past? I'm not sure.

Well, so far, they used complicated workaround. For example, before PyType_FromModuleAndSpec(), they copied 100 to 200 lines of C code from CPython to fill the lack of this feature. It works, but it's fragile if Python code evolves and they don't update their old copy of the C code.

Cython only has a partial support of the limited C API because it's limited and there are missing features. I'm not sure why Cython didn't come with their usecase "we need an API format a type name in the limited C API". Well, so far, we have to be proactive to fill gaps.

When do you write "The urgency is entirely artificial": do you mean that the stable ABI is not your agenda, and people must continue to workaround missing features?

[encukou]

For example, I'm now working on an API to format a type name in an error message (type(obj).__name__) in C and a Cython dev said that Cython is also impacted by this issue (when Cython targets the limited C API).

Great, thank you!
The WG might have objections when it is formed.
If this feature introduces a new function, I'll ask you to hold off adding it to limited API. So far it seems it doesn't need a new function, so the limited API doesn't apply here.

I tried to convert some stdlib extensions to the limited C API but I was quickly blocked by the lack of some stupid functions.

Well, I don't think the stable ABI should be driven by the needs of the standard library. It's good to dogfood, sure, but IMO the features for Cython are much more important.

I'm not sure why, but these devs don't ask for missing features.

You do have to ask (and then follow up, too).
Wenzel Jakob asked for a long list of features, which I mostly took as a TODO list for 3.12. JPype's Karl Nelson has another list of asks -- alas, many of these don't fall into my immediate area of interest.
I didn't ask more projects for such detailed lists, as my queue was full; but I did ask for general themes to get an idea of which ideas are too specific to one project.

When do you write "The urgency is entirely artificial": do you mean that the stable ABI is not your agenda, and people must continue to workaround missing features?

Yes, I'd rather have them work around missing features for one more year, if that means better API for decades to come. Though i do hope it won't be a year in most cases.

[zooba]

Though i do hope it won't be a year in most cases.

The reality is that it's more likely to be 3-4 years before anyone really adopts the benefits. Nobody is releasing anything right now with a minimum Python version of 3.12 - I believe most are currently working on moving their minimums to 3.8 (apps seem to target "whatever Ubuntu LTS has by default", whether intentionally or indirectly I'm not sure, but that isn't 3.12 either).

So we're already looking at a long-term plan here, and "3-4 years" vs "4-5 years" isn't a huge distinction, particularly when the additions are spread out over multiple releases and all we're really creating is a multi-year process sometime down the track for "does this have all my APIs yet? No? Okay, I'll wait another year then".

[vstinner]

The reality is that it's more likely to be 3-4 years before anyone really adopts the benefits. Nobody is releasing anything right now with a minimum Python version of 3.12 - I believe most are currently working on moving their minimums to 3.8 (apps seem to target "whatever Ubuntu LTS has by default", whether intentionally or indirectly I'm not sure, but that isn't 3.12 either).

Sometimes, I wish a pythoncapi-compat-like library for the stable ABI: provide new functions on top of the existing limited C API. I created capi-workgroup/api-revolution#10 for that.

[davidhewitt]

Rust extensions only use the stable ABI (PyO3).

@vstinner slight correction here: PyO3 does use version-specific private APIs, this is for a mixture of legacy reasons and performance. We have an opt-in feature to use the stable ABI. For example cryptography builds against the stable ABI and pydantic-core builds against the version-specific ABIs.

If it is useful I can audit the functionality we use from "private" APIs so that we can make a plan how to move PyO3 wholly to the limited C API / stable ABI. I also haven't yet done any testing against the 3.13 alphas with so many functions removed.

[vstinner]

@davidhewitt:

@vstinner slight correction here: PyO3 does use version-specific private APIs, this is for a mixture of legacy reasons and performance.

Oh sorry, I didn't know about that. Would you mind to give some examples and explain why the limited C API doesn't fit your these cases?

[davidhewitt]

@vstinner sure thing, I can audit the PyO3 code for usage of private APIs and compile a list of cases for you. I think it might be slightly off topic for this thread; where would you like me to post it? Might be a couple of weeks before I can find a moment to put the list together.

[vstinner]

where would you like me to post it?

As you wish :-)

Might be a couple of weeks before I can find a moment to put the list together.

That's perfectly fine.

[davidhewitt]

@vstinner @encukou thanks for joining my stream yesterday and helping me take a look at the APIs prefixed with _Py in particular.

I've just pushed PyO3/pyo3#3762 (comment) which lists what we found on stream and proposals how to make PyO3 not use those private APIs. Fortunately it's not that long. We can continue any discussion there I think, as it's a bit off topic for this particular thread.