[C API] Meta issue: add new public functions with doc+tests to replace removed private functions

[vstinner]

Many private functions were removed in issue #106320. My plan is to add public functions for removed private functions which are used by 3rd party extensions: see C API: My plan to clarify private vs public functions in Python 3.13.

I don't think that the fact that a private function was used must always justify to directly expose the exact same API to the public C API. Usually, private functions expose implementation details, use weaker or not check such as checking argument types, are not documented, and are not tested.

To make a function public, we should:

• Follow new C API guidelines to avoid past design mistakes:

  • https://devguide.python.org/developer-workflow/c-api/
  • PEP 733: An Evaluation of Python's Public C API (it's currently a draft)
  • https://github.com/capi-workgroup/api-evolution/issues
  • This part is still a work-in-progress, but there is an agreement on some common good practice, like don't return borrowed references, and design the API so the caller doesn't have to check for exception by calling PyErr_Occurred().

• Document the API.

• Write some basic unit tests (for non regression). The bare minimum is to make sure that the constant exists and call each function at least once.

I also would like to add new public functions to https://pythoncapi-compat.readthedocs.io/ to provide them to Python 3.12 and older (by calling the old private functions).

This work will take time. Depending on how it goes, we should consider reverting some removals if there is too much pressure, to unblock some critical dependencies.

The target is to make most C extensions compatible with Python 3.13 when Python 3.13 beta1 is released. Obviously, sooner the better.

---

C extensions which want to access implementation details can use the internal C API. By default, I modified the internal functions to no longer export internal functions, only make them usable by Python internals: they cannot be used by shared libraries such as stdlib C extensions. This part is unclear to me. Should we expose more internal functions. Which ones? How do you decide if a function should be exported or not?

[vstinner]

Issues asking to add public APIs for removed private APIs.

DONE:

• Py_HashPointer()
• Py_IsFinalizing()
• PyDict_Pop()
• PyDict_PopString()
• PyList_Extend()
• PyList_Clear()
• PyLong_AsInt()
• PyTime API
• PyLong_FromNativeBytes() and PyLong_AsNativeBytes()
• _PyHash constants
• Py_complex API -- private API restored

TODO / WIP:

• Py_HashDouble()
• _PyLong_New()
• _PyArg_Parser

Rejected:

• PyLong_GCD(), but math.gcd(int, int) case was optimized.
• _PyOS_ReadlineTState
• _Py_Identifier

Note: I update this comment time to time.

[vstinner]

See also discussion: Is Python 3.13 going to be Python 4?.

[encukou]

Guido said in #106320 (comment):

I strongly prefer revert over fix, for removed APIs that cause problems.

Can we get consensus on that -- that is, revert the removal for these? Or should we get a SC decision?

[vstinner]

Can we get consensus on that -- that is, revert the removal for these? Or should we get a SC decision?

My plan mentioned at the beginning of this issue is to add public functions asked by users. And if we run out of time before beta1, start to revert and consider adding more public functions later.

Alpha 1 was just released a few days ago. Can we start by adding public functions for now, and only move to the "revert" backup plan later if needed?

Beta 1 is scheduled in May 2024: in 6 months. Usually it takes several months to get more C extensions compatible with the new Python versions. I'm surprised that for Python 3.13, people want suddenly to get "everything" compatible at day 1 of the alpha1 release. Isn't it not a little bit early to expect that?

[gpshead]

I'd much prefer 'revert' for any API anyone is found using in 3.13.

We need to treat 3.13 as a more special than usual release and aim to minimize compatibility headaches for existing project code. That way more things that build and run on 3.12 build can run on 3.13 as is or with minimal work.

This will enable ecosystem code owners to focus on the bigger picture task of enabling existing code to be built and tested on an experimental pep703 free-threading build rather than having a pile of unrelated cleanup trivia blocking that.

[encukou]

I talked to Victor privately, and confirmed that we fundamentally disagree with each other.
@vstinner, correct me if I'm wrong:

Victor believes that the plan (https://discuss.python.org/t/30131) has overwhelming support and no significant opposition, and it doesn't need a PEP.
I believe that is not the case, and it should have been a PEP, to centralize the discussion. (Personally, I'd appreciate a “motivation” section -- a description of the problem being solved -- to be able to suggest alternative approaches. So far I have been unable to find one.)
I have been assured that my opinion was taken into account, but it was decided to go ahead with the plan as it is. By now, I don't see how a personal opinion could change the decision.

[vstinner]

I'd much prefer 'revert' for any API anyone is found using in 3.13.

This issue is about adding public functions for removed private functions. Would you mind to open a new issue to discuss revert? Otherwise it's unclear what is the scope of this issue. Or are you saying that we must not add public functions, but stick to private functions forever?

[vstinner]

I propose to revert Python 3.13 C API incompatible changes causing most troubles right now.

[vstinner]

See also issue gh-112026: [C API] Revert of private functions removed in Python 3.13 causing most problems.

[vstinner]

Indirectly related: Python 3.13 removed deprecated functions to configure the Python initialization, such as Py_SetPath(). The replacement API PyConfig is excluded from the limited C API. I'm working on a replacement for the stable ABI: Add PyInitConfig C API.

[vstinner]

List of removed C functions reverted in Python 3.13 alpha2: #112026 (comment)