[C API] Revert of private functions removed in Python 3.13 causing most problems

[vstinner]

Copy of my Discourse message.

Hi,

My C API: My plan to clarify private vs public functions in Python 3.13 uses Python 3.13 beta 1 (May 2024) as deadline to make most C extensions compatible with Python 3.13.

Problem: I wasn't prepared for the number of people eagger to test their project on Python 3.13 alpha 1 as soon as it was released. Some people are working on porting their own code to Python 3.13, whereas others are blocked by dependencies not compatible yet. For Python 3.12, it took months to get Cython (7 months) and numpy (10 months) to become compatible (with a release), and apparently people were fine with this pace. I expected to have 6 months to go through issues and make decisions on a case by case basis. But Cython was made compatible with Python 3.13 (alpha1) in 1 week! Well, for the Cython case, becoming compatible mostly means "disable optimizations" sadly. I don't want Cython (or other projects) to become slower.

Overall, it's great that people test Python 3.13 alpha 1! I love that, getting feedback as soon as possible! I want to encourage that and help people who are going through issue.

I'm still bug triaging C API reports, whereas some people ask to get Python 3.13 being fixed "right now". My plan is about designing public C API to replace removed privated functions, but this work takes time since we want to design "future proof" API which cover most use cases, expose less implementation details, and following the latest C API guidelines. It's not just "take the private API and put it in the public C API": even if new public functions are added, they will not be drop-in replacements (there might be more work than "remove the underscore prefix").

I tried to communicate as much as I can in advance about my plan removing private C functions. Now that the changes are done, the impact is known better, and we should reevaluate the situation with these new information.

Another problem is that it's unclear to me how C API decisions are taken. Latest C API questions to the Steering Council were delegated to the future C API Working Group, but the Working Group doesn't exist yet.

---

So well, let me propose concrete actions.

What should be done right now:

• Revert immediately C API changes impacting at least 5 projects. My colleague Karolina did a great bug triage work on couting build failures per C API issue by recompiling 4000+ Python packages in Fedora with Python 3.13.
• Consider reverting more changes on a case by case basis.

What should be done next:

• For removed functions which impact at least one project but we decide to not revert, document a solution in What's New in Python 3.13 (even if the function was "private", usually we don't document changes related to private APIs). I expected the existing public C API to be complete enough, but the devil is in the details, and "it's complicated". Some functions are only available in the private/internal C API, there is "no good" replacement in the public C API. Well, that's the part that I'm trying to fix.
• Add better public C API function to replace private C API functions.
• Help projects to migrate to better public C API functions.

What can be done in the long term:

• Clarify what _Py prefix means: does it mean "can be changed anytime, no backward compatibility"? Or does mean "it depends..."?
• Design a new approach to introduce incompatible C API changes. IMO the most practical one would be to introduce the concept of a "compatibility version" and let C extensions opt-in for "future" incompatible changes. It may give more control on who is impacted and when. For example, you can target the "C API version 3.15" and you will be impacted by all planned changes, especially deprecated functions scheduled for removal.
• Design guidelines on how to introduce incompatible changes.

---

The overall plan on private APIs was done in the frame of the private API contract:

Names prefixed by an underscore, such as _Py_InternalState, are private API that can change without notice even in patch releases.

Well. That's the theory, and the practice shows me that users have "different expectations". Apparently, we have to update/complete this contract...

---

If we revert private C API function removals, does it mean that these private functions become part of the public C API and have to stay forever, be documented and tested? Or would they move back to their previous state: not supported and "can change anything (which includes being removed)"?

---

See also:

• Is Python 3.13 going to be Python 4? discussion
• C API: Meta issue to replace removed functions with new clean public functions
• C API: Remove private C API functions (move them to the internal C API): link to recent discussions in this issue. I asked people to create new issues, since this one history is too long, but people prefer to commenting this issues instead.

Linked PRs

• gh-112026: Add again <unistd.h> include in Python.h #112046
• gh-112026: Restore removed private C API #112115
• gh-112026: Restore removed _PyDict_GetItemStringWithError() #112119
• gh-112026: Add again _PyThreadState_UncheckedGet() function #112121
• gh-112026: Update What's New: _PyObject_Vectorcall() was restored #112171

[vstinner]

Sub-issues:

• _Py_c_sum, _Py_c_diff, etc - were removed from Python.h
• steering-council: Is it OK to remove _PyObject_Vectorcall? steering-council#212

[vstinner]

See also issue gh-111481: [C API] Meta issue to replace removed functions with new clean public functions.

[Lincoln-developer]

Hey, I am requesting to work on this issue if you don't mind, thanks.

[vstinner]

Hey, I am requesting to work on this issue if you don't mind, thanks.

I'm actively working on this issue. Reviews are welcome :-)

[vstinner]

So far, 50 private API have been restored in the future Python 3.13 alpha2 release:

• _PyDict_GetItemStringWithError()
• _PyDict_GetItem_KnownHash()
• _PyDict_NewPresized()
• _PyHASH_BITS
• _PyHASH_IMAG
• _PyHASH_INF
• _PyHASH_MODULUS
• _PyHASH_MULTIPLIER
• _PyLong_Copy()
• _PyLong_FromDigits()
• _PyLong_New()
• _PyLong_Sign()
• _PyObject_CallMethodId()
• _PyObject_CallMethodNoArgs()
• _PyObject_CallMethodOneArg()
• _PyObject_CallOneArg()
• _PyObject_EXTRA_INIT
• _PyObject_FastCallDict()
• _PyObject_GetAttrId()
• _PyObject_Vectorcall()
• _PyObject_VectorcallMethod()
• _PyStack_AsDict()
• _PyThreadState_UncheckedGet()
• _PyThread_CurrentFrames()
• _PyUnicodeWriter structure
• _PyUnicodeWriter_Dealloc()
• _PyUnicodeWriter_Finish()
• _PyUnicodeWriter_Init()
• _PyUnicodeWriter_Prepare()
• _PyUnicodeWriter_PrepareKind()
• _PyUnicodeWriter_WriteASCIIString()
• _PyUnicodeWriter_WriteChar()
• _PyUnicodeWriter_WriteLatin1String()
• _PyUnicodeWriter_WriteStr()
• _PyUnicodeWriter_WriteSubstring()
• _PyUnicode_AsString()
• _PyUnicode_FromId()
• _PyVectorcall_Function()
• _Py_HashDouble()
• _Py_HashPointer()
• _Py_IDENTIFIER()
• _Py_c_abs()
• _Py_c_diff()
• _Py_c_neg()
• _Py_c_pow()
• _Py_c_prod()
• _Py_c_quot()
• _Py_c_sum()
• _Py_static_string()
• _Py_static_string_init()

[tacaswell]

aio-libs/multidict is also using _PyArg_Parser and friends (e.g. https://github.com/aio-libs/multidict/blob/18d981284b9e97b11a4c0cc1e2ad57a21c82f323/multidict/_multidict.c#L452-L456 but there are many)

[vstinner]

aio-libs/multidict is also using _PyArg_Parser and friends (e.g. https://github.com/aio-libs/multidict/blob/18d981284b9e97b11a4c0cc1e2ad57a21c82f323/multidict/_multidict.c#L452-L456 but there are many)

I created issue #112136 for the removed _PyArg_Parser API.

[vstinner]

Python 3.13 alpha2 was released with these reverts. Apparently, it unblocked many projects. So I close the issue.