C API: Remove private C API functions (move them to the internal C API) #106320

vstinner · 2023-07-01T21:35:51Z

Over the years, we accumulated many private functions as part of the public C API header files. I propose to remove them: move them to the internal C API.

If many C extensions are affected by these removals, it's a sign that we should consider promoted used private functions as public functions: provide a better API, add error handling, write documentation, write tests.

Summary: My plan to clarify private vs public functions in Python 3.13

Private functions converted to public functions:
- PyLong_AsInt(): issue [C API] Add PyLong_AsInt() public function #108444
- Py_IsFinalizing(): PR gh-108014: Add Py_IsFinalizing() function #108032
Discussions:
- (pssst) Let’s treat all API in public headers as public
- list of PyPI affected projects
- C API: How much private is the private _Py_IDENTIFIER() API? -- _Py_Identifier
- _Py_c_xxx complex functions (ex: _Py_c_abs())
- PR gh-107137: Add _PyTupleBuilder API to the internal C API #107139: Add _PyTupleBuilder API to the internal C API
Notes:
- While being private, _PyBytes_Resize() and _PyTuple_Resize() are documented
- The private _PyCrossInterpreterData API is used by 3rd party project on PyPI: gh-106320: Remove private _PyCrossInterpreterData API #107068 (comment). I closed my PR gh-106320: Remove private _PyCrossInterpreterData API #107068.

Linked PRs

The text was updated successfully, but these errors were encountered:

Replace calls to the (removed) slow _PyInterpreterState_Get() with fast inlined _PyInterpreterState_GET() function.

vstinner · 2023-07-01T22:04:07Z

See also issue #106316: remove pytime.h header file (_PyTime functions).

vstinner · 2023-07-01T22:04:33Z

See also issue #106084: C API: Remove private functions from abstract.h.

vstinner · 2023-07-01T22:16:19Z

See also commit 0b51463:

Remove the following private functions of the C API:

* _PyCodecInfo_GetIncrementalDecoder()
* _PyCodecInfo_GetIncrementalEncoder()
* _PyCodec_DecodeText()
* _PyCodec_EncodeText()
* _PyCodec_Forget()
* _PyCodec_Lookup()
* _PyCodec_LookupTextEncoding()

Replace calls to the (removed) slow _PyInterpreterState_Get() with fast inlined _PyInterpreterState_GET() function.

* Remove private _PyTracemalloc C API functions: move them to the internal C API. * Rename Include/tracemalloc.h to Include/cpython/tracemalloc.h

* Remove private _PyTracemalloc C API functions: move them to the internal C API. * Don't export private _PyTracemalloc functions anymore. * Rename Include/tracemalloc.h to Include/cpython/tracemalloc.h

* Remove private _PyTraceMalloc C API functions: move them to the internal C API. * Don't export most of these functions anymore, but still export _PyTraceMalloc_GetTraceback() used by tests. * Rename Include/tracemalloc.h to Include/cpython/tracemalloc.h

Remove private _PyThreadState and _PyInterpreterState C API functions: move them to the internal C API (pycore_pystate.h and pycore_interp.h). Don't export most of these functions anymore, but still export functions used by tests. Remove _PyThreadState_Prealloc() and _PyThreadState_Init() from the C API, but keep it in the stable API.

* Remove private _PyTraceMalloc C API functions: move them to the internal C API. * Don't export most of these functions anymore, but still export _PyTraceMalloc_GetTraceback() used by tests. * Rename Include/tracemalloc.h to Include/cpython/tracemalloc.h

Remove private _PyThreadState and _PyInterpreterState C API functions: move them to the internal C API (pycore_pystate.h and pycore_interp.h). Don't export most of these functions anymore, but still export functions used by tests. Remove _PyThreadState_Prealloc() and _PyThreadState_Init() from the C API, but keep it in the stable API.

Replace PyInterpreterState_Get() with inlined _PyInterpreterState_GET().

Remove private _PyThreadState and _PyInterpreterState C API functions: move them to the internal C API (pycore_pystate.h and pycore_interp.h). Don't export most of these functions anymore, but still export functions used by tests. Remove _PyThreadState_Prealloc() and _PyThreadState_Init() from the C API, but keep it in the stable API.

Replace PyInterpreterState_Get() with inlined _PyInterpreterState_GET().

Add internal pycore_complexobject.h header file. Move _Py_c_xxx() functions and _PyComplex_FormatAdvancedWriter() function to this new header file.

…ng() Replace private _PyUnicode_EqualToASCIIString() with public PyUnicode_CompareWithASCIIString().

vstinner · 2023-11-02T14:15:47Z

Does any of the other removed API have a similar issue?

I am not aware of any PEP asking to keep private names prefixed by an underscore.

gvanrossum · 2023-11-02T14:22:26Z

We can talk till we’re blue in the face but please no more action (i.e., no more moving/removing APIs) until the full WG has had a chance to discuss this and make a decision.

(Restoring removed APIs at users’ requests is fine.)

encukou · 2023-11-02T14:49:32Z

FWIW, some of the the user requests are now tracked here: #111481 (comment)

victor · 2023-11-02T17:22:42Z

@victor, you mentioned in python/steering-council#212 that you didn't know _PyObject_Vectorcall was specified in a PEP. (You must have known they were documented, though, since you removed the documentation.)

@encukou I didn’t do any such mention. Please be more careful with @-mentions

gpshead · 2023-11-06T06:56:20Z

Does any of the other removed API have a similar issue?

I am not aware of any PEP asking to keep private names prefixed by an underscore.

That wasn't what was being asked. The question was more: Which of the so far removed APIs were documented.

"private" in the context of breaking changes has nothing to do with the _ prefix (a mere convention) but to do with them being publicly accessible at all outside of our internals via a Python.h include. That means code may have used them regardless of our intent. Documentation is one indicator of intent.

gvanrossum · 2023-11-06T07:42:58Z

Note there is a moratorium on removing APIs until the term “private” has been clarified by the C API WG.

vstinner · 2023-11-06T12:10:44Z

Which of the so far removed APIs were documented.

The only removed functions which were documented in Doc/c-api/ were removed by PR #106085 and PR #106273: the "vectorcall" aliases (doc removed by commit 80b3d8f).

No other removed API were documented in Doc/c-api/. There are a few private documented functions, such as _PyTuple_Resize() and _PyBytes_Resize():

allocation.rst:.. c:function:: PyObject* _PyObject_New(PyTypeObject *type)
allocation.rst:.. c:function:: PyVarObject* _PyObject_NewVar(PyTypeObject *type, Py_ssize_t size)
bytes.rst:.. c:function:: int _PyBytes_Resize(PyObject **bytes, Py_ssize_t newsize)
complex.rst:.. c:function:: Py_complex _Py_c_sum(Py_complex left, Py_complex right)
complex.rst:.. c:function:: Py_complex _Py_c_diff(Py_complex left, Py_complex right)
complex.rst:.. c:function:: Py_complex _Py_c_neg(Py_complex num)
complex.rst:.. c:function:: Py_complex _Py_c_prod(Py_complex left, Py_complex right)
complex.rst:.. c:function:: Py_complex _Py_c_quot(Py_complex dividend, Py_complex divisor)
complex.rst:.. c:function:: Py_complex _Py_c_pow(Py_complex num, Py_complex exp)
frame.rst:.. c:function:: PyObject* PyUnstable_InterpreterFrame_GetCode(struct _PyInterpreterFrame *frame);
frame.rst:.. c:function:: int PyUnstable_InterpreterFrame_GetLasti(struct _PyInterpreterFrame *frame);
frame.rst:.. c:function:: int PyUnstable_InterpreterFrame_GetLine(struct _PyInterpreterFrame *frame);
init.rst:.. c:function:: _PyFrameEvalFunction _PyInterpreterState_GetEvalFrameFunc(PyInterpreterState *interp)
init.rst:.. c:function:: void _PyInterpreterState_SetEvalFrameFunc(PyInterpreterState *interp, _PyFrameEvalFunction eval_frame)
init_config.rst:.. c:function:: PyStatus _Py_InitializeMain(void)
object.rst:.. c:function:: PyObject** _PyObject_GetDictPtr(PyObject *obj)
tuple.rst:.. c:function:: int _PyTuple_Resize(PyObject **p, Py_ssize_t newsize)

Stats on private functions documented with the .. c:function:: markup (it wasn't the case for the removed aliases, that I mentioned):

$ cd ~/python/main/Doc/c-api/; git grep 'function::.*_Py'|wc -l
17

$ cd ~/python/3.12/Doc/c-api/; git grep 'function::.*_Py'|wc -l
17

vstinner · 2023-11-08T11:01:07Z

I propose to revert Python 3.13 C API incompatible changes causing most troubles right now.

gvanrossum · 2023-11-08T16:57:25Z

Yeah, just do it already. :-)

Change the API of the internal _PyDict_Pop_KnownHash() function: return an int. Co-Authored-By: Stefan Behnel <stefan_ml@behnel.de>

Change the API of the internal _PyDict_Pop_KnownHash() function to return an int. Co-Authored-By: Stefan Behnel <stefan_ml@behnel.de>

encukou · 2023-11-13T07:53:10Z

The only removed functions which were documented in Doc/c-api/ were removed by PR #106085 and PR #106273: the "vectorcall" aliases (doc removed by commit 80b3d8f).

No other removed API were documented in Doc/c-api/.

Note that the documented functions _Py_c_sum, _Py_c_diff & others were removed (in #106339), despite what you said here.
I wonder if there are any more.

vstinner · 2023-11-13T11:05:21Z

Oh I forgot the details about _Py_c_sum() functions, they are now discussion in gh-112019. I suggest to move the discussion there.

…#106335) Remove private _PyThreadState and _PyInterpreterState C API functions: move them to the internal C API (pycore_pystate.h and pycore_interp.h). Don't export most of these functions anymore, but still export functions used by tests. Remove _PyThreadState_Prealloc() and _PyThreadState_Init() from the C API, but keep it in the stable API.

…#111162) * pythongh-106320: Re-add _PyLong_FromByteArray(), _PyLong_AsByteArray() and _PyLong_GCD() to the public header files since they are used by third-party packages and there is no efficient replacement. See python#111140 See python#111139 * pythongh-111262: Re-add _PyDict_Pop() to have a C-API until a new public one is designed.

See python/cpython#106320 and python/cpython#107032 Closes #1526

…#106335) Remove private _PyThreadState and _PyInterpreterState C API functions: move them to the internal C API (pycore_pystate.h and pycore_interp.h). Don't export most of these functions anymore, but still export functions used by tests. Remove _PyThreadState_Prealloc() and _PyThreadState_Init() from the C API, but keep it in the stable API.

vstinner added the topic-C-API label Jul 1, 2023

vstinner added a commit to vstinner/cpython that referenced this issue Jul 1, 2023

pythongh-106320: Remove _PyInterpreterState_Get() alias

8cf018e

Replace calls to the (removed) slow _PyInterpreterState_Get() with fast inlined _PyInterpreterState_GET() function.

vstinner added a commit to vstinner/cpython that referenced this issue Jul 1, 2023

pythongh-106320: Remove _PyInterpreterState_Get() alias

c053d2b

Replace calls to the (removed) slow _PyInterpreterState_Get() with fast inlined _PyInterpreterState_GET() function.

vstinner added a commit to vstinner/cpython that referenced this issue Jul 1, 2023

pythongh-106320: Remove _PyInterpreterState_Get() alias

1b38110

Replace calls to the (removed) slow _PyInterpreterState_Get() with fast inlined _PyInterpreterState_GET() function.

bedevere-bot mentioned this issue Jul 1, 2023

gh-106320: Remove _PyInterpreterState_Get() alias #106321

Merged

vstinner added a commit to vstinner/cpython that referenced this issue Jul 1, 2023

pythongh-106320: Remove _PyInterpreterState_Get() alias

096b84a

Replace calls to the (removed) slow _PyInterpreterState_Get() with fast inlined _PyInterpreterState_GET() function.

vstinner added a commit that referenced this issue Jul 1, 2023

gh-106320: Remove _PyInterpreterState_Get() alias (#106321)

18b1fde

Replace calls to the (removed) slow _PyInterpreterState_Get() with fast inlined _PyInterpreterState_GET() function.

bedevere-bot mentioned this issue Jul 2, 2023

gh-106320: Remove private _PyTraceMalloc C API functions #106324

Merged

bedevere-bot mentioned this issue Jul 2, 2023

gh-106320: Remove private _PyInterpreterState functions #106325

Merged

bedevere-bot mentioned this issue Jul 2, 2023

gh-106320: Remove private _PyInterpreterState functions #106335

Merged

vstinner added a commit to vstinner/cpython that referenced this issue Jul 2, 2023

pythongh-106320: Use _PyInterpreterState_GET()

72f9063

Replace PyInterpreterState_Get() with inlined _PyInterpreterState_GET().

vstinner added a commit to vstinner/cpython that referenced this issue Jul 2, 2023

pythongh-106320: Use _PyInterpreterState_GET()

85ea281

Replace PyInterpreterState_Get() with inlined _PyInterpreterState_GET().

bedevere-bot mentioned this issue Jul 2, 2023

gh-106320: Use _PyInterpreterState_GET() #106336

Merged

vstinner added a commit to vstinner/cpython that referenced this issue Jul 2, 2023

pythongh-106320: Use _PyInterpreterState_GET()

606da6d

Replace PyInterpreterState_Get() with inlined _PyInterpreterState_GET().

vstinner added a commit that referenced this issue Jul 2, 2023

gh-106320: Use _PyInterpreterState_GET() (#106336)

bc7eb17

Replace PyInterpreterState_Get() with inlined _PyInterpreterState_GET().

bedevere-bot mentioned this issue Jul 2, 2023

gh-106320: Add pycore_complexobject.h header file #106339

Merged

vstinner added a commit to vstinner/cpython that referenced this issue Jul 2, 2023

pythongh-106320: _testcapi: avoid private _PyUnicode_EqualToASCIIStri…

3ea652f

…ng() Replace private _PyUnicode_EqualToASCIIString() with public PyUnicode_CompareWithASCIIString().

This comment was marked as resolved.

Sign in to view

gpshead mentioned this issue Nov 7, 2023

Is it OK to remove _PyObject_Vectorcall? python/steering-council#212

Closed

vstinner added a commit to vstinner/cpython that referenced this issue Nov 10, 2023

pythongh-106320: Add PyDict_Pop() function

408ef3a

Change the API of the internal _PyDict_Pop_KnownHash() function: return an int. Co-Authored-By: Stefan Behnel <stefan_ml@behnel.de>

vstinner added a commit to vstinner/cpython that referenced this issue Nov 10, 2023

pythongh-106320: Add PyDict_Pop() function

5bd1e51

Change the API of the internal _PyDict_Pop_KnownHash() function to return an int. Co-Authored-By: Stefan Behnel <stefan_ml@behnel.de>

bedevere-app bot mentioned this issue Nov 10, 2023

gh-111262: Add PyDict_Pop() function [with default value] #111939

Closed

skirpichev mentioned this issue Nov 13, 2023

_Py_c_sum, _Py_c_diff, etc - were removed from Python.h #112019

Closed

vstinner mentioned this issue Nov 13, 2023

[C API] Revert of private functions removed in Python 3.13 causing most problems #112026

Closed

vstinner mentioned this issue Nov 14, 2023

gh-112026: Add again <unistd.h> include in Python.h #112046

Merged

clin1234 mentioned this issue Jan 22, 2024

Building cffi from pip fails on Python 3.13 python-cffi/cffi#48

Closed

musicinmybrain mentioned this issue Jan 23, 2024

Internal API _PyBytes_Join() is removed in Python 3.13 pycompression/python-zlib-ng#30

Closed

lachbr pushed a commit to toontownretro/panda that referenced this issue Feb 17, 2024

event: Replace _PyGen_FetchStopIterationValue in Python 3.13+

043417b

See python/cpython#106320 and python/cpython#107032 Closes #1526

rogerbinns mentioned this issue Mar 14, 2024

_PyErr_ChainExceptions1 no longer avallable to extension modules in Python 3.13 alpha headers #116809

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

C API: Remove private C API functions (move them to the internal C API) #106320

C API: Remove private C API functions (move them to the internal C API) #106320

vstinner commented Jul 1, 2023 •

edited

vstinner commented Jul 1, 2023

vstinner commented Jul 1, 2023

vstinner commented Jul 1, 2023

vstinner commented Nov 2, 2023

gvanrossum commented Nov 2, 2023

encukou commented Nov 2, 2023

victor commented Nov 2, 2023

This comment was marked as resolved.

This comment was marked as resolved.

gpshead commented Nov 6, 2023

gvanrossum commented Nov 6, 2023

vstinner commented Nov 6, 2023

vstinner commented Nov 8, 2023

gvanrossum commented Nov 8, 2023

encukou commented Nov 13, 2023

vstinner commented Nov 13, 2023

C API: Remove private C API functions (move them to the internal C API) #106320

C API: Remove private C API functions (move them to the internal C API) #106320

Comments

vstinner commented Jul 1, 2023 • edited

Linked PRs

vstinner commented Jul 1, 2023

vstinner commented Jul 1, 2023

vstinner commented Jul 1, 2023

vstinner commented Nov 2, 2023

gvanrossum commented Nov 2, 2023

encukou commented Nov 2, 2023

victor commented Nov 2, 2023

This comment was marked as resolved.

This comment was marked as resolved.

gpshead commented Nov 6, 2023

gvanrossum commented Nov 6, 2023

vstinner commented Nov 6, 2023

vstinner commented Nov 8, 2023

gvanrossum commented Nov 8, 2023

encukou commented Nov 13, 2023

vstinner commented Nov 13, 2023

vstinner commented Jul 1, 2023 •

edited