Move the pythoncapi_compat project under the GitHub python or psf organization?

[vstinner]

Hi,

Would it make sense to move the pythoncapi_compat project under the GitHub python or psf organization to make it more "official" and make it a little more sustainable?

In the past, I managed to move my pyperf project under the psf organization. Now other core developers are contributing and using it. It's better than using it on my personal organization.

pythoncapi_compat respects the PSF requirements to move a project in the GitHub psf organization: contributors are requires to respect the PSF Code of Conduct and the project has 3 maintainers (@corona10, @erlend-aasland and me: @vstinner).

On Discord, @Yhg1s and @gpshead told me that this question is for the SC, so I'm asking the question here :-)

---

Some context.

C API changes in Python 3.10 and 3.11 require adding compatibility code to write C code compatible with old and new Python versions.

For example, What's New in Python 3.10 suggests adding the following code to your project to get Py_SET_REFCNT() on Python 3.8 and older:

#if PY_VERSION_HEX < 0x030900A4
#  define Py_SET_REFCNT(obj, refcnt) ((Py_REFCNT(obj) = (refcnt)), (void)0)
#endif

Python 3.11 requires even more compatibility functions:

• Py_SET_TYPE()
• Py_SET_SIZE()
• PyFrame_GetCode()
• PyFrame_GetBack()
• PyThreadState_GetFrame()
• PyThreadState_EnterTracing()
• PyThreadState_LeaveTracing()

I added a reference to my pythoncapi_compat project to What's New in Python 3.11:

Or use the pythoncapi_compat project to get these functions on old Python functions.

This project provides a pythoncapi_compat.h header file which contains all the required compatibility code. There is no need to manually copy/paste it in your code, you just need to add #include "pythoncapi_compat.h" and copy the file to your project.

My concern is that currently the project lives in my https://github.com/pythoncapi organization which is not really a sustainable option for the future.

---

The pythoncapi_compat project is used by more and more Python projects, like Mercurial or mypy.

Victor

[corona10]

Personally, I prefer python org because the pythoncapi_compat project is very close to CPython's CAPI.

[brettcannon]

Added to the agenda.

[encukou]

AFAICS, the reason to move it to Python/PSF is to send some kind of signal. What kind of signal would that be? That the core devs or the PSF now maintain the library?

As for the project itself: does it have some documentation other than the README? There's a mix of usage notes, contribution guidelines and research/implementation notes. It seems to be aimed at people that are already familiar with the project.

What is the scope and backwars compatibility policy of pythoncapi_compat? Should it include all changes since 3.5 forever? Or since 2.7? Should it support changes to private API?

[vstinner]

AFAICS, the reason to move it to Python/PSF is to send some kind of signal. What kind of signal would that be? That the core devs or the PSF now maintain the library?

The signal is that the Python core devs bless and trust this project to be a good solution for the problem that it is trying to solve.

As for the project itself: does it have some documentation other than the README? There's a mix of usage notes, contribution guidelines and research/implementation notes. It seems to be aimed at people that are already familiar with the project.

README is the only documentation. I understand that you suggest making the documentation easier to read for users of this project.

What is the scope and backwars compatibility policy of pythoncapi_compat? Should it include all changes since 3.5 forever? Or since 2.7?

My plan is to keep Python 2.7 support until it breaks, if maintaining it remains manageable. pythoncapi_compat is used by Mercurial which still supports Python 2.7. So far, supporting Python 2.7 remains easy as soon as a CI makes sure that it still works.

Should it support changes to private API?

My plan is to only support the public API in the pythoncapi_comapt.h header file. pythoncapi_compat is written to have a single code base without #if on the Python version.

If a private API is replaced with a new clean public API, pythoncapi_compat.h should get an implementation for older Python versions.

If a private API is changed or removed and there is no public API, IMO users must still use the private API or even switch to the internal C API. Nothing is added to pythoncapi_compat.h.

About the upgrade_pythoncapi.py script, so far, I tried to only make changes when it's possible to have a new code compatible with old and new Python versions. For example, if you replace _PyObject_Vectorcall with PyObject_Vectorcall, you drop support with Python 3.8.

Or maybe it could be an upgrade_pythoncapi.py option to only target Python 3.9 and newer: in this case, the change makes sense. The pyupgrade project has such option. For example, --py36-plus will remove Python <= 3.5 only blocks.

[vstinner]

README is the only documentation. I understand that you suggest making the documentation easier to read for users of this project.

I created a Sphinx project and reorganized the documentation to make it a little bit more users friendly: https://pythoncapi-compat.readthedocs.io/

[encukou]

The signal is that the Python core devs bless and trust this project to be a good solution for the problem that it is trying to solve.

Did you ask the core devs? python-dev or Discourse would probably be the place for such a question.
(Yes, I'm aware I'm probably giving you different advice than other SC members. Unfortunately, only the whole SC can give official answers/advice.)
(If you ask me, this solution to the "private API is replaced with a new clean public API" problem is interesting, but for other issues I'd prefer that we don't create the issues in the first place. But you probably knew that already...)

I understand that you suggest making the documentation easier to read for users of this project.

IMO, pinpointing the problem that it solves, and the scope, is more important now. The Docs team can work on the docs, if we make it their job ;)

[vstinner]

but for other issues I'd prefer that we don't create the issues in the first place. But you probably knew that already...)

I suppose that you're talking about the incompatible C API changes that I introduced in Python 3.10 and 3.11 (Py_SET_REFCNT, Py_SET_TYPE, Py_SET_SIZE).

See my first message, Python 3.11 now requires to call the following functions rather than accessing directly PyFrameObject and PyThreadState structures:

• PyFrame_GetCode()
• PyFrame_GetBack()
• PyThreadState_GetFrame()
• PyThreadState_EnterTracing()
• PyThreadState_LeaveTracing()

These changes were motived by optimizing Python.

The question is not if these changes are required or not. For example, the PyThreadState changes about "tracing" already landed in the released Python 3.10 version: see https://bugs.python.org/issue43760 Sadly, Python 3.10 doesn't provide a new clean public API for these changes, I only added them in Python 3.11. PyThreadState_EnterTracing() now has 3 implementations. Simplified code:

• Use tstate->use_tracing on Python 3.9 and older
• Use tstate->cframe->use_tracing on Python 3.10
• Use PyThreadState_EnterTracing() on Python 3.11

Using pythoncapi_capi.h, you can use PyThreadState_EnterTracing() on all Python versions.

[vstinner]

Did you ask the core devs? python-dev or Discourse would probably be the place for such a question.

See my first message:

On Discord, @Yhg1s and @gpshead told me that this question is for the SC, so I'm asking the question here :-)

[encukou]

Well, frame and tstate were documented as private (subject to change at any time), so those look great to me :)

On Discord, @Yhg1s and @gpshead told me that this question is for the SC, so I'm asking the question here :-)

But did they know that you the move means that "core devs bless and trust this project"?
It's up to the SC do decide what moving to a shared space means, but if it agrees with you, IMO it would be good to ask the core devs :)

(I didn't read the Discord conversation. I can't keep up with Discord and leave a lot of discussions unread there. )

[vstinner]

In June 2021, I created a thread explicitly about pythoncapi_compat and in the email. In this email, I asked if it would make sense to move the project under the PSF organization: New pythoncapi_compat project adding Python 3.10 support to your C extensions without losing Python 2.7-3.9 support.

I mentioned the pythoncapi_compat project multiple times in previous python-dev discussions, to propose using it as a practical solutions to C API compatibility issues:

• January 2021: Please explain how to migrate when a function is removed, thanks ;-)
• June 2020: PEP 620: Hide implementation details from the C API
• January 2022: Slowly bend the C API towards the limited API to get a stable ABI for everyone

I also mentioned pythoncapi_compat on the capi-sig list:

• September 2021:Py_TYPE() and Py_SIZE() become static inline functions
• December 2020: (in my replies to) Pre-PEP: Maintaining the Stable ABI
• December 2020: (creation of the project) New script: add Python 3.10 support to your C extensions without losing Python 3.6 support
• June 2020: (before the project creation) How can we ship a header file, or even a dynamic library, to port existing C extension modules to the new incompatible Python C API?

And others emails to python-dev and capi-sig.

What changed recently is that the quantify of compatibility code needed to write C code compatible with Python 3.11 and older Python versions increased a lot: see my first message.

[brettcannon]

In June 2021, I created a thread explicitly about pythoncapi_compat and in the email.

You probably need to ask again then, or assume people don't want it moved under the Python org to prevent from feeling responsible for it. You could open a poll on discuss.python.org to try and make it very clear you're after an answer for pulling the repo over. But I could keep the messaging short and to the point.

[vstinner]

You probably need to ask again then, or assume people don't want it moved under the Python org to prevent from feeling responsible for it.

Right. I started a new thread on python-dev:
https://mail.python.org/archives/list/python-dev@python.org/thread/XI23GU54UC2Y6UBMPY6US4IK4PSUXXPI/

You could open a poll on discuss.python.org to try and make it very clear you're after an answer for pulling the repo over. But I could keep the messaging short and to the point.

Good idea, I created a poll on Discourse:
https://discuss.python.org/t/move-the-pythoncapi-compat-project-under-the-github-python-or-psf-organization/13651

It will close automatically in 10 days.

[vstinner]

Good idea, I created a poll on Discourse:
https://discuss.python.org/t/move-the-pythoncapi-compat-project-under-the-github-python-or-psf-organization/13651
It will close automatically in 10 days.

Results of the poll:

• Move pythoncapi_compat: 19 votes (90%)
• Don't move pythoncapi_compat: 2 votes (10%)

@brettcannon: What's the next step? Should the SC approve moving this project?

[brettcannon]

I'll make sure it's on the agenda and I guess we have a chat as to whether we think it belongs under python/ or suggest it for psf/ and what we would want in terms of project structure, expectations, etc.

[Yhg1s]

The SC decided the python org is the best place to put it, and I've asked @ewdurbin if they could work with @vstinner to get this done.