Skip to content
HPy: a better API for Python
Python C C++ Other
Branch: master
Clone or download

Latest commit

Fetching latest commit…
Cannot retrieve the latest commit at this time.


Type Name Latest commit message Commit time
Failed to load latest commit information.
docs move these here for now Nov 15, 2019
proof-of-concept Rename 'HPyNumber_' to simply 'HPy_' Mar 15, 2020
tools Rename 'HPyNumber_' to simply 'HPy_' Mar 15, 2020
.travis.yml Fix notification template Nov 13, 2019
LICENSE rename hpy_devel into hpy.devel Jan 31, 2020
Makefile actually generate the files Nov 15, 2019 Add GraalPython to the list of other Pythons. Mar 17, 2020
azure-pipelines.yml Add argparse.c to the list of source files. Mar 2, 2020

HPy: a better API for Python

Build Status

IRC channel : #hpy on

Mailing list:

The goal of the project is to design a better API for extending Python in C. The current API is specific to the current implementation of CPython: it exposes a lot of internal details which makes it hard:

  • to implement it for other Python implementations (e.g. PyPy, GraalPython, Jython, IronPython, etc.)

  • to experiment with new things inside CPython itself: e.g. using a GC instead of refcounting, or to remove the GIL

The goal of this project is to improve the situation by designing a new API which solves some of the current problems.

More specifically, the goals include (but are not necessarily limited to):

  • to be usable on CPython right now with no (or almost no) performance impact

  • to make the adoption incremental: it should be possible to migrate existing C extensions piece by piece and to use the old and the new API side-by-side during the transition

  • to provide better debugging experience: in debug mode, you could get precise notification about which handles are kept open for too long or used after being closed.

  • to be more friendly for other implementations: in particular, we do not want reference counting to be part of the API: we want a more generic way of managing resources which is possible to implement with different strategies, including the existing reference counting and/or with a moving GC (like the ones used by PyPy or Java, for example)

  • to be smaller and easier to study/use/manage than the existing one

  • to avoid to expose internal details of a specific implementation, so that each implementation can experiment with new memory layout of objects, add optimizations, etc.

  • to be written in a way which could make it possible in the future to have a single binary which is ABI-compatible across multiple Python versions and/or multiple implementations

  • internal details might still be available, but in a opt-in way: for example, if Cython wants to iterate over a list of integers, it can ask if the implementation provides a direct low-level access to the content (e.g. in the form of a int64_t[] array) and use that. But at the same time, be ready to handle the generic fallback case.

More concrete goals

  • we will write a small C library which implements the new API on top of the existing one: no changes to CPython needed

  • PyPy will implement this natively: extensions using this API will be orders of magnitude faster than the ones using the existing old API (see this blog post for details)

  • Cython will adopt this from day one: exiting Cython programs will benefit from this automatically

Why should I care about this stuff?

  • the existing C API is becoming a problem for CPython and for the evolution of the language itself: this project makes it possible to make experiments which might be "officially" adopted in the future

  • for PyPy, it will give obvious speed benefits: for example, data scientists will be able to get the benefit of fast C libraries and fast Python code at the same time, something which is hard to achieve now

  • the current implementation is too tied to CPython and proved to be a problem for almost all the other alternative implementations. Having an API which is designed to work well on two different implementations will make the job much easier for future ones: going from 2 to N is much easier than going from 1 to 2

  • arguably, it will be easier to learn and understand than the massive CPython C API

See also Python Performance: Past, Present, Future by Victor Stinner.

What does HPy means?

The "H" in HPy stands for "handle": one of the key idea of the new API is to use fully opaque handles to represent and pass around Python objects.

You can’t perform that action at this time.