dev call 20210107

Dev Call: 7 January 2021

Present

Antonio, Carl, Florian, Matti, Ronan, Simon, Tim

Trial port of Numpy

Ronan has been doing a trial port of numpy to HPy. The code is available at https://github.com/hpyproject/numpy-hpy/ and there is a pull request containing the initial work, https://github.com/hpyproject/numpy-hpy/pull/2.

So far the PyArray type spec has been converted to an HPy type spec. This paves the way for incrementally migrating the individual array type methods.

A few hacks are required because PyType_FromSpec does not yet support all the type slots numpy uses:

• tp_as_buffer (buffer protocol methods) could not be set using PyType_Slot before Python 3.9. In 3.9 it may now be set using the unlimited API.
• tp_weaklistoffset can not be set using PyType_Slot.

We should investigate extending HPyType_FromSpec to include the cases that PyType_FromSpec does not currently support. We should also ask Victor if he knows what C Python's plans are regarding PyType_Slot and whether the intention is for all slots to be supported eventually.

See https://docs.python.org/3/c-api/type.html#c.PyType_Slot.PyType_Slot.slot for the details on the limitations of PyType_Slot.

The pull request currently only works on C Python because it uses _HPy_GetContext but this should be fairly easy to fix by making the array module an HPy module.

An important next step is to get benchmarks running. We don't expect any performance degredation, but performance is important for numpy, so we should check that there have been no accidental regressions. Numpy has a benchmark suite and to start with we can just run its array benchmarks.

The numpy port has reminded us that we need documentation on how to port an extension, and in particular to make clear that one should start by porting the module definition itself (so that the module becomes an HPy module filled with legacy C API pieces).

There was a discussion of whether we should change "hpy.h" to "HPy.h" in order to match "Python.h". There were no strong opinions, but the concensus seemed to be mildly for keeping "hpy.h".

Debug mode

Antonio has been working on the long awaited debug mode. After trying a few different approaches, it is now written as a C wrapper for the HPyContext.

This wrapper is statically compiled as part of each HPy universal implementation, but the intention is that the C code can be shared as is across all implementations (CPython, PyPy, GraalPython, etc).

See https://github.com/hpyproject/hpy/pull/142 for the progress so far.

HPy_CallTupleDict

After a long discussion in https://github.com/hpyproject/hpy/issues/122, it was decided to implement HPy_CallTupleDict(HPy callable, HPy args, HPy kw) where args is a tuple (or null handle) and kw is a dict (or null handle).

The implementation is up for review at https://github.com/hpyproject/hpy/pull/147.

Florian mentioned that calling with a tuple and dict might be faster for GraalPython than calling with C objects because tuple and dict are managed objects (i.e. managed by the JVM) while C objects come with some overhead. From the discussion it was unclear how this trade-off would play out in different scenarios (e.g. if the C objects already exist and have to be packed into a tuple or dictionary) and we'll probably have to wait and see how things turn out in practice.

Updating GraalPython

GraalPython's HPy universal implementation has fallen a bit behind CPython and PyPy's over the holidays, but the intention is that Simon and Florian will update it in the coming week.