While in CPython one always uses PyObject * to reference to Python objects,
in HPy there are several types of handles that should be used depending on the
life-time of the handle: HPy, HPyField, and HPyGlobal.
HPyrepresents short lived handles that live no longer than the duration of one call from Python to HPy extension function. Rule of thumb: use for local variables, arguments, and return values.HPyFieldrepresents handles that are Python object struct fields, i.e., live in native memory attached to some Python object.HPyGlobalrepresents handles stored in C global variables.HPyGlobalcan provide isolation between subinterpreters.
Warning
Never use a local variable of type HPyField, for any reason! If
the GC kicks in, it might become invalid and become a dangling pointer.
Warning
Never store HPy handles to a long-lived memory, for example: C global variables or Python object structs.
The HPy/HPyField dichotomy might seem arbitrary at first, but it is
needed to allow Python implementations to use a moving GC, such as PyPy. It is
easier to explain and understand the rules by thinking about how a moving GC
interacts with the C code inside an HPy extension.
It is worth remembering that during the collection phase, a moving GC might move an existing object to another memory location, and in that case it needs to update all the places which store a pointer to it. In order to do so, it needs to know where the pointers are. If there is a local C variable which is unknown to the GC but contains a pointer to a GC-managed object, the variable will point to invalid memory as soon as the object is moved.
Back to HPy vs HPyField vs HPyGlobal:
HPyhandles must be used for all C local variables, function arguments and function return values. They are supposed to be short-lived and closed as soon as they are no longer needed. The debug mode will report a long-livedHPyas a potential memory leak.- In PyPy and GraalPy,
HPyhandles are implemented using an indirection: they are indexes inside a big list of GC-managed objects: this big list is tracked by the GC, so when an object moves its pointer is correctly updated.HPyFieldis for long-lived references, and the GC must be aware of their location in memory. In PyPy, anHPyFieldis implemented as a direct pointer to the object, and thus we need a way to inform the GC where it is in memory, so that it can update its value upon moving: this job is done bytp_traverse, as explained in the next section.HPyGlobalis for long-lived references that are supposed to be closed implicitly when the module is unloaded (once module unloading is actually implemented).HPyGlobalprovides indirection to isolate subinterpreters. Implementation wise,HPyGlobalwill usually contain an index to a table with Python objects stored in the interpreter state.- On CPython without subinterpreters support,
HPy,HPyGlobal, andHPyFieldare implemented asPyObject *.- On CPython with subinterpreters support,
HPyGlobalwill be implemented by an indirection through the interpreter state. Note that thanks to the HPy design, switching between this and the more efficient implementation without subinterpreter support will not require rebuilding of the extension (in HPy universal mode), nor rebuilding of CPython.
Note
If you write a custom type using HPyField, you MUST also write
a tp_traverse slot. Note that this is different than the old Python.h
API, where you need tp_traverse only under certain conditions. See the
next section for more details.
Note
The contract of tp_traverse is that it must visit all members of
type HPyField contained within given struct, or more precisely owned by
given Python object (in the sense of the owner argument to
HPyField_Store), and nothing more, nothing less. Some Python
implementations may choose to not call the provided tp_traverse if they
know how to visit all members of type HPyField by other means (for
example, when they track them internally already). The debug mode will check
this contract.
Let's quote the Python.h documentation about GC support
Python's support for detecting and collecting garbage which involves circular references requires support from object types which are “containers” for other objects which may also be containers. Types which do not store references to other objects, or which only store references to atomic types (such as numbers or strings), do not need to provide any explicit support for garbage collection.
A good rule of thumb is that if your type contains PyObject * fields, you
need to:
- provide a
tp_traverseslot;- provide a
tp_clearslot;- add the
Py_TPFLAGS_GCto thetp_flags.
However, if you know that your PyObject * fields will contain only
"atomic" types, you can avoid these steps.
In HPy the rules are slightly different:
- if you have a field of type
HPyField, you always MUST provide atp_traverse. This is needed so that a moving GC can track the relevant areas of memory. However, you MUST NOT rely ontp_traverseto be called;tp_cleardoes not exist. On CPython,HPyautomatically generates one for you, by usingtp_traverseto know which are the fields to clear. Other implementations are free to ignore it, if it's not needed;HPy_TPFLAGS_GCis still needed, especially on CPython. If you don't specify it, your type will not be tracked by CPython's GC and thus it might cause memory leaks if it's part of a reference cycle. However, other implementations are free to ignore the flag and track the objects anyway, if their GC implementation allows it.
Generally speaking, if you have one or more PyObject * fields in the old
Python.h, you must provide a tp_dealloc slot where you Py_DECREF all
of them. In HPy this is not needed and will be handled automatically by the
system.
In particular, when running on top of CPython, HPy will automatically provide
a tp_dealloc which decrefs all the fields listed by tp_traverse.
See also, :ref:`dealloc`.
In many cases, migrating to HPy is as easy as just replacing a certain C API function by the appropriate HPy API function. Table :ref:`table-mapping` gives a mapping between C API and HPy API functions. This mapping is generated together with the code for the :term:`CPython ABI` mode, so it is guaranteed to be correct.
The equivalents of Py_INCREF and Py_DECREF are essentially
:c:func:`HPy_Dup` and :c:func:`HPy_Close`, respectively. The main difference is
that :c:func:`HPy_Dup` gives you a new handle to the same object which means
that the two handles may be different if comparing them with memcmp but
still reference the same object. As a consequence, you may close a handle only
once, i.e., you cannot call :c:func:`HPy_Close` twice on the same HPy
handle, even if returned from HPy_Dup. For examples, see also sections
:ref:`api:handles` and :ref:`api:handles vs ``pyobject *```
Both PyObject_Call and PyObject_CallObject are replaced by
HPy_CallTupleDict(callable, args, kwargs) in which either or both of
args and kwargs may be null handles.
PyObject_Call(callable, args, kwargs) becomes:
HPy result = HPy_CallTupleDict(ctx, callable, args, kwargs);
PyObject_CallObject(callable, args) becomes:
HPy result = HPy_CallTupleDict(ctx, callable, args, HPy_NULL);
If args is not a handle to a tuple or kwargs is not a handle to a
dictionary, HPy_CallTupleDict will return HPy_NULL and raise a
TypeError. This is different to PyObject_Call and
PyObject_CallObject which may segfault instead.
Both the tp_call and vectorcall calling protocols are replaced by HPy's
calling protocol. This is done by defining slot HPy_tp_call. HPy uses only
one calling convention which is similar to the vectorcall calling convention.
In the following example, we implement a call function for a simple Euclidean
vector type. The function computes the dot product of two vectors.
.. literalinclude:: examples/snippets/hpycall.c :start-after: // BEGIN EuclideanVectorObject :end-before: // END EuclideanVectorObject
.. literalinclude:: examples/snippets/hpycall.c :start-after: // BEGIN HPy_tp_call :end-before: // END HPy_tp_call
Positional and keyword arguments are passed as C array args. Argument
nargs specifies the number of positional arguments. Argument kwnames is
a tuple containing the names of the keyword arguments. The keyword argument
values are appended to positional arguments and start at args[nargs] (if
there are any).
In the above example, function call_impl will be used by default to call all
instances of the corresponding type. It is also possible to install (maybe
specialized) call function implementations per instances by using function
:c:func:`HPy_SetCallFunction`. This needs to be done in the constructor of an
object. For example:
.. literalinclude:: examples/snippets/hpycall.c :start-after: // BEGIN HPy_SetCallFunction :end-before: // END HPy_SetCallFunction
- It is not possible to use slot
HPy_tp_callfor a var object (i.e. if :c:member:`HPyType_Spec.itemsize` is greater0). Reason: HPy installs a hidden field in the object's data to store the call function pointer which is appended to everything else. In case ofEuclideanVectorObject, a field is implicitly appended after membery. This is not possible for var objects because the variable part will also start after the fixed members.- It is also not possible to use slot
HPy_tp_callwith a legacy type that inherits the basicsize (i.e. if :c:member:`HPyType_Spec.basicsize` is0) for the same reason as above.
To overcome these limitations, it is still possible to manually embed a field
for the call function pointer in a type's C struct and tell HPy where this field
is. In this case, it is always necessary to set the call function pointer using
:c:func:`HPy_SetCallFunction` in the object's constructor. This procedure is
less convenient than just using slot HPy_tp_cal but still not hard to use.
Consider following example. We define a struct FooObject and declare field
HPyCallFunction call_func which will be used to store the call function's
pointer. We need to register the offset of that field with member
__vectorcalloffset__ and in the constructor Foo_new, we assign the call
function Foo_call_func.
.. literalinclude:: examples/snippets/hpycall.c :start-after: // BEGIN FooObject :end-before: // END FooObject
.. literalinclude:: examples/snippets/hpycall.c :start-after: // BEGIN vectorcalloffset :end-before: // END vectorcalloffset
Note
In contrast to CPython's vectorcall protocol, nargs will never have flag
PY_VECTORCALL_ARGUMENTS_OFFSET set. It will only be the positional
argument count.
In order to support incremental migration, HPy provides helper function :c:func:`HPyHelpers_PackArgsAndKeywords` that converts from HPy's calling convention to CPython's tp_call calling convention. Consider following example:
.. literalinclude:: examples/snippets/hpycall.c :start-after: // BEGIN pack_args :end-before: // END pack_args
In this example, args, nargs, and kwnames are used to create a tuple
of positional arguments args_tuple and a keyword arguments dictionary
kwd.
PyModule_AddObject is replaced with a regular :c:func:`HPy_SetAttr_s`. There
is no HPyModule_AddObject function because it has an unusual refcount
behavior (stealing a reference but only when it returns 0).
Py_tp_dealloc essentially becomes HPy_tp_destroy. The name intentionally
differs because there are major differences: while the slot function of
Py_tp_dealloc receives the full object (which makes it possible to resurrect
it) and while there are no restrictions on what you may call in the C API
deallocator, you must not do that in HPy's deallocator.
The two major restrictions apply to the slot function of HPy_tp_destroy:
- The function must be thread-safe.
- The function must not call into the interpreter.
The idea is, that HPy_tp_destroy just releases native resources (e.g. by
using C lib's free function). Therefore, it only receives a pointer to the
object's native data (and not a handle to the object) and it does not receive an
HPyContext pointer argument.
For the time being, HPy will support the HPy_tp_finalize slot where those
tight restrictions do not apply at the (significant) cost of performance.
There is no direct replacement for C API slots Py_tp_methods,
Py_tp_members, and Py_tp_getset because they are no longer needed.
Methods, members, and get/set descriptors are specified flatly together with
the other slots, using the standard mechanisms of :c:macro:`HPyDef_METH`,
:c:macro:`HPyDef_MEMBER`, and :c:macro:`HPyDef_GETSET`. The resulting HPyDef
structures are then accumulated in :c:member:`HPyType_Spec.defines`.
The C API way of creating lists and tuples is to create an empty list or tuple
object using PyList_New(n) or PyTuple_New(n), respectively, and then to
fill the empty object using PyList_SetItem / PyList_SET_ITEM or
PyTuple_SetItem / PyTuple_SET_ITEM, respectively.
This is in particular problematic for tuples because they are actually immutable. HPy goes a different way and provides a dedicated builder API to avoid the (temporary) inconsistent state during object initialization.
Long story short, doing the same in HPy with builders is still very simple and straight forward. Following an example for creating a list:
PyObject *list = PyList_New(5);
if (list == NULL)
return NULL; /* error */
PyList_SET_ITEM(list, 0, item0);
PyList_SET_ITEM(list, 1, item0);
...
PyList_SET_ITEM(list, 4, item0);
/* now 'list' is ready to use */becomes
HPyListBuilder builder = HPyListBuilder_New(ctx, 5);
HPyListBuilder_Set(ctx, builder, 0, h_item0);
HPyListBuilder_Set(ctx, builder, 1, h_item1);
...
HPyListBuilder_Set(ctx, builder, 4, h_item4);
HPy h_list = HPyListBuilder_Build(ctx, builder);
if (HPy_IsNull(h_list))
return HPy_NULL; /* error */Note
In contrast to PyList_SetItem, PyList_SET_ITEM,
PyTuple_SetItem, and PyTuple_SET_ITEM, the builder functions
:c:func:`HPyListBuilder_Set` and :c:func:`HPyTupleBuilder_Set` are NOT
stealing references. It is necessary to close the passed item handles (e.g.
h_item0 in the above example) if they are no longer needed.
If an error occurs during building the list or tuple, it is necessary to call :c:func:`HPyListBuilder_Cancel` or :c:func:`HPyTupleBuilder_Cancel`, respectively, to avoid memory leaks.
For details, see the API reference documentation :doc:`api-reference/hpy-sequence`.
The buffer API in HPy is implemented using the HPy_buffer struct, which looks
very similar to Py_buffer (refer to the CPython documentation for the
meaning of the fields):
typedef struct {
void *buf;
HPy obj;
HPy_ssize_t len;
HPy_ssize_t itemsize;
int readonly;
int ndim;
char *format;
HPy_ssize_t *shape;
HPy_ssize_t *strides;
HPy_ssize_t *suboffsets;
void *internal;
} HPy_buffer;
Buffer slots for HPy types are specified using slots HPy_bf_getbuffer and
HPy_bf_releasebuffer on all supported Python versions, even though the
matching PyType_Spec slots, Py_bf_getbuffer and Py_bf_releasebuffer, are
only available starting from CPython 3.9.
HPy supports only multi-phase module initialization (PEP 451). This means that
the module object is typically created by interpreter from the HPyModuleDef
specification and there is no "init" function. However, the module can define
one or more HPy_mod_exec slots, which will be executed just after the module
object is created. Inside the code of those slots, one can usually perform the same
initialization as before.
Example of legacy single phase module initialization that uses Python/C API:
.. literalinclude:: examples/snippets/legacyinit.c :start-after: // BEGIN :end-before: // END
The same code structure ported to HPy and multi-phase module initialization:
.. literalinclude:: examples/snippets/hpyinit.c :start-after: // BEGIN :end-before: // END