Clone this wiki locally
CEP 523: Native dispatch through Python wrappers around C functions
def square(double x): return x*x import scipy.integrate integrator = scipy.integrate.ode(square) ... integrator.integrate(10)
Now, this is efficient compiled code in Cython, and the (Fortran) code in SciPy calling the callback is also compiled, but because of the Python in the middle each callback pays a major penalty.
- Extend the
defsyntax so that it takes a return value
- Define a protocol for getting access to the underlying cdef function on a Cython def function.
- Allow using this from Cython by allowing casting of (callable) objects to C function pointers. 1. For objects exposing the protocol of 1., this is simple 2. For other objects we need to use libffi and create a thunk
After this, one can also add another optimizing branch for all ``PyObject_Call``s.
def double square(double x): return x*x cdef double (*fncptr)(double) fncptr = <double (*)(double)>square # (A) Directly access underlying cdef function funcptr = <double (*)(double)>somepythonmodule.somefunc # (B) use libffi and create thunk somemodule.otherfunc(2) # (C) Has two branches: If it has the native interface in (2) above, use that, # otherwise do PyObject_Call
Protocol for getting the underlying
cdef of a
print square._native_signature # e.g., "d>d" print square._native_callptr # A Python capsule
Let's leave the string format unspecified until a time comes such that somebody is ready to implement this.
Fast Cython-only protocol
Cython's own C function sub-type should contain this information, and
PyType_Check can then be used for a fast check, useful for case (C) in the example. A goal here is to still achieve somewhat high speed when not using a pxd file for early binding. For this to work well it is therefore necesarry that Cython's own C function sub-type is canonicalized across Cython modules (one can coordinate, e.g., through some
_cythoninternal entry in sys.modules).
Casting Python callables to C function pointers
The case of casting a Cython
def function, supporting the above protocol, is obvious.
In the other case we need to create a thunk using libffi. Since libffi ships with Python on most platforms this can probably be made pretty portable. At any rate this must be made an optional feature as we can't support it everywhere.
Thunking has a lot of potential for optimizing other parts of Cython, so efforts spent here could be used for more things.
bytes->``char*``, we are going to assume it is the users responsibility to hold the Python object for the life of the pointer.
We must use a
WeakKeyDictionary in order to reuse existing thunks (think repeatedly casting a function in a loaded module), and also hold a weak reference that can destruct the thunk when, e.g., a closure object is freed.
cdef float x = someobj(1.0, 3)
We may, e.g., have three cases:
someobjis a Cython function (
- ...and the format string matches "di>f". Then just dispatch directly (do a C cast and call).
- ...otherwise, build the stack using libffi? (Very optional, and potentially a lot faster than boxing/unboxing)
- Fall back to