enhancements buffer_external

## page was renamed from CEP 402 - Passing Python buffers to external code

CEP 402 - Passing python buffers to external code

Overview

Currently, there is no way to pass a numpy array or python buffer to an external function without requiring one or more calls to the Python/Numpy API. A more direct way to pass a numpy array or buffer object to an external function would be beneficial, while still handling the array at a high level (i.e. not manually unpacking the data, shape and stride information).

If the cython external function declaration syntax were extended to allow the following:

cdef extern double func(object[int, ndim=2] arr, unsigned int foo, float bar),

then Cython would be able to interface more directly with 'Cython array aware' wrappings of other languages, with very little overhead.

The compiled code in the above case would pass a struct with all the necessary array information to the external function, and it would be the responsibility of the external function wrapper to accept the struct and hand it off to the wrapped function.

Fortran 90/95/2003 supports high-level arrays by associating essential array information (size, number of dimensions, shape of each dimension, etc) with the array 'object' itself. This allows the simple passing of arrays between fortran functions in a natural, uncomplicated way without argument-list 'clutter', and allows using assumed-shape array declarations in functions, etc. This is a boon to numerical computation with Fortran, and is a powerful feature of the language. The ability to call Fortran 90/95/2003 code from Cython without touching the Python API layer and without Python argument unpacking would be a natural extension to the external function interface. Note that there is no Fortran-specific syntax being proposed here, so the impact on Cython is minimal.

As a separate part of this project external to Cython, we will work on extending/enhancing f2py to provide cython buffer-aware wrappers that conform to the ISO C binding interface for the Fortran 2003 standard.

Example

Let 'x' be a numpy array defined in Cython as:

cdef ndarray[int, ndim=2] x = np.zeros((100,100),dtype=int)

Suppose we have a Fortran 90 function 'myfunc':

function myfunc(arr, a, b) result(c)
    implicit none
    integer, dimension(:,:), intent(in) :: arr
    double precision a
    real b
    integer :: c

    !!! myfunc body !!!
end function myfunc

Currently there is no way to wrap the above function myfunc using f2py, since assumed shape arrays (the 'integer dimension(:,:) :: arr' array argument) aren't supported.

One would have to modify 'myfunc' thusly:

function myfunc(arr, n, m, a, b) result(c)
    implicit none
    integer, intent(in) :: n,m
    integer, dimension(n,m), intent(in) :: arr
    double precision a
    real b
    integer :: c

    c = sum(arr)
end function myfunc

Currently, f2py generates a shared object file that is importable by python. The fortran code is behind a fortran object that handles the interfacing with python code. If one wanted to pass the 'x' array to the 'myfunc' code, one would do in Cython:

from fortran_module import myfunc
import numpy as np
cimport numpy as np

cdef np.ndarray[np.int, ndim=2] x = np.zeros((100,100),dtype=np.int)
cdef int out = 0

out = myfunc(x, 10., 3.)

The above is straightforward, but one must go through the "Python layer" to call myfunc(). Within the myfunc wrapper, 'x' is coerced to a numpy array if it is not one already, and the other arguments are passed in as python objects and unpacked to their fortran equivalents. A more direct call method would be ideal.

By enabling external functions to take python buffer objects, one could do the equivalent thusly:

# file fortran_module.pxd
# this file could be generated by an external utility

cdef extern int myfunc(np.ndarray[int, ndim=2] arr, double a, float b)

# file example.pyx

import numpy as np
cimport numpy as np
cimport fortran_module
from fortran_module cimport myfunc

cdef np.ndarray[np.int, ndim=2] x = np.zeros((100,100),dtype=np.int)
cdef int out = 0

out = myfunc(x, 10., 3.)

The external utility is enhanced to generate f90 wrappers and corresponding pxd files that are capable of accepting ndarray objects. A minimum of argument unpacking and Python API calls are generated, resulting in a seamless and unchanged interface that allows calling Fortran code with array arguments.

Notes

Implementation

The following declaration:

cdef extern double func(object[int, ndim=2, mode="strided"] arr, unsigned int foo, float bar),

would mean (in C)

double func(Cython2DStridedBuffer arr, unsigned int foo, float bar)

with additional guarantees from Cython that any passed buffer would have the "int" dtype. Similarily

cdef extern double func(object[int, ndim=2, mode="c"] arr, unsigned int foo, float bar),

could use a Cython2DContiguousBuffer struct instead.

The exact type for Cython2DStridedBuffer and friends would have to be determined. Py_buffer* is a natural choice containing all information one needs and more, however custom structs passed on the stack with pointer, shape and stride may be better (Py_buffer* contains strides in a new array, so that's two pointer lookups). Benchmarks will decide.

Syntax

There are two competing alternatives for syntax:

First:

cdef extern void takes_strided(object[int, ndim=2, mode="strided"] arr)
cdef extern void takes_full(object[int, ndim=2] arr) # not passable to Fortran etc!
cdef extern void takes_strided2(np.ndarray[int, ndim=2] arr) # ndarray makes mode="strided" the default

• Pro: Uses existing and familiar syntax
• Con: It looks like one is passing a PyObject*, but one is not!
• Con: The "magic" about different default modes depending on types may make this a bit vulnerable/confusing

Second:

from cython cimport buffer
cdef extern takes_strided(int[[,]] arr)
cdef extern takes_full(int[::buffer.any,::buffer.any] arr)
cdef extern takes_strided_explicit(int[::buffer.direct,::buffer.direct] arr)

# Then a "mixed" mode: First dim are pointers to contiguous second dim
cdef extern takes_strided_explicit(int[::buffer.indirect,::buffer.direct(1)] arr)

I.e. the "stride" location takes the role of a stride descriptor object which defaults to buffer.direct. This is in alignment with enhancements/buffersyntax, but could be used here independently of that proposal.

Third: If benchmarks show that Py_buffer is just as good for transporting the information also for smallish arrays, one could simply make it

cdef extern takes_anything(Py_buffer* arr)

and add a coercion from Python objects to Py_buffer which involves acquiring the buffer.