Numba cfuncs not usable as scipy LowLevelCallable on Windows

[dionhaefner]

On most platforms, this seems to work fine (inspired by this blog post):

import numba
from numba import cfunc, carray
from numba.types import intc, CPointer, float64, intp, voidptr
from scipy import LowLevelCallable


def jit_filter_function(filter_function):
    """Decorator for use with scipy.ndimage.generic_filter."""
    jitted_function = numba.jit(filter_function, nopython=True)

    @cfunc(intc(CPointer(float64), intp, CPointer(float64), voidptr))
    def wrapped(values_ptr, len_values, result, data):
        values = carray(values_ptr, (len_values,), dtype=float64)
        result[0] = jitted_function(values)
        return 1
    return LowLevelCallable(wrapped.ctypes)

import numpy as np
import scipy.ndimage as ndi

@jit_filter_function
def fmin(values):
    result = np.inf
    for v in values:
        if v < result:
            result = v
    return result

ndi.generic_filter(np.random.random((20, 30)), fmin, 3)

But on platforms where sizeof(long) == sizeof(int), it fails with

ValueError: Invalid scipy.LowLevelCallable signature "long (double *, long, double *, void *)". Expected one of: ['int (double *, intptr_t, double *, void *)', 'int (double *, npy_intp, double *, void *)', 'int (double *, long, double *, void *)', 'int (double *, long long, double *, void *)']

regardless of the signature I pass to @cfunc. SciPy explicitly expects the return type to be int, not long, but it seems I can't force Numba to produce that (equivalent) signature.

Happy for any suggestions.

[jni]

@sklam thank you for your work yesterday on allowing calls for LowLevelCallable. For skimage getting this issue resolved would also be critical, as we can't incorporate this pattern (a major one for skimage) when it's failing on a massive install base...

[muhammad-tayyab]

I am facing same issue under windows 8.1 64-bit.

[esc]

Two years have now passed w/o any activity on this issue? Does anyone still experience it with recent Numba and SciPy releases?

[jni]

@esc hello! Missed your ping here. I don't have a Windows machine handy to test this. Do you or anyone else on the numba team have a Windows machine handy to test this with recent versions? The MRE at the top of this issue should still work wth all the libraries involved.

[esc]

@jni probably yes, but it may be a while until we get around to it.

[jni]

it may be a while until we get around to it.

Do you mean another three years? 😂

Personally I am comfortable with long-running issues, and prefer that to things being closed without resolution. We have received fixes for 5y+ old issues in skimage! It's very satisfying. 😂

[dhirschfeld]

Looks like the same a similar error on a recent stack:

Python 3.7.7 (default, May  6 2020, 11:45:54) [MSC v.1916 64 bit (AMD64)]
Type "copyright", "credits" or "license" for more information.

IPython 7.17.0 -- An enhanced Interactive Python.

In [3]: import numba
   ...: import numpy as np
   ...: import scipy
   ...: import scipy.ndimage as ndi
   ...: from numba import cfunc, carray
   ...: from numba.types import intc, CPointer, float64, intp, voidptr
   ...: from scipy import LowLevelCallable

In [4]: def jit_filter_function(filter_function):
   ...:     """Decorator for use with scipy.ndimage.generic_filter."""
   ...:     jitted_function = numba.jit(filter_function, nopython=True)
   ...: 
   ...:     @cfunc(intc(CPointer(float64), intp, CPointer(float64), voidptr))
   ...:     def wrapped(values_ptr, len_values, result, data):
   ...:         values = carray(values_ptr, (len_values,), dtype=float64)
   ...:         result[0] = jitted_function(values)
   ...:         return 1
   ...:     return LowLevelCallable(wrapped.ctypes)
   ...:     

In [5]: @jit_filter_function
   ...: def fmin(values):
   ...:     result = np.inf
   ...:     for v in values:
   ...:         if v < result:
   ...:             result = v
   ...:     return result
   ...:     

In [6]: ndi.generic_filter(np.random.random((20, 30)), fmin, 3)
Traceback (most recent call last):

  File "<ipython-input-6-eb72b4db3506>", line 1, in <module>
    ndi.generic_filter(np.random.random((20, 30)), fmin, 3)

  File "C:\Users\dhirschf\envs\dev\lib\site-packages\scipy\ndimage\filters.py", line 1528, in generic_filter
    cval, origins, extra_arguments, extra_keywords)

ValueError: Invalid scipy.LowLevelCallable signature "long (double *, longlong, double *, void *)". Expected one of: ['int (double *, intptr_t, double *, void *)', 'int (double *, npy_intp, double *, void *)', 'int (double *, long long, double *, void *)']


In [7]: np.__version__
Out[7]: '1.19.1'

In [8]: scipy.__version__
Out[8]: '1.5.2'

In [9]: numba.__version__
Out[9]: '0.51.2'

[github-actions]

This issue is marked as stale as it has had no activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with any updates and confirm that this issue still needs to be addressed.

[jni]

👋 I'm unclear whether this is fixed or not because I don't have a regular Windows machine on which to test (much less a recent Windows machine). I can confirm that the reproducer above continues to work fine on Mac (including M1), so it's just a question of whether it works on Windows.

[j08lue]

The issue persists, with the exact same error message, on Windows 11, Python 3.10, numba 0.56.4, numpy 1.23.5, scipy 1.10.0, conda-forge.

[github-actions]

This issue is marked as stale as it has had no activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with any updates and confirm that this issue still needs to be addressed.

[jni]

👋 5 months seems rather aggressive for a stalebot?

[github-actions]

This issue is marked as stale as it has had no activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with any updates and confirm that this issue still needs to be addressed.

[jni]

The issue remains relevant and open...

[github-actions]

This issue is marked as stale as it has had no activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with any updates and confirm that this issue still needs to be addressed.

[jni]

Hello again. I would love for this bot to be less aggressive...

[gmarkall]

I've changed the labels so that the bot will be much more placid towards this issue in future :-)

[edfrench6]

This article has a solution: jni/llc-tools#3. Code below was provided by @trhallam in 2021.

def jit_filter_function(filter_function):
    """Decorator for use with scipy.ndimage.generic_filter."""
    jitted_function = numba.jit(filter_function, nopython=True)

    @cfunc(intc(CPointer(float64), intp, CPointer(float64), voidptr))
    def wrapped(values_ptr, len_values, result, data):
        values = carray(values_ptr, (len_values,), dtype=float64)
        result[0] = jitted_function(values)
        return 1
    return LowLevelCallable(wrapped.ctypes, signature="int (double *, npy_intp, double *, void *)")