How to call a cython function from within @njit compiled code?

[tsoernes]

from Levenshtein import matching_blocks
import numba

mb = numba.njit()(matching_blocks)

where matching_blocks is a Cython function, yields

File /home/torstein/anaconda3/lib/python3.7/site-packages/numba/decorators.py, line 192, in wrapper
    **dispatcher_args)
File /home/torstein/anaconda3/lib/python3.7/site-packages/numba/dispatcher.py, line 652, in __init__
    pysig = utils.pysignature(py_func)
File /home/torstein/anaconda3/lib/python3.7/inspect.py, line 3083, in signature
    return Signature.from_callable(obj, follow_wrapped=follow_wrapped)
File /home/torstein/anaconda3/lib/python3.7/inspect.py, line 2833, in from_callable
    follow_wrapper_chains=follow_wrapped)
File /home/torstein/anaconda3/lib/python3.7/inspect.py, line 2288, in _signature_from_callable
    skip_bound_arg=skip_bound_arg)
File /home/torstein/anaconda3/lib/python3.7/inspect.py, line 2112, in _signature_from_builtin
    raise ValueError("no signature found for builtin {!r}".format(func))

ValueError: no signature found for builtin <built-in function matching_blocks>

Is there any way to work around that? The __text_signature__ field is not writeable and providing a signature of the function to njit does not help.

[stuartarchibald]

Thanks for the report. Just wondering what it is you want to achieve? If matching_blocks is already a compiled cython function, Numba cannot do anything with it (apart from perhaps call it?!).

If you are new to Numba, perhaps take a look at the 5 minute guide first and/or click one of the Try Now buttons here http://numba.pydata.org/ to open up an interactive Numba demo in your browser, this one demonstrates the basics.

[tsoernes]

@stuartarchibald Yes, it is already a compiled cython function, but the code that's calling it is not, and that's the code I want to JIT!

[stuartarchibald]

@tsoernes you have a couple of options.

1. If the cython extension module is compatible, http://numba.pydata.org/numba-doc/latest/extending/high-level.html#importing-cython-functions
2. Use an object mode block to make the call http://numba.pydata.org/numba-doc/latest/user/withobjmode.html#the-objmode-context-manager

[tsoernes]

@stuartarchibald A great oversight of mine, but the function is a (wrapper to a?) compiled C function (https://github.com/ztane/python-Levenshtein) and not a Cython function.
Attempting to get the cython function address results in:

AttributeError: module 'Levenshtein' has no attribute '__pyx_capi__'

[stuartarchibald]

@tsoernes ok, then you can perhaps bind using ctypes: http://numba.pydata.org/numba-doc/dev/reference/pysupported.html#ctypes with a method like in http://numba.pydata.org/numba-doc/latest/extending/high-level.html#importing-cython-functions, but instead of the address coming from get_cython_function_address use the address from ctypes.CDLL and then finding the function by name ?

[tsoernes]

@stuartarchibald Okay. The function returns a Python list of tuples so I guess I'm out of luck since I cannot build a supported ctype.

[stuartarchibald]

@tsoernes ok, how about instead using an objmode block for the call? It will involve the JIT function going back into python to invoke the cython call but will let the code around the block be compiled into machine code and have the associated benefits. Docs: http://numba.pydata.org/numba-doc/latest/user/withobjmode.html#the-objmode-context-manager

[tsoernes]

@stuartarchibald Does it work when the return type is a list of tuples? I've tried this, without knowing quite how to annotate the return type.

from Levenshtein import matching_blocks, editops
import numba
from numba import typeof, cfunc

str_ty = typeof('x')
block_ty = typeof([('x', 1, 2)])

@numba.njit
def matching_blocks2(edit_operations, source_length, destination_length) -> str:
    # with numba.objmode(mb=block_ty):
    with numba.objmode(mb='list((unicode_type, int64, int64))'):
        mb = matching_blocks(edit_operations, source_length, destination_length)
    return mb

a, b = 'spam', 'park'
matching_blocks(editops(a, b), a, b)
matching_blocks2(editops(a, b), a, b)

[stuartarchibald]

@tsoernes what is being returned, something like block_ty, i.e. list of tuple of string and two ints?

[stuartarchibald]

Here's an example of creating a tuple in an objmode block and then using it in nopython mode:

from numba import njit, objmode
import numpy as np

@njit
def foo():
    with objmode(ret = 'Tuple((unicode_type, int64, int64))'):
        ret = ('foo', 1, 2)
    return ret + ('bar',)

print(foo())

For lists, I'd recommend avoiding reflected-lists, and use typed listed instead, example:

from numba import njit, objmode
from numba.typed import List
import numpy as np

@njit
def foo():
    with objmode(l = 'ListType(Tuple((unicode_type, int64, int64)))'):
        ret = ('foo', 1, 2)
        l = List()
        l.append(ret)
    return l

print(foo())

docs on why: http://numba.pydata.org/numba-doc/latest/reference/pysupported.html#list

[tsoernes]

from Levenshtein import matching_blocks, editops
import numba
from numba import typeof, cfunc
import numpy as np
from numba.typed import List

str_ty = typeof('x')
block_ty = typeof([('x', 1, 2)])

@numba.njit
def matching_blocks2(edit_operations, source_length, destination_length) -> str:
    with numba.objmode(mb='ListType(Tuple((int64, int64, int64)))'):
        x = matching_blocks(edit_operations, source_length, destination_length)
        mb = List()
        mb.extend(x)
        
    return mb

My mistake. The return type is always a List of tuples with 3 Ints. Still no luck with the above example:

Does not support list type inputs into with-context for arg 2
[1] During: resolving callee type: type(ObjModeLiftedWith(<function matching_blocks2 at 0x7f4f34579680>))
[2] During: typing of call at /home/torstein/code/fuzzywuzzy/fuzzywuzzy/test.py (16)


File "../../../fuzzywuzzy/fuzzywuzzy/test.py", line 16:
def matching_blocks2(edit_operations, source_length, destination_length) -> str:
    with numba.objmode(mb='ListType(Tuple((int64, int64, int64)))'):
    ^

[stuartarchibald]

Perhaps try this:

from Levenshtein import matching_blocks, editops
from numba import njit, objmode
import numpy as np
from numba import typeof, cfunc
import numpy as np
from numba.typed import List

str_ty = typeof('x')
block_ty = typeof([('x', 1, 2)])

@njit
def matching_blocks2(edit_operations, source_length, destination_length):
    with objmode(mb='ListType(Tuple((int64, int64, int64)))'):
        l_edit_operations = [_ for _ in edit_operations]
        x = matching_blocks(l_edit_operations, source_length,
                            destination_length)
        mb = List()
        mb.extend(x)

    return mb


a, b = 'spam', 'park'
print(matching_blocks2(tuple(editops(a, b)), a, b))

the objmode block does not support reflected list. As these seem like config options, tuple will suffice, and then make them into a list once in the objmode block as the cython function is likely hardcoded to take that.

[tsoernes]

Direct iteration is not supported for arrays with dimension > 1. Try using indexing instead.
Edit operations is a list of arbitrarily many (str, int, int) tuples.

[stuartarchibald]

Is the code above not working for you?

[tsoernes]

It works! I had a type in the code. Thanks for the help!

[stuartarchibald]

No problem, thanks for using Numba.