You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have included a self contained code sample to reproduce the problem.
i.e. it's possible to run as 'python bug.py'.
I have some code, which has to call a list of functions in a loop. I know that Numba doens't handle heterogeneous lists, so the recommended approach is literal_unroll. The function I need to call, calls an external c-function. For this I am using ctypes to load the function from the library. The problem I experience occurs during compilation where numba throws an error: ctypes objects containing pointers cannot be pickled.
Here's an example of a code where this occurs:
import ctypes
from ctypes import cdll
from ctypes.util import find_library
from numba import literal_unroll, njit, cfunc, types
from numba.core.extending import intrinsic
import numpy as np
@intrinsic
def address_as_void_pointer(typingctx, src):
""" returns a void pointer from a given memory address """
from numba.core import types, cgutils
sig = types.voidptr(src)
def codegen(context, builder, sig, args):
ptr = builder.inttoptr(args[0], cgutils.voidptr_t)
return ptr
return sig, codegen
dll = cdll.LoadLibrary(find_library('C'))
printf_ptr = getattr(dll, 'printf')
printf_ptr.argtypes = [ctypes.c_void_p]
printf_ptr.restype = None
sig = types.void(types.int64)
@cfunc(sig)
def printf_cfunc1(string_ptr):
print("this is 1 cfunc")
printf_ptr(address_as_void_pointer(string_ptr))
@cfunc(sig)
def printf_cfunc2(string_ptr):
print("this is 2 cfunc")
printf_ptr(address_as_void_pointer(string_ptr))
@njit()
def printf_njit_1(string_addr):
print("this is 1 njit")
printf_ptr(address_as_void_pointer(string_addr))
@njit()
def printf_njit_2(string_addr):
print("this is 2 njit")
printf_ptr(address_as_void_pointer(string_addr))
s = "hello world!\n".encode('utf-8')
s_ptr = np.array([s]).ctypes.data_as(ctypes.c_void_p)
s_ = s_ptr.value
@njit()
def printf_litunroll(string_addr, printf_list_):
for printf_ in literal_unroll(printf_list_):
printf_(string_addr)
printf_litunroll(s_, (printf_cfunc1, printf_cfunc2))
printf_litunroll(s_, (printf_njit_1, printf_njit_2))
Calling this function prints out:
this is 1 cfunc
hello world!
this is 2 cfunc
hello world!
...
ValueError: Failed in nopython mode pipeline (step: handles literal_unroll)
Failed in literal_unroll_subpipeline mode pipeline (step: performs mixed container unroll)
ctypes objects containing pointers cannot be pickled
So it calls the @cfunc decorated function ok, but not the njit decorated one. This lead me to try out adding the signature for the njitted function as well, and lo' and behold, the code runs if I add @njit(sig) as decorator for the njitted functions:
this is 1 cfunc
hello world!
this is 2 cfunc
hello world!
this is 1 njit
hello world!
this is 2 njit
hello world!
I cannot find anywhere in the documentation that says this is normal behavior. If anything the error code is wrong, because clearly we are able to call a ctypes function containing a pointer inside the literal_unroll.
The text was updated successfully, but these errors were encountered:
Hi @johnhelt, Thank you for reporting the issue. I can confirm this behaviour locally and it does seem like inconsistent behaviour that needs addressing. Marking this as a bug for now.
Reporting a bug
visible in the release notes
(https://numba.readthedocs.io/en/stable/release-notes-overview.html).
i.e. it's possible to run as 'python bug.py'.
I have some code, which has to call a list of functions in a loop. I know that Numba doens't handle heterogeneous lists, so the recommended approach is literal_unroll. The function I need to call, calls an external c-function. For this I am using ctypes to load the function from the library. The problem I experience occurs during compilation where numba throws an error:
ctypes objects containing pointers cannot be pickled
.Here's an example of a code where this occurs:
Calling this function prints out:
So it calls the
@cfunc
decorated function ok, but not thenjit
decorated one. This lead me to try out adding the signature for the njitted function as well, and lo' and behold, the code runs if I add@njit(sig)
as decorator for the njitted functions:I cannot find anywhere in the documentation that says this is normal behavior. If anything the error code is wrong, because clearly we are able to call a ctypes function containing a pointer inside the literal_unroll.
The text was updated successfully, but these errors were encountered: