Internal error when using CFFI .lib directly

[astrojuanlu]

Apparently numba.njit can infer the type of the CFFI function if I extract it first to a variable, but fails if I acces mod.lib.func directly. Example:

# hyper/hyper.py
from numba import cffi_support
import _hyper
cffi_support.register_module(_hyper)

# numba_cffi.py
from hyper.hyper import _hyper
from numba import njit


c_hyp = _hyper.lib.hyp2f1x

@njit
def cffi_test_ok(x):
    return c_hyp(1, 1, 1, x)

@njit
def cffi_test_fail(x):
    return _hyper.lib.hyp2f1x(1, 1, 1, x)


if __name__ == '__main__':
    print(cffi_test_ok(1))
    print(cffi_test_fail(1))

$ python numba_cffi.py 
1.0
Traceback (most recent call last):
  File "numba_cffi.py", line 19, in <module>
    print(cffi_test_fail(1))
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/dispatcher.py", line 171, in _compile_for_args
    return self.compile(sig)
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/dispatcher.py", line 349, in compile
    flags=flags, locals=self.locals)
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/compiler.py", line 665, in compile_extra
    return pipeline.compile_extra(func)
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/compiler.py", line 366, in compile_extra
    return self.compile_bytecode(bc, func_attr=self.func_attr)
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/compiler.py", line 375, in compile_bytecode
    return self._compile_bytecode()
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/compiler.py", line 652, in _compile_bytecode
    return pm.run(self.status)
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/compiler.py", line 251, in run
    raise patched_exception
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/compiler.py", line 243, in run
    res = stage()
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/compiler.py", line 463, in stage_nopython_frontend
    self.locals)
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/compiler.py", line 780, in type_inference_stage
    infer.propagate()
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/typeinfer.py", line 565, in propagate
    raise errors[0]
numba.errors.TypingError: Failed at nopython (nopython frontend)
Internal error at resolving type of attribute "lib" of "$0.1":
--%<-----------------------------------------------------------------
Traceback (most recent call last):
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/typeinfer.py", line 111, in propagate
    constraint(typeinfer)
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/typeinfer.py", line 352, in __call__
    attrty = typeinfer.context.resolve_getattr(ty, self.attr)
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/typing/context.py", line 149, in resolve_getattr
    attrty = self.resolve_module_constants(typ, attr)
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/typing/context.py", line 195, in resolve_module_constants
    ty = self.resolve_value_type(attrval)
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/typing/context.py", line 214, in resolve_value_type
    tp = typeof(val, Purpose.constant)
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/typing/typeof.py", line 29, in typeof
    return typeof_impl(val, c)
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/functools.py", line 743, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
AttributeError: cffi library '_hyper' has no function, constant or global variable named '__class__'
--%<-----------------------------------------------------------------

File "numba_cffi.py", line 14

[pitrou]

cffi libraries don't have a __class__? That issue should be reported to the cffi project.

[astrojuanlu]

Fair enough. conda cffi is a bit behind though, so I'll try with the latest version and see what happens.

[astrojuanlu]

Confirmed that a release was fixed for cffi 1.5.0 https://bitbucket.org/cffi/cffi/issues/236/compiledlib-breaks I guess you folks have an excuse to update the package :)

The error persists in a different way though:

1.0
Traceback (most recent call last):
  File "numba_cffi.py", line 19, in <module>
    print(cffi_test_fail(1))
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/dispatcher.py", line 171, in _compile_for_args
    return self.compile(sig)
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/dispatcher.py", line 349, in compile
    flags=flags, locals=self.locals)
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/compiler.py", line 665, in compile_extra
    return pipeline.compile_extra(func)
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/compiler.py", line 366, in compile_extra
    return self.compile_bytecode(bc, func_attr=self.func_attr)
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/compiler.py", line 375, in compile_bytecode
    return self._compile_bytecode()
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/compiler.py", line 652, in _compile_bytecode
    return pm.run(self.status)
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/compiler.py", line 251, in run
    raise patched_exception
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/compiler.py", line 243, in run
    res = stage()
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/compiler.py", line 463, in stage_nopython_frontend
    self.locals)
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/compiler.py", line 780, in type_inference_stage
    infer.propagate()
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/typeinfer.py", line 565, in propagate
    raise errors[0]
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/typeinfer.py", line 111, in propagate
    constraint(typeinfer)
  File "/home/juanlu/.miniconda3/envs/hyper35/lib/python3.5/site-packages/numba/typeinfer.py", line 354, in __call__
    raise UntypedAttributeError(ty, self.attr, loc=self.inst.loc)
numba.errors.UntypedAttributeError: Failed at nopython (nopython frontend)
Unknown attribute "lib" of type Module(<module '_hyper' from '/home/juanlu/Development/Python/Pybonacci/cffi_test/src/_hyper.cpython-35m-x86_64-linux-gnu.so'>)
File "numba_cffi.py", line 14

[seibert]

Is this final issue a duplicate of #1849?

[pitrou]

I don't think so, since that's not the same error.

[stuartarchibald]

I've had a look at this and can replicate with a trivial cffi module that accesses cos() from libm.
The problem seems to be that the lib attribute on the cffi module requires type resolution when called directly, this fails. The lib attribute is of type <class 'CompiledLib'> which is an internal representation in cffi. If you hack at the resolver to force it to resolve the type of lib the attribute e.g. cos() on lib then refuses to resolve etc.