Errors with various reduce calls

[ivirshup]

After asking on gitter about whether my usage of reduce was supported, @stuartarchibald asked me to open this issue here.

The basic issue is that I'd like to use functools.reduce, but was running into some errors. Here are three implementations I've tried and the errors they throw:

from numba import njit
from functools import reduce
from operator import add
import numpy as np

a = np.arange(100)

@njit
def sumnumba1(it):
    return reduce(add, it, 0)

sumnumba1(a)

Traceback (ValueError, invalid reduction function)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-06d54deac5b5> in <module>
      3     return reduce(add, it, 0)
      4 
----> 5 sumnumba1(a)

/usr/local/lib/python3.7/site-packages/numba/dispatcher.py in _compile_for_args(self, *args, **kws)
    393                     e.patch_message(''.join(e.args) + help_msg)
    394             # ignore the FULL_TRACEBACKS config, this needs reporting!
--> 395             raise e
    396 
    397     def inspect_llvm(self, signature=None):

/usr/local/lib/python3.7/site-packages/numba/dispatcher.py in _compile_for_args(self, *args, **kws)
    350                 argtypes.append(self.typeof_pyval(a))
    351         try:
--> 352             return self.compile(tuple(argtypes))
    353         except errors.TypingError as e:
    354             # Intercept typing error that may be due to an argument

/usr/local/lib/python3.7/site-packages/numba/compiler_lock.py in _acquire_compile_lock(*args, **kwargs)
     30         def _acquire_compile_lock(*args, **kwargs):
     31             with self:
---> 32                 return func(*args, **kwargs)
     33         return _acquire_compile_lock
     34 

/usr/local/lib/python3.7/site-packages/numba/dispatcher.py in compile(self, sig)
    691 
    692             self._cache_misses[sig] += 1
--> 693             cres = self._compiler.compile(args, return_type)
    694             self.add_overload(cres)
    695             self._cache.save_overload(sig, cres)

/usr/local/lib/python3.7/site-packages/numba/dispatcher.py in compile(self, args, return_type)
     74 
     75     def compile(self, args, return_type):
---> 76         status, retval = self._compile_cached(args, return_type)
     77         if status:
     78             return retval

/usr/local/lib/python3.7/site-packages/numba/dispatcher.py in _compile_cached(self, args, return_type)
     88 
     89         try:
---> 90             retval = self._compile_core(args, return_type)
     91         except errors.TypingError as e:
     92             self._failed_cache[key] = e

/usr/local/lib/python3.7/site-packages/numba/dispatcher.py in _compile_core(self, args, return_type)
    106                                       args=args, return_type=return_type,
    107                                       flags=flags, locals=self.locals,
--> 108                                       pipeline_class=self.pipeline_class)
    109         # Check typing error if object mode is used
    110         if cres.typing_error is not None and not flags.enable_pyobject:

/usr/local/lib/python3.7/site-packages/numba/compiler.py in compile_extra(typingctx, targetctx, func, args, return_type, flags, locals, library, pipeline_class)
    970     pipeline = pipeline_class(typingctx, targetctx, library,
    971                               args, return_type, flags, locals)
--> 972     return pipeline.compile_extra(func)
    973 
    974 

/usr/local/lib/python3.7/site-packages/numba/compiler.py in compile_extra(self, func)
    388         self.lifted = ()
    389         self.lifted_from = None
--> 390         return self._compile_bytecode()
    391 
    392     def compile_ir(self, func_ir, lifted=(), lifted_from=None):

/usr/local/lib/python3.7/site-packages/numba/compiler.py in _compile_bytecode(self)
    901         """
    902         assert self.func_ir is None
--> 903         return self._compile_core()
    904 
    905     def _compile_ir(self):

/usr/local/lib/python3.7/site-packages/numba/compiler.py in _compile_core(self)
    888         self.define_pipelines(pm)
    889         pm.finalize()
--> 890         res = pm.run(self.status)
    891         if res is not None:
    892             # Early pipeline completion

/usr/local/lib/python3.7/site-packages/numba/compiler_lock.py in _acquire_compile_lock(*args, **kwargs)
     30         def _acquire_compile_lock(*args, **kwargs):
     31             with self:
---> 32                 return func(*args, **kwargs)
     33         return _acquire_compile_lock
     34 

/usr/local/lib/python3.7/site-packages/numba/compiler.py in run(self, status)
    264                     # No more fallback pipelines?
    265                     if is_final_pipeline:
--> 266                         raise patched_exception
    267                     # Go to next fallback pipeline
    268                     else:

/usr/local/lib/python3.7/site-packages/numba/compiler.py in run(self, status)
    255                 try:
    256                     event("-- %s" % stage_name)
--> 257                     stage()
    258                 except _EarlyPipelineCompletion as e:
    259                     return e.result

/usr/local/lib/python3.7/site-packages/numba/compiler.py in stage_inline_pass(self)
    619                                             self.parfor_diagnostics.replaced_fns,
    620                                             typed_pass)
--> 621         inline_pass.run()
    622         # Remove all Dels, and re-run postproc
    623         post_proc = postproc.PostProcessor(self.func_ir)

/usr/local/lib/python3.7/site-packages/numba/inline_closurecall.py in run(self)
     71 
     72                         if guard(self._inline_reduction,
---> 73                                  work_list, block, i, expr, call_name):
     74                             modified = True
     75                             break # because block structure changed

/usr/local/lib/python3.7/site-packages/numba/ir_utils.py in guard(func, *args, **kwargs)
   1376     """
   1377     try:
-> 1378         return func(*args, **kwargs)
   1379     except GuardException:
   1380         return None

/usr/local/lib/python3.7/site-packages/numba/inline_closurecall.py in _inline_reduction(self, work_list, block, i, expr, call_name)
    124                             "three arguments including initial "
    125                             "value required")
--> 126         check_reduce_func(self.func_ir, expr.args[0])
    127         def reduce_func(f, A, v):
    128             s = v

/usr/local/lib/python3.7/site-packages/numba/inline_closurecall.py in check_reduce_func(func_ir, func_var)
    223     if not (hasattr(reduce_func, 'code')
    224             or hasattr(reduce_func, '__code__')):
--> 225         raise ValueError("Invalid reduction function")
    226     f_code = (reduce_func.code if hasattr(reduce_func, 'code')
    227                                     else reduce_func.__code__)

ValueError: Failed in nopython mode pipeline (step: inline calls to locally defined closures)
Invalid reduction function

@njit
def sumnumba2(it):
    return reduce(np.add, it, 0)

sumnumba2(a)

Traceback (KeyError)

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-3-7adadecb5d72> in <module>
      3     return reduce(np.add, it, 0)
      4 
----> 5 sumnumba2(a)

/usr/local/lib/python3.7/site-packages/numba/dispatcher.py in _compile_for_args(self, *args, **kws)
    393                     e.patch_message(''.join(e.args) + help_msg)
    394             # ignore the FULL_TRACEBACKS config, this needs reporting!
--> 395             raise e
    396 
    397     def inspect_llvm(self, signature=None):

/usr/local/lib/python3.7/site-packages/numba/dispatcher.py in _compile_for_args(self, *args, **kws)
    350                 argtypes.append(self.typeof_pyval(a))
    351         try:
--> 352             return self.compile(tuple(argtypes))
    353         except errors.TypingError as e:
    354             # Intercept typing error that may be due to an argument

/usr/local/lib/python3.7/site-packages/numba/compiler_lock.py in _acquire_compile_lock(*args, **kwargs)
     30         def _acquire_compile_lock(*args, **kwargs):
     31             with self:
---> 32                 return func(*args, **kwargs)
     33         return _acquire_compile_lock
     34 

/usr/local/lib/python3.7/site-packages/numba/dispatcher.py in compile(self, sig)
    691 
    692             self._cache_misses[sig] += 1
--> 693             cres = self._compiler.compile(args, return_type)
    694             self.add_overload(cres)
    695             self._cache.save_overload(sig, cres)

/usr/local/lib/python3.7/site-packages/numba/dispatcher.py in compile(self, args, return_type)
     74 
     75     def compile(self, args, return_type):
---> 76         status, retval = self._compile_cached(args, return_type)
     77         if status:
     78             return retval

/usr/local/lib/python3.7/site-packages/numba/dispatcher.py in _compile_cached(self, args, return_type)
     88 
     89         try:
---> 90             retval = self._compile_core(args, return_type)
     91         except errors.TypingError as e:
     92             self._failed_cache[key] = e

/usr/local/lib/python3.7/site-packages/numba/dispatcher.py in _compile_core(self, args, return_type)
    106                                       args=args, return_type=return_type,
    107                                       flags=flags, locals=self.locals,
--> 108                                       pipeline_class=self.pipeline_class)
    109         # Check typing error if object mode is used
    110         if cres.typing_error is not None and not flags.enable_pyobject:

/usr/local/lib/python3.7/site-packages/numba/compiler.py in compile_extra(typingctx, targetctx, func, args, return_type, flags, locals, library, pipeline_class)
    970     pipeline = pipeline_class(typingctx, targetctx, library,
    971                               args, return_type, flags, locals)
--> 972     return pipeline.compile_extra(func)
    973 
    974 

/usr/local/lib/python3.7/site-packages/numba/compiler.py in compile_extra(self, func)
    388         self.lifted = ()
    389         self.lifted_from = None
--> 390         return self._compile_bytecode()
    391 
    392     def compile_ir(self, func_ir, lifted=(), lifted_from=None):

/usr/local/lib/python3.7/site-packages/numba/compiler.py in _compile_bytecode(self)
    901         """
    902         assert self.func_ir is None
--> 903         return self._compile_core()
    904 
    905     def _compile_ir(self):

/usr/local/lib/python3.7/site-packages/numba/compiler.py in _compile_core(self)
    888         self.define_pipelines(pm)
    889         pm.finalize()
--> 890         res = pm.run(self.status)
    891         if res is not None:
    892             # Early pipeline completion

/usr/local/lib/python3.7/site-packages/numba/compiler_lock.py in _acquire_compile_lock(*args, **kwargs)
     30         def _acquire_compile_lock(*args, **kwargs):
     31             with self:
---> 32                 return func(*args, **kwargs)
     33         return _acquire_compile_lock
     34 

/usr/local/lib/python3.7/site-packages/numba/compiler.py in run(self, status)
    264                     # No more fallback pipelines?
    265                     if is_final_pipeline:
--> 266                         raise patched_exception
    267                     # Go to next fallback pipeline
    268                     else:

/usr/local/lib/python3.7/site-packages/numba/compiler.py in run(self, status)
    255                 try:
    256                     event("-- %s" % stage_name)
--> 257                     stage()
    258                 except _EarlyPipelineCompletion as e:
    259                     return e.result

/usr/local/lib/python3.7/site-packages/numba/compiler.py in stage_inline_pass(self)
    619                                             self.parfor_diagnostics.replaced_fns,
    620                                             typed_pass)
--> 621         inline_pass.run()
    622         # Remove all Dels, and re-run postproc
    623         post_proc = postproc.PostProcessor(self.func_ir)

/usr/local/lib/python3.7/site-packages/numba/inline_closurecall.py in run(self)
     71 
     72                         if guard(self._inline_reduction,
---> 73                                  work_list, block, i, expr, call_name):
     74                             modified = True
     75                             break # because block structure changed

/usr/local/lib/python3.7/site-packages/numba/ir_utils.py in guard(func, *args, **kwargs)
   1376     """
   1377     try:
-> 1378         return func(*args, **kwargs)
   1379     except GuardException:
   1380         return None

/usr/local/lib/python3.7/site-packages/numba/inline_closurecall.py in _inline_reduction(self, work_list, block, i, expr, call_name)
    124                             "three arguments including initial "
    125                             "value required")
--> 126         check_reduce_func(self.func_ir, expr.args[0])
    127         def reduce_func(f, A, v):
    128             s = v

/usr/local/lib/python3.7/site-packages/numba/inline_closurecall.py in check_reduce_func(func_ir, func_var)
    221         raise ValueError("Reduce function cannot be found for njit \
    222                             analysis")
--> 223     if not (hasattr(reduce_func, 'code')
    224             or hasattr(reduce_func, '__code__')):
    225         raise ValueError("Invalid reduction function")

/usr/local/lib/python3.7/site-packages/numba/ir.py in __getattr__(self, name)
    305         if name.startswith('_'):
    306             return Inst.__getattr__(self, name)
--> 307         return self._kws[name]
    308 
    309     def __setattr__(self, name, value):

KeyError: "Failed in nopython mode pipeline (step: inline calls to locally defined closures)\n'code'"

@njit
def sumnumba3(it):
    return np.add.reduce(it)

sumnumba3(a)

Traceback (TypingError)

---------------------------------------------------------------------------
TypingError                               Traceback (most recent call last)
<ipython-input-4-47423639c0e1> in <module>
      3     return np.add.reduce(it)
      4 
----> 5 sumnumba3(a)

/usr/local/lib/python3.7/site-packages/numba/dispatcher.py in _compile_for_args(self, *args, **kws)
    374                 e.patch_message(msg)
    375 
--> 376             error_rewrite(e, 'typing')
    377         except errors.UnsupportedError as e:
    378             # Something unsupported is present in the user code, add help info

/usr/local/lib/python3.7/site-packages/numba/dispatcher.py in error_rewrite(e, issue_type)
    341                 raise e
    342             else:
--> 343                 reraise(type(e), e, None)
    344 
    345         argtypes = []

/usr/local/lib/python3.7/site-packages/numba/six.py in reraise(tp, value, tb)
    656             value = tp()
    657         if value.__traceback__ is not tb:
--> 658             raise value.with_traceback(tb)
    659         raise value
    660 

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Unknown attribute 'reduce' of type Function(<ufunc 'add'>)

File "<ipython-input-4-47423639c0e1>", line 3:
def sumnumba3(it):
    return np.add.reduce(it)
    ^

[1] During: typing of get attribute at <ipython-input-4-47423639c0e1> (3)

File "<ipython-input-4-47423639c0e1>", line 3:
def sumnumba3(it):
    return np.add.reduce(it)
    ^

This is not usually a problem with Numba itself but instead often caused by
the use of unsupported features or an issue in resolving types.

To see Python/NumPy features supported by the latest release of Numba visit:
http://numba.pydata.org/numba-doc/latest/reference/pysupported.html
and
http://numba.pydata.org/numba-doc/latest/reference/numpysupported.html

For more information about typing errors and how to debug them visit:
http://numba.pydata.org/numba-doc/latest/user/troubleshoot.html#my-code-doesn-t-compile

If you think your code should work with Numba, please report the error message
and traceback, along with a minimal reproducer at:
https://github.com/numba/numba/issues/new

As pointed out on gitter, the first two errors seem to be caused by numba expecting the function passed to reduce was defined locally.

I think the third example has more to do with the extent of ufunc support in numba, and might be a separate issue.

I ran these examples with numba v0.45.1 and numpy v1.17.1.

[stuartarchibald]

Thanks for the report. The issue for the first two examples is that the reduce impl in Numba expects the func argument to be a closure. The last is that the np.add ufunc impl doesn't have reduce supported.

[ivirshup]

Thanks for pointing me here!

About the third example, should numpy and numba ufuncs in general have a .reduce method that can be compiled?

[stuartarchibald]

Thanks for pointing me here!

No problem.

About the third example, should numpy and numba ufuncs in general have a .reduce method that can be compiled?

NumPy definitely has reduce on all it's ufuncs.

In [13]: x = np.arange(1, 6)                                                                                                    

In [14]: np.multiply.reduce(x)                                                                                                  
Out[14]: 120

In [15]: np.add.reduce(x)                                                                                                       
Out[15]: 15

Numba @vectorize decorated functions inherit the .reduce ability:

from numba import njit, vectorize
import numpy as np

@vectorize(['int64(int64, int64)'])
def _my_add(a, b):
    return np.add(a, b)

a = np.arange(1, 6)
print(_my_add.reduce(a))

However, in compiled code in general the .reduce() method is not supported.