A prototype of first-class functions [DON'T MERGE]

[pearu]

Tackles the issue #3405 but closes once support for njit decorated functions is implemented.

The middle-level description of the first-class function model is as follows:

1. A function, as an instance of the first-class type, is represented by its address, signature, and a pointer to a parent python object. The address enables accessing the compiled implementations of the function as well as to enable using function instances as items within njitted functions. The signature enables constructing the LLVM IR level call to the compiled functions. The parent object is used as a reference to the high-level implementation (python function, for instance) that enables compiling the function for a different set of argument types in just-in-time manner as well as it is used as return value of the first-class function.
2. Wrapper address protocol: When a first-class function is passed into a njitted function, it needs to be "unboxed". The unboxing requires a method for acquiring the address of the compiled function. For instance, if the input argument is a numba.cfunc(<signature>) decorated function (of type CFunc), then the address is readily available via its _wrapper_address attribute. This PR introduces so-called wrapper address protocol (see https://github.com/numba/numba/pull/4967/files#diff-275bdd7060d042bc811293de00e7878dR115-R143 for the definition and support) that enables interpreting arbitrary python objects as first-class functions when passed to a njitted function as an argument. For instance, to retrieve a function address from a python object such as CFunc, the function numba.function._get_wrapper_address(obj, signature) is called from the unboxing procedure. The implementation of the _get_wrapper_address for the CFunc objects would be as simple as (assuming that CFunc instance is compiled against a correct signature):

def _get_wrapper_address(obj, signature):
    return obj._wrapper_address

3. Extensibility: The wrapper address protocol model allows for extending first-class function model to arbitrary objects (not just for cfunc decorated functions). For instance, one can implement caching of compiled functions as an independent method to numba jit caching model, or one can call arbitrary library functions from njitted functions (as long as one is able to determine the pointer values of the library functions, one can use ctypes for that). See https://github.com/numba/numba/pull/4967/files#diff-75b50def42de3256e721ce2c6e79c47dR282-R322 for an example of how to call a libc function time from a njitted function.

Current status:

• [next PR] Pass functions as arguments to njitted functions:
  • cfunc decorated functions
  • njit decorated functions
  • [next PR] pure Python functions
  • [next PR] ctypes functions
• [next PR] Use functions as items within njitted functions:
  Both constant (available via namespace lookup) and passed in functions (available as arguments of njitted functions)
  • cfunc decorated functions
  • njit decorated functions
  • [next PR] pure Python functions
  • [next PR] ctypes functions
• [next PR] Use functions as return values from njitted functions
  Both constant and passed in functions
  • cfunc decorated functions
  • njit decorated functions
  • [next PR] pure Python functions
  • [next PR] ctypes functions
• [next PR] Overload function calls within njitted functions
  Both constant and passed in functions
  • [next PR] cfunc decorated functions
  • njit decorated functions
• [next PR] Constuct function signatures
  • int64 arguments
  • other argument types
  • [next PR] tests, partially blocked by Signature vs FunctionType issue
• Unit-tests
• [next PR] Documentaton

Example:

from numba import cfunc, njit, int64

@cfunc(int64(int64))
def a(i):
    return i + 1

@cfunc(int64(int64))
def b(i):
    return i + 2

@njit
def foo(f, g):
    i = f(2)
    seq = (f, g)
    for fun in seq:
        i += fun(i)
    return i

a_ = a._pyfunc
b_ = b._pyfunc
assert foo(a, b) == a_(2) + a_(a_(2)) + b_(a_(2) + a_(a_(2)))

[stuartarchibald]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[pearu]

The current PR implements first-class function type support for cfunc decorated functions.
Some features work also for njit decorated functions but the full support will be implemented in a separate PR.
@sklam , please review.

[pearu]

@sklam @stuartarchibald I have addressed all Siu's notes:

1. replace test_all method with explicit test methods.
2. eliminate numbatype function as not needed here.
3. signatures are expected to be given as Signature instances
   The PR is ready for review.

[stuartarchibald]

Thanks @pearu I've marked it as such. I'll hopefully be able to look at it in the next day or so.

[stuartarchibald]

Thanks for the PR, great to see this feature implemented. I've given this a review solely from the standpoint of looking at the code, please find suggestions/comments inline. The next phase will be to manually stress test the implementation. Thanks again!

[stuartarchibald]

Thanks for the PR, great to see this feature implemented. I've given this a review solely from the standpoint of looking at the code, please find suggestions/comments inline. The next phase will be to manually stress test the implementation. Thanks again!

[stuartarchibald]

Before this branch is merged, this feature needs to be well documented and some examples need writing as users are sure to ask. Thanks.

[sklam]

With this PR, i think CFunc needs to be callable, otherwise we break the contract that code still works without the decorator.

Sample code that fails without `@njit`

import numpy as np
from numba import njit, cfunc, intp, float64, function

@cfunc("intp(intp, float64)")
def foo1(x, y):
    return x + y


@cfunc("intp(intp, float64)")
def foo2(x, y):
    return x - y

# @njit   # doesn't work without @njit
def bar(fx, fy, i):
    a = np.array([10], dtype=np.intp)
    b = np.array([12], dtype=np.float64)
    if i == 0:
        f = fx
    elif i == 1:
        f = fy
    else:
        return
    return f(a[0], b[0])    # TypeError: 'CFunc' object is not callable

r = bar(foo1, foo2, 0)

print(r)

[pearu]

@sklam @stuartarchibald, please review:

• rebased to master,
• removed signature from __wrapper_address__ method.

[pearu]

Note that PR #5287 adds the jit-decorated functions support on the top of this PR.
So, if preferred, this PR can be shelved and we can proceed working with PR #5287 only.

[sklam]

I have some suggestions at #5307. 8156a19 would simplify the logic for .get_call_type()'s casting rule.

[pearu]

I have some suggestions at #5307. 8156a19 would simplify the logic for .get_call_type()'s casting rule.

Thanks, applied the suggestion to #5287 .

[stuartarchibald]

Note to self. Check docs. @sklam approves code.

[stuartarchibald]

If #5287 is replacing, can this be closed now?

[pearu]

yes, this can be closed.