[Numpy] Update fallback.py

[barry-jin]

Description

Workaround for #19454 to make mxnet numpy operators fall back to the correct official numpy function.

Checklist

Essentials

• PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage
• Code is well-documented

Changes

• Add get_func to get official numpy function

Comments

• If this change is a backward incompatible change, why must this change be made.
• Interesting edge cases to note here

[mxnet-bot]

Hey @barry-jin , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

• To trigger all jobs: @mxnet-bot run ci [all]
• To trigger specific jobs: @mxnet-bot run ci [job1, job2]

---

CI supported jobs: [clang, miscellaneous, unix-cpu, website, centos-gpu, edge, unix-gpu, windows-cpu, centos-cpu, windows-gpu, sanity]

---

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

[leezu]

Please also add a test case that catches the bug

[barry-jin]

I have updated test_numpy_interoperability.py. But there will be segmentation fault when running test on operator atleast_1d (click toggle to see detailed test logs)

Test Logs

====================================================================== test session starts =======================================================================
platform darwin -- Python 3.8.5, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /Users/zhenghuj/GitHub/incubator-mxnet, configfile: pytest.ini
plugins: env-0.6.2, flaky-3.7.0
collected 4 items                                                                                                                                                

tests/python/unittest/test_numpy_interoperability.py .F..                                                                                                  [100%]

============================================================================ FAILURES ============================================================================
________________________________________________________________ test_np_array_function_protocol _________________________________________________________________

args = (), kwargs = {}

    @functools.wraps(func)
    def _run_with_array_func_proto(*args, **kwargs):
        if cur_np_ver >= np_1_17_ver:
            try:
>               func(*args, **kwargs)

python/mxnet/numpy_dispatch_protocol.py:55: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    @use_np
    @with_array_function_protocol
    @pytest.mark.serial
    def test_np_array_function_protocol():
>       check_interoperability(_NUMPY_ARRAY_FUNCTION_LIST)

tests/python/unittest/test_numpy_interoperability.py:3330: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

op_list = ['all', 'any', 'sometrue', 'argmin', 'argmax', 'around', ...]

    def check_interoperability(op_list):
        OpArgMngr.randomize_workloads()
        for name in op_list:
            if name in _TVM_OPS and not is_op_runnable():
                continue
            if name in ['shares_memory', 'may_share_memory', 'empty_like',
                        '__version__', 'dtype', '_NoValue']:  # skip list
                continue
            if name in ['delete']: # https://github.com/apache/incubator-mxnet/issues/18600
                continue
            if name in ['full_like', 'zeros_like', 'ones_like'] and \
                    StrictVersion(platform.python_version()) < StrictVersion('3.0.0'):
                continue
            default_tols = (1e-3, 1e-4)
            tols = {'linalg.tensorinv': (1e-2, 5e-3),
                    'linalg.solve':     (1e-3, 5e-2)}
            (rel_tol, abs_tol) = tols.get(name, default_tols)
            print('Dispatch test:', name)
            workloads = OpArgMngr.get_workloads(name)
            assert workloads is not None, 'Workloads for operator `{}` has not been ' \
                                          'added for checking interoperability with ' \
                                          'the official NumPy.'.format(name)
            for workload in workloads:
>               _check_interoperability_helper(name, rel_tol, abs_tol, *workload['args'], **workload['kwargs'])

tests/python/unittest/test_numpy_interoperability.py:3310: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

op_name = 'atleast_1d', rel_tol = 0.001, abs_tol = 0.0001
args = ([array([[1., 2.],
       [1., 2.]]), array([[1., 2.],
       [1., 2.]])], [array([[2., 3.],
       [2., 3.]]), array([[2., 3.],
       [2., 3.]])])
kwargs = {}, strs = ['atleast_1d'], onp_op = <function atleast_1d at 0x11a09fc10>, mxnp_op = <function atleast_1d at 0x1537e6dc0>

    def _check_interoperability_helper(op_name, rel_tol, abs_tol, *args, **kwargs):
        strs = op_name.split('.')
        if len(strs) == 1:
            onp_op = getattr(_np, op_name)
            mxnp_op = getattr(np, op_name)
        elif len(strs) == 2:
            onp_op = getattr(getattr(_np, strs[0]), strs[1])
            mxnp_op = getattr(getattr(np, strs[0]), strs[1])
        else:
            assert False
        if not is_op_runnable():
            return
>       out = mxnp_op(*args, **kwargs)

tests/python/unittest/test_numpy_interoperability.py:3266: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

arys = ([array([[1., 2.],
       [1., 2.]]), array([[1., 2.],
       [1., 2.]])], [array([[2., 3.],
       [2., 3.]]), array([[2., 3.],
       [2., 3.]])])

    @set_module('mxnet.numpy')
    def atleast_1d(*arys):
        """
        Convert inputs to arrays with at least one dimension.
    
        Scalar inputs are converted to 1-dimensional arrays, whilst higher-dimensional inputs are preserved.
    
        Parameters
        ----------
        arys1, arys2, ... : ndarray
            One or more input arrays.
    
        Returns
        -------
        ret : ndarray
            An array, or list of arrays, each with a.ndim >= 1. Copies are made only if necessary.
    
        See also
        --------
        atleast_2d, atleast_3d
    
        Examples
        --------
        >>> np.atleast_1d(1.0)
        array([1.])
        >>> x = np.arange(9.0).reshape(3,3)
        >>> np.atleast_1d(x)
        array([[0., 1., 2.],
               [3., 4., 5.],
               [6., 7., 8.]])
        >>> np.atleast_1d(np.array(1), np.array([3, 4]))
        [array([1.]), array([3., 4.])]
        """
>       return _mx_nd_np.atleast_1d(*arys)

python/mxnet/numpy/multiarray.py:11385: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

arys = ([array([[1., 2.],
       [1., 2.]]), array([[1., 2.],
       [1., 2.]])], [array([[2., 3.],
       [2., 3.]]), array([[2., 3.],
       [2., 3.]])])

    @set_module('mxnet.ndarray.numpy')
    def atleast_1d(*arys):
        """
        Convert inputs to arrays with at least one dimension.
    
        Scalar inputs are converted to 1-dimensional arrays, whilst higher-dimensional inputs are preserved.
    
        Parameters
        ----------
        arys1, arys2, ... : ndarray
            One or more input arrays.
    
        Returns
        -------
        ret : ndarray
            An array, or list of arrays, each with a.ndim >= 1. Copies are made only if necessary.
    
        See also
        --------
        atleast_2d, atleast_3d
    
        Examples
        --------
        >>> np.atleast_1d(1.0)
        array([1.])
        >>> x = np.arange(9.0).reshape(3,3)
        >>> np.atleast_1d(x)
        array([[0., 1., 2.],
               [3., 4., 5.],
               [6., 7., 8.]])
        >>> np.atleast_1d(np.array(1), np.array([3, 4]))
        [array([1.]), array([3., 4.])]
        """
        if len(arys) == 1:
            return _api_internal.atleast_1d(*arys)[0]
>       return list(_api_internal.atleast_1d(*arys))

python/mxnet/ndarray/numpy/_op.py:8901: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???

mxnet/_ffi/_cython/./function.pxi:188: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???

mxnet/_ffi/_cython/./function.pxi:120: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???

mxnet/_ffi/_cython/./function.pxi:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???

mxnet/_ffi/_cython/./function.pxi:36: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???

mxnet/_ffi/_cython/./convert.pxi:75: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???

mxnet/_ffi/_cython/./convert.pxi:63: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   TypeError: Don't know how to convert type <class 'mxnet.numpy.ndarray'>

mxnet/_ffi/_cython/./convert.pxi:81: TypeError

During handling of the above exception, another exception occurred:

args = (), kwargs = {}

    @functools.wraps(func)
    def _with_np_shape(*args, **kwargs):
        with np_shape(active=True):
>           return func(*args, **kwargs)

python/mxnet/util.py:299: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
python/mxnet/util.py:480: in _with_np_array
    return func(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (), kwargs = {}

    @functools.wraps(func)
    def _run_with_array_func_proto(*args, **kwargs):
        if cur_np_ver >= np_1_17_ver:
            try:
                func(*args, **kwargs)
            except Exception as e:
>               raise RuntimeError('Running function {} with NumPy array function protocol failed'
                                   ' with exception {}'
                                   .format(func.__name__, str(e)))
E               RuntimeError: Running function test_np_array_function_protocol with NumPy array function protocol failed with exception Don't know how to convert type <class 'mxnet.numpy.ndarray'>

python/mxnet/numpy_dispatch_protocol.py:57: RuntimeError

Segmentation Fault can be reproduced as follows

>>> from mxnet import np
>>> import numpy as _np
>>> x1 = _np.arange(4.0).reshape(2,2)
>>> p1 = (x1, x1, x1, x1)
>>> x2 = np.arange(4.0).reshape(2,2)
>>> p2 = (x2, x2, x2, x2)
>>> _np.atleast_1d(p1)
array([[[0., 1.],
        [2., 3.]],

       [[0., 1.],
        [2., 3.]],

       [[0., 1.],
        [2., 3.]],

       [[0., 1.],
        [2., 3.]]])
>>> np.atleast_1d(p2)

Fatal Error: Segmentation fault: 11
Stack trace:
Segmentation fault: 11

[szha]

you can use functools.wraps