Testing function signatures

[asmeurer]

For example, for https://github.com/data-apis/array-api/blob/master/spec/API_specification/elementwise_functions.md. The spec specifies:

• Positional parameters must be positional-only parameters. Positional-only parameters have no externally-usable name. When a function accepting positional-only parameters is called, positional arguments are mapped to these parameters based solely on their order.
• Optional parameters must be keyword-only arguments.

It also specifies a signature for each function.

However, I don't know how to test these things. inspect.signature doesn't work for NumPy ufuncs, and presumably won't work for many other objects implemented in C. Such things could probably make themselves work with it by defining __signature__, but do we want to require this?

Otherwise, I don't think positional-only can be tested. I can probably test other aspect of the signature like the number of positional arguments by checking that the wrong number of arguments leads to a TypeError.

(also as an aside, out is not keyword-only for NumPy functions)

[asmeurer]

Also I guess a function could be defined with *args, **kwargs but still implement the given signature. So just checking against inspect.signature isn't sufficient even for a pure Python function. Plus positional-only is only 3.8+ so anything that supports 3.7 would have to implement positional-only with *args.

[rgommers]

Good questions. I would indeed not want to require __signature__, doesn't seem essential and can be quite a bit of work.

Looks like you can test it though, reliably gives a ValueError if you try to use a positional-only argument with a name:

In [36]: np.sin(x)                                                             
Out[36]: array([0., 0., 0.])

In [37]: np.sin(x=x)                                                           
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-37-454dedab3814> in <module>
----> 1 np.sin(x=x)

ValueError: invalid number of arguments

In [38]: np.sin(wrongname=x)                                                   
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-38-59275d4f4c63> in <module>
----> 1 np.sin(wrongname=x)

ValueError: invalid number of arguments

(also as an aside, out is not keyword-only for NumPy functions)

out will be removed everywhere (on account of being a bad API and not universally implementable), so that solves this problem nicely:)

[rgommers]

Plus positional-only is only 3.8+ so anything that supports 3.7 would have to implement positional-only with *args.

So for the time being I think it's more user-facing advice: "only use this with positional arguments". It can only be enforced with reasonable effort in ~1.5 years time (we're just about to drop Python 3.6 now).

[asmeurer]

Another issue, numpy raises ValueError for incorrect call signatures. For standard Python functions, they raise TypeError. Do we want to require any specific error here? (IMHO numpy raise ValueError instead of TypeError is wrong, but I don't know if that needs to be part of the spec)

[rgommers]

No, I think that's too detailed for the spec. And yes, I agree that what NumPy does seems wrong.

[asmeurer]

OK, I pushed up some work in progress for this for elementwise functions. Here are the failures for numpy (ARRAY_API_TESTS_MODULE=numpy pytest):

FAILED array_api_tests/test_signatures.py::test_has_names_elementwise[acos] - AssertionError: numpy is missing the elementwise function acos
FAILED array_api_tests/test_signatures.py::test_has_names_elementwise[acosh] - AssertionError: numpy is missing the elementwise function acosh
FAILED array_api_tests/test_signatures.py::test_has_names_elementwise[asin] - AssertionError: numpy is missing the elementwise function asin
FAILED array_api_tests/test_signatures.py::test_has_names_elementwise[asinh] - AssertionError: numpy is missing the elementwise function asinh
FAILED array_api_tests/test_signatures.py::test_has_names_elementwise[atan] - AssertionError: numpy is missing the elementwise function atan
FAILED array_api_tests/test_signatures.py::test_has_names_elementwise[atanh] - AssertionError: numpy is missing the elementwise function atanh
FAILED array_api_tests/test_signatures.py::test_function_parameters[abs] - AssertionError: abs should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_parameters[add] - AssertionError: add should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_parameters[equal] - AssertionError: equal should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_parameters[greater] - AssertionError: greater should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_parameters[greater_equal] - AssertionError: greater_equal should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_parameters[less] - AssertionError: less should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_parameters[less_equal] - AssertionError: less_equal should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_parameters[logical_and] - AssertionError: logical_and should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_parameters[logical_not] - AssertionError: logical_not should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_parameters[logical_or] - AssertionError: logical_or should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_parameters[logical_xor] - AssertionError: logical_xor should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_parameters[multiply] - AssertionError: multiply should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_parameters[not_equal] - AssertionError: not_equal should not accept 3 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_parameters[round] - AssertionError: round should not accept 2 positional arguments
FAILED array_api_tests/test_signatures.py::test_function_parameters[subtract] - AssertionError: subtract should not accept 3 positional arguments

We'll probably also want to have meta-tests that test that the tests fail correctly in all the cases.

[asmeurer]

Also added a test for keyword-only arguments. Example error message from dask.array:

AssertionError: Unexpected exception TypeError("round() got an unexpected keyword argument 'out'"): round() should accept the keyword-only argument 'out'

[rgommers]

Nice - test failures all look good.

These tests are going to be hard to understand, so some docstrings/comments on what each test does is for could be useful. Also for something like raises, I immediately wondered how it was different from pytest.raises.

[asmeurer]

Also for something like raises, I immediately wondered how it was different from pytest.raises.

Mainly I want to be able to include a custom error message, which pytest.raises does not seem to support.

[asmeurer]

Actually, should we really be testing that each function takes no more than n positional arguments? Or should we allow additional arguments which may also be positional (the tests will test that at least the first n are as in the spec)? For example, out (which has been removed) could be implemented as positional. Or something like 3-arg pow could be implemented.

[rgommers]

Yes I think we should be testing for the exact amount of positional args. Because if that differs between implementations, there is no way to write portable code. You'd have to write stuff like

if arraylib == np:
    func(a, b)
elif arraylib == torch:
    func(a, b, out)
...

[asmeurer]

Couldn't you say the same thing about additional keyword arguments that aren't in the spec? Is the idea for libraries have exactly the signature in the spec or at least that signature? In your example, out isn't part of the spec, so you can't include it in any way that is portable.

[asmeurer]

Maybe you're confusing what I'm asking. If f has two positional arguments in the spec, then f(x, y) with exactly two arguments should definitely work. But the question is if f(x, y, z) with a third argument should not be allowed work. It is possible for both f(x, y) and f(x, y, z) to work if the third argument is an optional positional argument (keyword but not keyword-only). Right now we test that f(x, y) doesn't raise and that f(), f(x), and f(x, y, z) do raise, but I'm not sure if the last of these (f(x, y, z) raises) should be checked. If out were a required positional argument, then the check that f(x, y) doesn't raise would fail, but I'm only wondering if it should be allowed as a non-required positional argument.

[rgommers]

It is possible for both f(x, y) and f(x, y, z) to work if the third argument is an optional positional argument (keyword but not keyword-only).

Ah, that I'd call a keyword argument, not "optional positional argument".

Maybe you're confusing what I'm asking.

Yes I guess so.

but I'm not sure if the last of these (f(x, y, z) raises) should be checked

Agreed, let's not check that.

[asmeurer]

Ah, that I'd call a keyword argument, not "optional positional argument".

Right. In Python, the most common way to implement an optional positional argument is to make the last arguments keyword arguments, but it's not necessary. For example, you could also implement *args and do different things depending on the length of args. Or you could (in theory) use some kind of dispatching library that lets you write different length signatures in a C++ style (the point only being that optional positional arguments and keyword arguments aren't necessarily the same thing). It only matters for us because "number of positional arguments" is the only reasonable thing we can test by poking at a function (i.e., we can test that a function only accepts n positional arguments, but we can't test that a function accepts only X keyword arguments).