Add np.in1d, np.isin, np.setxor1d, np.setdiff1d, extend np.intersect1d.

[jaredjeya]

• Adds support for the functions np.in1d, np.isin, np.setxor1d, and np.setdiff1d, all including optional argument assume_unique as well as invert for np.in1d and np.isin.

• Extends np.intersect1d to support the first optional argument (assume_unique).

Resolves #4677. Improves on #4074.

I have written unit tests for all the new functions and functionality, and applied flake8 to my code.

The only question I have is how to handle the optional argument kind in np.in1d: I've implemented the kind="sort" algorithm, but this is not the default behaviour of numpy (which picks an algorithm based on performance heuristics), so I've made it explicit that only kind="sort" is supported.

[jaredjeya]

I’ve just now seen #4815, possibly some of the code could be ported across from there too.

[jaredjeya]

I'm not sure why those tests are failing all of a sudden - it seems to be at the flake8 step? But the errors flake8 is highlighting are unrelated to code I've modified and it didn't fail in the earlier test.

Edit: nvm, spotted #9342.

[stuartarchibald]

@jaredjeya feel free to ignore the flake8 failures until #9342 is merged later today (a new version of flake8 was released and it picked up a couple of new issues). Thanks.

[jaredjeya]

@jaredjeya feel free to ignore the flake8 failures until #9342 is merged later today (a new version of flake8 was released and it picked up a couple of new issues). Thanks.

Now it’s been merged, how do I get the failed tests to run again? Is there anything I need to do to keep the new code up-to-date with the main branch? (Sorry, this is my first ever PR to a project in active development!)

[gmarkall]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[gmarkall]

@jaredjeya Many thanks for the PR! I'm tentatively moving it to the 0.60 milestone - at the moment we're working on the 0.59 release, and the 0.60 release will be focused around supporting NumPy 2.0, so it may be some time before it's possible to review this PR.

[guilhermeleobas]

Thanks, @jaredjeya. Code seems fine as it is almost a 1:1 port of what NumPy does. Just have some minor comments that should be easy to address.

[guilhermeleobas]

@jaredjeya, could you include the tests from NumPy for the set of functions added in this PR?
https://github.com/numpy/numpy/blob/b0371ef240560e78b651a5d7c9407ae3212a3d56/numpy/lib/tests/test_arraysetops.py#L15

[jaredjeya]

So what I've done here is add the numpy test cases to the tests I'd already written, rather than writing new tests entirely. Also fixed that I'd been casting all the inputs to arrays before testing them, so I wasn't testing the functionality with lists.

Otherwise I also added a check to catch at compile time if someone calls these functions with incorrect literal values for kind, but I don't enforce that it's a compile time constant. I also changed some of the error messages for readability and consistency.

[jaredjeya]

I forgot to add the attribution for test_setops_manyways, which should read:

# https://github.com/numpy/numpy/blob/b0371ef240560e78b651a5d7c9407ae3212a3d56/numpy/lib/tests/test_arraysetops.py#L588 # noqa: E501

Since it's just a comment is there a way to add that in without re-running all the tests?

[guilhermeleobas]

The default value for kind should be None, not "sort".

The following should work on your branch:

import numpy as np
from numba import njit

@njit
def foo():
    return np.isin(3, 4, kind=None)

print(foo())

[jaredjeya]

The reason I've done it like I did is because I didn't implement the numpy kind="table" method. (The output is equivalent, but I suppose the performance is higher in certain cases with the table method (I haven't tested)). When kind=None, the numpy code will use the table method if certain conditions are met, otherwise the sorting method, so we don't support that either -- strictly, we only support kind="sort". So I thought np.isin(3, 4, kind=None) should fail.

If you think it should work I'm happy to change it, or otherwise we just say we don't support the "kind" argument and leave it at that. Or, I could even implement the "table" method, it isn't that much more complicated than the sort method, and then we fully support the "kind" argument.

[guilhermeleobas]

Okay. I think it is best to match as close as possible the NumPy signature but keep a check internally for kind == "sort".

[jaredjeya]

I decided to add some tests to make sure the correct errors result when nonsense inputs are supplied, including e.g. literals like kind="foo". However I can't for the life of me figure out how to get numba to actually compile these as literals.

This is the full type-checking code within the @overload function:

if kind is not None:
    if (isinstance(kind, types.StringLiteral)
            and kind.literal_value != "sort"):
        raise NumbaValueError('isin: Only kind="sort" is supported')
    elif not isinstance(kind, (types.UnicodeType, str, types.NoneType)):
        raise TypingError('isin: Argument "kind" must be a string or None')

this is the code in the actual implementation,

if kind is not None and kind != "sort":
    raise ValueError('isin: Only kind="sort" is supported')

and this is the testing code:

@njit()
def table_isin(a, b):
    return np.isin(a, b, kind="table")
with self.assertRaises(NumbaValueError):
    table_isin(a, b)

I also tried kind=literal("table") and kind=literally("table"). But the code always raises a ValueError at runtime rather than a NumbaValueError at compilation time, and if I insert debug print statements, I see repr(kind) == unicode_type and type(kind) == <class 'numba.core.types.misc.UnicodeType'>.

What am I doing wrong? Other functions e.g. np.searchsorted use identical patterns in their tests.

[jaredjeya]

Nevermind, I sorted it

[guilhermeleobas]

If kind=None, NumPy will use table by default, which is not supported in this PR. Change the clause to something like this:

Suggested change

	if kind is not None and kind != "sort":
	raise ValueError('isin: Only kind="sort" is supported')
	if kind != "sort":
	raise ValueError('isin: Only kind="sort" is supported')

This will force a user to spell the kind="sort" keyword argument and not assume Numba supports the table option.

[jaredjeya]

If kind=None, NumPy will use table by default, which is not supported in this PR.

Yes, that is why I didn't implement kind=None in the first place. I'll change it back.

[guilhermeleobas]

The function signature needs to be with kind=None as it is what NumPy specifies. But let's keep a check to only run the function if the user specifies kind="sort".

Let me know if you have any questions regarding this.

[guilhermeleobas]

That's fine, but we can keep only the ValueError to simplify the codebase.

[jaredjeya]

To be clear, I should get rid of this whole section?

[jaredjeya]

Actually I've reduced it to just,

if not isinstance(kind, (types.UnicodeType, str, types.StringLiteral)):
    raise TypingError('in1d: Only kind="sort" is supported')

which makes sure kind=None doesn't make it past compilation. It seemed to be causing some type inference issues if I didn't have that.

[jaredjeya]

@guilhermeleobas Tests seem to be failing now because numpy < 1.24 doesn't support the kind keyword, and because we now force this to be included explicitly, the numpy and numba behaviours diverge. What do you suggest?

According to the release notes, pre-1.24, the kind="sort" behaviour was the default. Newer versions automatically pick the optimal algorithm.

It might not be too hard to implement kind="table", but it would take me a while as I'm quite busy atm. Otherwise we can drop the keyword entirely and say we're only replicating pre-1.24 behaviour? The documentation suggests using a version gate but I don't know what that means in this context.

[guilhermeleobas]

According to the release notes, pre-1.24, the kind="sort" behaviour was the default. Newer versions automatically pick the optimal algorithm.

I think we can have an if-else statement to support the kind argument if NumPy version if greater or equal 1.24:

if numpy_version < (1, 24):
    @overload(np.isin)
    def jit_np_isin(element, test_elements, assume_unique=False, invert=False):
         ....

else:
    @overload(np.isin)
    def jit_np_isin(element, test_elements, assume_unique=False, invert=False,
                    kind=None):
        ...

You may have to update the tests as well to only pass kind="sort" if NumPy >= 1.24

Otherwise we can drop the keyword entirely and say we're only replicating pre-1.24 behaviour?

That's also an option. Choose the one that is easiest for you to implement.