Improve memory footprint of isin by using contains

[wence-]

Description

Previously, isin was implemented using an inner join between the column we are searching (the haystack) and the values we are searching for (the needles). This had a large memory footprint when there were repeated needles (since that blows up the cardinality of the merge).

To fix this, note that we don't need to do a merge at all, since libcudf provides a primitive (contains) to search for many needles in a haystack. The only thing we must bear in mind is that left.isin(right) is asking for the locations in left that match an entry in right, whereas contains(haystack, needles) provides a bool mask that selects needles that are in the haystack. To get the behaviour we want, we therefore need to do contains(right, left) and treat the values to search for as the haystack.

As well as having a much better memory footprint, this hash-based approach search is significantly faster than the previous merge-based one.

While we are here, lower the memory footprint of MultiIndex.isin by using a left-semi join (the implementation is separate from the isin implementation on columns and looks a little more complicated to unpick).

• Closes [BUG] OOM with isin when lhs/rhs contain repeats / spill fails #14298

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[wence-]

As well as having a much better memory footprint, this hash-based approach search is significantly faster than the previous merge-based one

Using the data from #14298:

Old:

In [1]: %timeit result = haystack.isin(needles.unique())
6.08 ms ± 24.6 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

New

# No need for unique now either
In [1]: %timeit result = haystack.isin(needles)
123 µs ± 273 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

Happy thanksgiving @thomcom

[wence-]

Hmph, broken somehow...

[wence-]

The problem is that contains preserves the null mask of the needles being searched for. I fix this by dropping the returned null mask on the floor, but marking as do-not-merge so that we can discuss if this is the best approach.

[wence-]

/merge