BUG: InvalidIndexError raised in Categorical.isin for categorical backed by interval with overlapping intervals

[TomAugspurger]

Code Sample, a copy-pastable example

In [1]: import pandas as pd
In [2]: idx = pd.IntervalIndex([pd.Interval(0, 2), pd.Interval(0, 1)])

In [3]: pd.Categorical(idx).isin(idx)
---------------------------------------------------------------------------
InvalidIndexError                         Traceback (most recent call last)
<ipython-input-3-801ab88eb4d0> in <module>
----> 1 pd.Categorical(idx).isin(idx)

~/sandbox/pandas/pandas/core/arrays/categorical.py in isin(self, values)
   2359         values = sanitize_array(values, None, None)
   2360         null_mask = np.asarray(isna(values))
-> 2361         code_values = self.categories.get_indexer(values)
   2362         code_values = code_values[null_mask | (code_values >= 0)]
   2363         return algorithms.isin(self.codes, code_values)

~/sandbox/pandas/pandas/core/indexes/interval.py in get_indexer(self, target, method, limit, tolerance)
    764         if self.is_overlapping:
    765             raise InvalidIndexError(
--> 766                 "cannot handle overlapping indices; "
    767                 "use IntervalIndex.get_indexer_non_unique"
    768             )

InvalidIndexError: cannot handle overlapping indices; use IntervalIndex.get_indexer_non_unique

Problem description

The result is unambigous

Expected Output

array([ True,  True])

I think that the call to self.categories.get_indexer might be self.categories.get_indexer_for instead.

[makslevental]

I have the same issue (I think) with just straight up IntervalIndex

J
Out[9]: 
                 s   r   c
  live_range              
0 [0, 3]      1024   0   3
1 [1, 3]      1024   1   3
2 [3, 4]      1024   3   4
3 [5, 10]      256   5  10
4 [6, 14]      256   6  14
5 [7, 10]      256   7  10
6 [8, 9]       256   8   9
7 [9, 12]      256   9  12
8 [10, 12]     256  10  12
9 [13, 14]     256  13  14
j.Index
Out[10]: (0, Interval(0, 3, closed='both'))
J.drop(j.Index)
Traceback (most recent call last):
  File "/Users/mlevental/opt/miniconda3/envs/pytorch_dev_shape_inference/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-11-ad3fd2b03dcb>", line 1, in <module>
    J.drop(j.Index)
  File "/Users/mlevental/opt/miniconda3/envs/pytorch_dev_shape_inference/lib/python3.9/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/Users/mlevental/opt/miniconda3/envs/pytorch_dev_shape_inference/lib/python3.9/site-packages/pandas/core/frame.py", line 4901, in drop
    return super().drop(
  File "/Users/mlevental/opt/miniconda3/envs/pytorch_dev_shape_inference/lib/python3.9/site-packages/pandas/core/generic.py", line 4147, in drop
    obj = obj._drop_axis(labels, axis, level=level, errors=errors)
  File "/Users/mlevental/opt/miniconda3/envs/pytorch_dev_shape_inference/lib/python3.9/site-packages/pandas/core/generic.py", line 4183, in _drop_axis
    result = self.reindex(**{axis_name: new_axis})
  File "/Users/mlevental/opt/miniconda3/envs/pytorch_dev_shape_inference/lib/python3.9/site-packages/pandas/util/_decorators.py", line 324, in wrapper
    return func(*args, **kwargs)
  File "/Users/mlevental/opt/miniconda3/envs/pytorch_dev_shape_inference/lib/python3.9/site-packages/pandas/core/frame.py", line 4767, in reindex
    return super().reindex(**kwargs)
  File "/Users/mlevental/opt/miniconda3/envs/pytorch_dev_shape_inference/lib/python3.9/site-packages/pandas/core/generic.py", line 4809, in reindex
    return self._reindex_axes(
  File "/Users/mlevental/opt/miniconda3/envs/pytorch_dev_shape_inference/lib/python3.9/site-packages/pandas/core/frame.py", line 4592, in _reindex_axes
    frame = frame._reindex_index(
  File "/Users/mlevental/opt/miniconda3/envs/pytorch_dev_shape_inference/lib/python3.9/site-packages/pandas/core/frame.py", line 4608, in _reindex_index
    new_index, indexer = self.index.reindex(
  File "/Users/mlevental/opt/miniconda3/envs/pytorch_dev_shape_inference/lib/python3.9/site-packages/pandas/core/indexes/multi.py", line 2520, in reindex
    indexer = self.get_indexer(
  File "/Users/mlevental/opt/miniconda3/envs/pytorch_dev_shape_inference/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3486, in get_indexer
    return self._get_indexer(target, method, limit, tolerance)
  File "/Users/mlevental/opt/miniconda3/envs/pytorch_dev_shape_inference/lib/python3.9/site-packages/pandas/core/indexes/multi.py", line 2682, in _get_indexer
    indexer = self._engine.get_indexer(target._values)
  File "pandas/_libs/index.pyx", line 623, in pandas._libs.index.BaseMultiIndexCodesEngine.get_indexer
  File "pandas/_libs/index.pyx", line 603, in pandas._libs.index.BaseMultiIndexCodesEngine._extract_level_codes
  File "/Users/mlevental/opt/miniconda3/envs/pytorch_dev_shape_inference/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3442, in get_indexer
    raise InvalidIndexError(self._requires_unique_msg)
pandas.errors.InvalidIndexError: cannot handle overlapping indices; use IntervalIndex.get_indexer_non_unique

what's the solution here?

EDIT:

some kind of class hierarchy hijinks:

self._index_as_unique
Out[19]: False
self.is_unique
Out[20]: True

EDIT2: if anyone gets here and wants a hack fix (as of 1.3.1):

drop @final from pandas/core/indexes/base.py on _validate_can_reindex in Index and copy paste this under IntervalIndex at pandas/core/indexes/interval.py:

    def _validate_can_reindex(self, indexer: np.ndarray) -> None:
        if isinstance(self, IntervalIndex):
            is_unique = self.is_unique
        else:
            is_unique = self._index_as_unique

        if not is_unique and len(indexer):
            raise ValueError("cannot reindex from a duplicate axis")