API: Index should support __inverse__ ops

[sinhrks]

• Add __inverse__ to Index.
• Change duplicated to return np.array (PR API: Index.duplicated should return np.array #9112)

Related to #7979

Because Index is no more the subclass of np.array, Index.duplicated returns Index with dtype=object. And Index don't accepts logical not.

idx = pd.Index([1, 2, 1, 3])

idx.duplicated()
# Index([False, False, True, False], dtype='object')

~idx.duplicated()
# TypeError: bad operand type for unary ~: 'Index'

As a result, it is impossible to drop data which have duplicated index using expression like df[~df.index.duplicated()]. This expression was worked at the timing of #7979.

Does Index.duplicated should return np.array with dtype=bool? Or Index should accept logical not?

[jtratner]

maybe index should allow ~? any reason why it shouldn't respond to that? (might not have been implemented previously)

[jreback]

need to define __invert__ in core/base.py (its defined in core/generic.py so Series works), but Index is left out

[sinhrks]

OK. Changed the title.

Currently Index doesn't support bool dtype and values are stored as object. To invert it as expected, values should be once converted to bool explicitly. This conversion is specific in Index, thus adding __invert__ to Index looks better (until Index can support bool)?

~np.array([True, False], dtype=object)
# [-2 -1]
~np.array([True, False], dtype=bool)
# [False  True]

[shoyer]

Actually, I think idx.duplicated() should indeed return an np.ndarray, as you originally suggested. There's not much use for Boolean indexes, since they can only take on two values.

That said, it's probably a good idea to support ~ for index objects, too. I think this was just overlooked in the removal of ndarray subclassing.

[jreback]

@sinhrks I agree with both of @shoyer suggestions. Currently boolness is detected in the constructor, so I just think .duplicated() is not creating the booled Index correctly, AND then if it supports the __invert__ op then should work.

[sinhrks]

OK. Then prepare 2 separate fixes:

• Add __inverse__ to Index.
• Change duplicated to return np.array

[mroeschke]

This look to work on master now. Could use a test for the inverse behavior

In [2]: idx = pd.Index([1, 2, 1, 3])
   ...:
   ...: idx.duplicated()
Out[2]: array([False, False,  True, False])

In [3]: ~idx.duplicated()
Out[3]: array([ True,  True, False,  True])

In [4]: pd.__version__
Out[4]: '1.1.0.dev0+1466.ga3477c769.dirty'

[simonjayhawkins]

This look to work on master now. Could use a test for the inverse behavior

see #8875 (comment)

Index.duplicated() has been changed to return an array, but the other part of the issue is to add _inverse_ to Index. This has not yet been done.

>>> pd.__version__
'1.1.0.dev0+1748.g0bd1f6f6f'
>>>
>>> idx = pd.Index([1, 2, 1, 3])
>>>
>>> pd.Index(idx.duplicated())
Index([False, False, True, False], dtype='object')
>>>
>>> ~pd.Index(idx.duplicated())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: bad operand type for unary ~: 'Index'
>>>

[YuRenee]

take