tm.assert_index_equal broken with pd.NA and np.NaT sentinel

[WillAyd]

Discovered in #31799

I couldn't reproduce this without using a np.* NaT sentinel...

>>> idx1 = pd.Index([pd.NA, np.datetime64("nat")])
>>> idx2 = pd.Index([pd.NA, pd.NaT])
>>> tm.assert_index_equal(idx1, idx2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/williamayd/clones/pandas/pandas/_testing.py", line 689, in assert_index_equal
    diff = np.sum((left.values != right.values).astype(int)) * 100.0 / len(left)
AttributeError: 'bool' object has no attribute 'astype'

[jbrockmendel]

cc @jorisvandenbossche any thoughts on how to handle this? The left.values != right.values in the OP is getting evaluated to True and emitting

DeprecationWarning: elementwise comparison failed; this will raise an error in the future.
    diff = np.sum((left.values != right.values).astype(int)) * 100.0 / len(left)

[jorisvandenbossche]

It's a general problem with how numpy handles the non-boolean result of comparing pd.NA:

In [18]: np.array([pd.NA], dtype=object) == 1
/home/joris/miniconda3/envs/dev/bin/ipython:1: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.
  #!/home/joris/miniconda3/envs/dev/bin/python
Out[18]: False

Basically storing pd.NA in object dtype numpy arrays is not really supported yet, see #32931 with an overview of issues related to this topic.

One possible solution for the specific case here of the asserts is adding a check for object dtype, and then specifically check the presence of pd.NA, that this presence is equal in both left and right, and then filter them out, and compare the remaining values.

Another short-term quick workaround would also be to put the specific line in assert_index_equal in a try/except, because where it is failing is only to give a more informative error message, not the actual check whether the indexes are equal or not.

[mroeschke]

Looks like the comparison is a little better now. Could use a test

In [6]: >>> idx1 = pd.Index([pd.NA, np.datetime64("nat")])
   ...: >>> idx2 = pd.Index([pd.NA, pd.NaT])
   ...: >>> tm.assert_index_equal(idx1, idx2)

AssertionError: Index are different

Attribute "inferred_type" are different
[left]:  mixed
[right]: datetime

[MrShevan]

Hi, @mroeschke ! If I understand correctly, bug is fixed and there is needed test for checking AssertionError: Index are different for these inputs format idx1 = pd.Index([pd.NA, np.datetime64("nat")]) and idx2 = pd.Index([pd.NA, pd.NaT]) to close this issue. Could I take it? :)

[mroeschke]

Correct, @MrShevan. A unit test will need to be added to test that the code example still raises an exception