BUG: pd.compare does not recognize differences with the nullable Int64 data type

[sergiykhan]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np

x = pd.Series([1, 2, np.NaN], dtype='Int64')
y = pd.Series([1, 2, 3], dtype='Int64')
x.compare(y)

Issue Description

.compare does not return the difference between two Series

Result:

     self  other

Expected Behavior

     self  other
2    NaN   3

Installed Versions

INSTALLED VERSIONS

commit : 87cfe4e
python : 3.9.14.final.0
python-bits : 64
OS : Linux
OS-release : 5.10.102.1-microsoft-standard-WSL2
Version : #1 SMP Wed Mar 2 00:30:59 UTC 2022
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : en_US.UTF-8
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.5.0
numpy : 1.23.2
pytz : 2022.2.1
dateutil : 2.8.2
setuptools : 65.0.0
pip : 22.2.2
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : 2.9.3
jinja2 : 3.1.2
IPython : 8.4.0
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : 1.0.9
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.5.3
numba : None
numexpr : None
odfpy : None
openpyxl : 3.0.10
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : 1.4.40
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : 2022.2

[vamsi-verma-s]

Hi @phofl
I would like to contribute.

I see this issue is somewhat related to #44014

the problem seems to be that the mask value evaluated like below ends up having NA values -

pandas/pandas/core/generic.py

Line 9264 in e25aa9d

mask = ~((self == other) | (self.isna() & other.isna()))

I think, since the mask here is to simply determine if there are differences, if a comparison has an NA result, we can consider it as a difference instead of missing/ignoring this comparison result.

A naive solution I have is to fill NA values in the mask as True to mark the values as different.

Making this change currently fails one test that was added to fix - #44014

pandas/pandas/tests/frame/methods/test_compare.py

Lines 241 to 254 in e25aa9d

	def test_compare_ea_and_np_dtype():
	# GH#44014
	df1 = pd.DataFrame({"a": [4.0, 4], "b": [1.0, 2]})
	df2 = pd.DataFrame({"a": pd.Series([1, pd.NA], dtype="Int64"), "b": [1.0, 2]})
	result = df1.compare(df2, keep_shape=True)
	expected = pd.DataFrame(
	{
	("a", "self"): [4.0, np.nan],
	("a", "other"): pd.Series([1, pd.NA], dtype="Int64"),
	("b", "self"): np.nan,
	("b", "other"): np.nan,
	}
	)
	tm.assert_frame_equal(result, expected)

When comparing the below dataframes -

df1

     a    b
0  4.0  1.0
1  4.0  2.0

df2

      a    b
0     1  1.0
1  <NA>  2.0

the test expects the result to be -

     a          b
  self other self other
0  4.0     1  NaN   NaN
1  NaN  <NA>  NaN   NaN

I think it should be -

     a          b
  self other self other
0  4.0     1  NaN   NaN
1  4.0  <NA>  NaN   NaN

please let me know if this makes sense and I can go ahead with PR or if there is something I'm missing here

[phofl]

Yep makes sense. The test if off. Please also add tests with pd.NA and pd.NA, I think this should be considered different, but not 100% sure.

[vamsi-verma-s]

an existing test compares having 2 NaN values and expects it to be considered same. is there any reason pd.NA should behave differently?

pandas/pandas/tests/frame/methods/test_compare.py

Lines 94 to 109 in e25aa9d

	def test_compare_with_equal_nulls():
	# We want to make sure two NaNs are considered the same
	# and dropped where applicable
	df = pd.DataFrame(
	{"col1": ["a", "b", "c"], "col2": [1.0, 2.0, np.nan], "col3": [1.0, 2.0, 3.0]},
	columns=["col1", "col2", "col3"],
	)
	df2 = df.copy()
	df2.loc[0, "col1"] = "c"
	result = df.compare(df2)
	indices = pd.Index([0])
	columns = pd.MultiIndex.from_product([["col1"], ["self", "other"]])
	expected = pd.DataFrame([["a", "c"]], index=indices, columns=columns)
	tm.assert_frame_equal(result, expected)

[vamsi-verma-s]

take

[vamsi-verma-s]

@phofl could you please review the PR. I have added additional tests for pd.NA . Let me know if anything else is required.