Description
Pandas version checks
I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
import pandas as pd
from pandas.testing import assert_frame_equal
df1 = pd.DataFrame(
{
"x": pd.Series([pd.NA], dtype="Int32"),
}
)
df2 = pd.DataFrame(
{
"x": pd.Series([pd.NA], dtype="object"),
}
)
assert_frame_equal(df1, df2, check_dtype=False) # fails, but should succeedIssue Description
Output of the above example:
AssertionError: DataFrame.iloc[:, 0] (column name="x") are different
DataFrame.iloc[:, 0] (column name="x") values are different (100.0 %)
[index]: [0]
[left]: [nan]
[right]: [<NA>]
When comparing DataFrames containing pd.NA using check_dtype=False, the test incorrectly fails despite the only difference being the dtype (Int32 vs object).
Note that the values in the dataframe really are the same:
print(type(df1["x"][0])) # prints <class 'pandas._libs.missing.NAType'>
print(type(df2["x"][0])) # prints <class 'pandas._libs.missing.NAType'>
Related issues:
- assert_frame_equal not differentiating NaN and None within object dtype #18463: Similar but "opposite": here the dataframes contain different values (nan vs None) which are incorrectly treated as equal. In this issue, the dataframes contain equal values which are incorrectly treated as different.
Expected Behavior
The test should succeed, since the only difference is the dtypes, and check_dtype=False.
Installed Versions
pandas : 2.2.3
numpy : 1.26.4
pytz : 2025.2
dateutil : 2.9.0.post0
pip : 24.0
Cython : None
sphinx : None
IPython : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : None
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.6
lxml.etree : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : 3.1.5
pandas_gbq : None
psycopg2 : None
pymysql : None
pyarrow : 19.0.1
pyreadstat : None
pytest : 8.3.5
python-calamine : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : 2.0.41
tables : None
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : 0.23.0
tzdata : 2025.2
qtpy : None
pyqt5 : None
Metadata
Metadata
Assignees
Labels
Type
Projects
Milestone
Relationships
Development
Participants
Activity
rhshadrach commentedon May 21, 2025
Thanks for the report, this would pass if when converting the EA to a NumPy array we cast to object dtype. I haven't looked to see if this might cause issues in other cases. Since this is aimed at tests, I'm wondering if changing to object dtype is okay here.
cc @jbrockmendel @mroeschke for any thoughts.
jbrockmendel commentedon May 21, 2025
Yah I'm pretty sure that the behavior of
df1['x'].to_numpy()casting to a float dtype was a much-discussed intentional decision. Changing that would be a can of worms.I'm inclined to just discourage the use of a) check_dtype=False and b) using pd.NA in an object dtype column (note that
df1 == df2raises)rhshadrach commentedon May 23, 2025
@jbrockmendel - sorry, I wasn't clear. I meant just inside
assert_frame_equalto use.to_numpy(dtype="object")whencheck_dtype=Falserather than just.to_numpy(). Agreed changing the behavior of.to_numpy()is off the table.jbrockmendel commentedon May 23, 2025
Gotcha, fine by me
BUG: Fix assert_frame_equal dtype handling when check_dtype=False (pa…
venturero commentedon Jun 1, 2025
how can i make contribution to solve this, can you please give advice to me? @iabhi4 @rhshadrach
11 remaining items