Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: pd.isna handles np.nan inconsistently between numpy and extension / arrow types #58151

Closed
3 tasks done
WillAyd opened this issue Apr 4, 2024 · 4 comments
Closed
3 tasks done
Labels
Bug Needs Triage Issue that has not been reviewed by a pandas team member

Comments

@WillAyd
Copy link
Member

WillAyd commented Apr 4, 2024

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np
import pyarrow as pa

np_arr = pd.Series([0, np.nan], dtype=np.float64) / 0
pd_arr = pd.Series([0, np.nan], dtype=pd.Float64Dtype()) / 0
pa_arr = pd.Series([0, np.nan], dtype=pd.ArrowDtype(pa.float64())) / 0
>>> np_arr
0   NaN
1   NaN
dtype: float64
>>> pd.isna(np_arr)
0    True
1    True
dtype: bool
>>> pd_arr
0     NaN
1    <NA>
dtype: Float64
>>> pd.isna(pd_arr)
0    False
1     True
dtype: bool
>>> pa_arr
0     NaN
1    <NA>
dtype: double[pyarrow]
>>> pd.isna(pa_arr)
0    False
1     True
dtype: bool

Issue Description

In the NumPy case, the NaN value resulting from the 0 / 0 division is caught by pd.isna but not in the pandas / pyarrow case

Expected Behavior

I think all should be True @jorisvandenbossche

Installed Versions

run on main

@WillAyd WillAyd added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Apr 4, 2024
@jbrockmendel
Copy link
Member

There are several issues about this. We talked about it at the sprint

@phofl
Copy link
Member

phofl commented May 13, 2024

agree with @jbrockmendel

this is a duplicate of a bunch of issues, so closing here

@phofl phofl closed this as completed May 13, 2024
@WillAyd
Copy link
Member Author

WillAyd commented May 13, 2024

@phofl can you link to some of the issues you know about so readers can follow the discussion? I am not sure where to track this myself

@phofl
Copy link
Member

phofl commented May 13, 2024

#53887 and a bunch of issues mentioned there

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Needs Triage Issue that has not been reviewed by a pandas team member
Projects
None yet
Development

No branches or pull requests

3 participants