Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: .corr() fails when input is Int64Dtype() #33803

Closed
3 tasks done
akanz1 opened this issue Apr 26, 2020 · 3 comments · Fixed by #33809
Closed
3 tasks done

BUG: .corr() fails when input is Int64Dtype() #33803

akanz1 opened this issue Apr 26, 2020 · 3 comments · Fixed by #33809
Labels
Bug NA - MaskedArrays Related to pd.NA and nullable extension arrays Numeric Operations Arithmetic, Comparison, and Logical operations
Milestone

Comments

@akanz1
Copy link

akanz1 commented Apr 26, 2020

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • This bug exists on the master branch of pandas. see below


Code Sample, a copy-pastable example

df = pd.DataFrame({'First':[1,2,3,4,np.nan],
                   'Second':[True, True, False, False, True],
                   'Third':[2,3,8,5,6]})
print(df.dtypes)
print(df.corr())
df
df_conv = df.convert_dtypes()
print(df_conv.dtypes)
print(df_conv)
df_conv.corr() # results in the following TypeError

TypeError: float() argument must be a string or a number, not 'NAType'

Problem description

When applying convert_dtypes() to a DataFrame, columns containing NaNs (np.nan) and integer values (stored as float64) are converted to "pd.NA" and actual integers (Int64). While this is desirable, it results in an error when calling the .corr() method on the DataFrame with the new dtypes.

Before convert_dtypes():
before_conv

After convert_dtypes():
after_conv

Error message:
corr_error

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.6.final.0
python-bits : 64
OS : Windows
OS-release : 10
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None

pandas : 1.0.3
numpy : 1.18.1
pytz : 2019.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 46.1.3.post20200325
Cython : None
pytest : 5.4.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.13.0
pandas_datareader: None
bs4 : 4.9.0
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.2.1
numexpr : None
odfpy : None
openpyxl : 3.0.3
pandas_gbq : None
pyarrow : None
pytables : None
pytest : 5.4.1
pyxlsb : None
s3fs : None
scipy : 1.4.1
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : 1.2.0
xlwt : None
xlsxwriter : None
numba : 0.48.0

@akanz1 akanz1 added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Apr 26, 2020
@dsaxton dsaxton added NA - MaskedArrays Related to pd.NA and nullable extension arrays Numeric Operations Arithmetic, Comparison, and Logical operations and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Apr 26, 2020
@dsaxton
Copy link
Member

dsaxton commented Apr 26, 2020

@akanz1 Thanks, looks like a bug to me and fails on master as well. Since the issue is with corr and not convert_dtypes could you update the title to reflect that?

@akanz1 akanz1 changed the title BUG: convert_dtypes() / convert_integer BUG: .corr() fails when input is Int64Dtype() Apr 26, 2020
@akanz1
Copy link
Author

akanz1 commented Apr 26, 2020

@dsaxton Thank you for your quick response, I've updated the title.

@akanz1
Copy link
Author

akanz1 commented Apr 26, 2020

Getting the same TypeError for .cov().

pointing to --> libalgos.nancorr(ensure_float64(mat) ...) as well

cov_error

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug NA - MaskedArrays Related to pd.NA and nullable extension arrays Numeric Operations Arithmetic, Comparison, and Logical operations
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants