Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Addition operation between NaN and list returns NaN #41764

Open
2 of 3 tasks
aj125 opened this issue Jun 1, 2021 · 1 comment
Open
2 of 3 tasks

BUG: Addition operation between NaN and list returns NaN #41764

aj125 opened this issue Jun 1, 2021 · 1 comment
Labels
Deprecate Functionality to remove in pandas Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Needs Discussion Requires discussion from core team before further action Nested Data Data where the values are collections (lists, sets, dicts, objects, etc.). Numeric Operations Arithmetic, Comparison, and Logical operations

Comments

@aj125
Copy link

aj125 commented Jun 1, 2021

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • (optional) I have confirmed this bug exists on the master branch of pandas.


>>> import pandas as pd
>>> import numpy as np
>>> df = pd.DataFrame({'c1':[[1,2,3]], 'c2':[np.nan]})
>>> df
          c1  c2
0  [1, 2, 3] NaN
>>> df.c1 + df.c2
0    NaN
dtype: object
>>> [1, 2, 3] + np.nan
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can only concatenate list (not "float") to list

Problem description

In Pandas when using the "+" operator on a Series entry containing a list and a Series entry containing a NaN value, a NaN value is returned. This is inconsistent with standard Python where attempting to concatenate a list and a NaN raises a TypeError. In general, list concatenation between a NaN and a list is an ambiguous operation and therefore should raise an error.

Expected Output

A TypeError should be raised.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 2cb9652
python : 3.7.4.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.18362
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None

pandas : 1.2.4
numpy : 1.19.5
pytz : 2021.1
dateutil : 2.8.1
pip : 21.1.1
setuptools : 52.0.0.post20210125
Cython : None
pytest : None
hypothesis : None
sphinx : 4.0.1
blosc : None
feather : None
xlsxwriter : 1.2.8
lxml.etree : 4.5.0
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.0
IPython : 7.16.1
pandas_datareader: None
bs4 : 4.8.0
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.4
numexpr : None
odfpy : None
openpyxl : 3.0.1
pandas_gbq : None
pyarrow : 0.17.1
pyxlsb : None
s3fs : None
scipy : 1.6.2
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : 1.2.0
xlwt : None
numba : None

@aj125 aj125 added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Jun 1, 2021
@jbrockmendel jbrockmendel added the Nested Data Data where the values are collections (lists, sets, dicts, objects, etc.). label Jun 8, 2021
@jbrockmendel
Copy link
Member

This is a consequence of the method falling back to using _masked_arith_op, which in this case is working as intended. I for one would be OK with deprecating that behavior, but we are talking about an API change, not a bug.

@jbrockmendel jbrockmendel added API Design Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Numeric Operations Arithmetic, Comparison, and Logical operations and removed Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Jun 8, 2021
@simonjayhawkins simonjayhawkins added this to the Contributions Welcome milestone Jun 8, 2021
@mroeschke mroeschke added Deprecate Functionality to remove in pandas Needs Discussion Requires discussion from core team before further action and removed API Design labels Aug 21, 2021
@mroeschke mroeschke removed this from the Contributions Welcome milestone Oct 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Deprecate Functionality to remove in pandas Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Needs Discussion Requires discussion from core team before further action Nested Data Data where the values are collections (lists, sets, dicts, objects, etc.). Numeric Operations Arithmetic, Comparison, and Logical operations
Projects
None yet
Development

No branches or pull requests

4 participants