BUG in handling of sum/mean of mixed type series

[deronnek]

>>> pd.Series([True, np.float64(5)>0.2, np.nan]).sum()
1

>>> pd.Series([np.nan, np.float64(5)>0.2, True]).sum()
2

Problem description

The only thing different between the two cases above is the order of the arguments, which should not affect the result.

.mean() is also affected, and the same result occurs when using the series as a column in a dataframe.

Expected Output

I would expect both of the above to return 2, as there are two True values in the list

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit : None python : 3.7.5.final.0 python-bits : 64 OS : Darwin OS-release : 18.2.0 machine : x86_64 processor : i386 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 0.25.3
numpy : 1.17.3
pytz : 2019.3
dateutil : 2.8.1
pip : 19.3.1
setuptools : 41.6.0.post20191030
Cython : 0.29.14
pytest : 5.2.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 7.9.0
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.1.1
numexpr : 2.7.0
odfpy : None
openpyxl : None
pandas_gbq : 0.11.0
pyarrow : None
pytables : None
s3fs : None
scipy : 1.3.1
sqlalchemy : None
tables : 3.6.1
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None

[WillAyd]

Hmm that does look strange. Investigation and PRs would certainly be welcome

[deronnek]

An (even more) succinct example--the issue seems to have something to do with the np.bool_ type:

>>> pd.Series([True, np.bool(0.2), np.nan]).sum()
2
>>> pd.Series([True, np.bool_(0.2), np.nan]).sum()
1

[rajibmitra]

Lets add one test case for this and let the test fail first.
Does that sound good ?

[rishabh7m]

An (even more) succinct example--the issue seems to have something to do with the np.bool_ type:

>>> pd.Series([True, np.bool(0.2), np.nan]).sum()
2
>>> pd.Series([True, np.bool_(0.2), np.nan]).sum()
1

Hi, I don't think this is because of np.bool_. In nanops.py dtype is assigned as object which makes sense as it has mixed data types. When dtype is changed to np.int64 it returns 2. I also tried to create same case by using numpy arrays

>>> np.array([True, np.float64(5)>0.2, 0], dtype=object).sum()
1
>>> np.array([True, np.float64(5)>0.2, 0], dtype=np.int64).sum()
2

I will look into numpy code. But I would suggest to replace nan with some other valid value and then change thedtype while passing the array.

With pandas:

>>> pd.Series([True, np.float64(5)>0.2, 0], dtype=object).sum()
1
>>> pd.Series([True, np.float64(5)>0.2, 0], dtype=np.int64).sum()
2

[rishabh7m]

An (even more) succinct example--the issue seems to have something to do with the np.bool_ type:

>>> pd.Series([True, np.bool(0.2), np.nan]).sum()
2
>>> pd.Series([True, np.bool_(0.2), np.nan]).sum()
1

It might not be treating np.bool and np.bool_ as same

>>> np.bool_(0.2) is True
False
>>> np.bool(0.2) is True
True

[deronnek]

Weirdly, even though np.bool_(0.2) is True evaluates to False:

>>> np.bool_(0.2)
True
>>> if np.bool_(0.2):
...     print("hi")
...
hi

[rishabh7m]

Stumbled upon this issue on numpy.

[deronnek]

It can't be just the dtype, though, as both of the series have type object in my original example:

>>> pd.Series([True, np.float64(5)>0.2, np.nan]).dtype
dtype('O')
>>> pd.Series([np.nan, np.float64(5)>0.2, True]).dtype
dtype('O')
>>>

[rishabh7m]

True, that can't be just dtype issue. Sorry, I still haven't looked at numpy.sum implementation. Will get back to you asap.