Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

groupby observed=True not working for aggregating a column #23970

Closed
Gerenuk opened this issue Nov 28, 2018 · 2 comments

Comments

Projects
None yet
4 participants
@Gerenuk
Copy link

commented Nov 28, 2018

In the following the parameter observed=True will be ineffective (in pandas 0.23.4)

import pandas as pd
dd=pd.DataFrame({"c":pd.Series([1,1,2], dtype="category"), "i":[1,2,2], "x":[1, 2, 3]})
gr=dd.groupby(["c", "i"], as_index=False, observed=True)["x"].sum()
gr   # will contain a NaN for c=2, i=1

It will only work as expected if I aggregate over all DataFrame columns, i.e. leaving out ["x"].

INSTALLED VERSIONS ------------------ commit: None python: 3.6.7.final.0 python-bits: 64 OS: Windows OS-release: 10 machine: AMD64 processor: Intel64 Family 6 Model 78 Stepping 3, GenuineIntel byteorder: little LC_ALL: None LANG: None LOCALE: None.None

pandas: 0.23.4
pytest: 3.10.1
pip: 18.1
setuptools: 40.5.0
Cython: 0.29
numpy: 1.15.4
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 7.1.1
sphinx: None
patsy: 0.5.1
dateutil: 2.7.5
pytz: 2018.7
blosc: None
bottleneck: 1.2.1
tables: None
numexpr: 2.6.8
feather: None
matplotlib: 3.0.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.3
html5lib: 1.0.1
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

@chris-b1

This comment has been minimized.

Copy link
Contributor

commented Nov 29, 2018

Seems to be an interaction with as_index - investigation/PR welcome!

In [120]: dd.groupby(["c", "i"], observed=True)["x"].sum()
Out[120]: 
c  i
1  1    1
   2    2
2  2    3
Name: x, dtype: int64

In [121]: dd.groupby(["c", "i"], observed=True)["x"].sum().reset_index()
Out[121]: 
   c  i  x
0  1  1  1
1  1  2  2
2  2  2  3
@Koustav-Samaddar

This comment has been minimized.

Copy link
Contributor

commented Dec 24, 2018

@chris-b1 Hi! I just created a PR fixing the bug.

Please ignore #24411 since I had some weird rebasing issue that caused a ton of merge conflicts, due to an out of date fork.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.