groupby observed=True not working for aggregating a column

[Gerenuk]

In the following the parameter observed=True will be ineffective (in pandas 0.23.4)

import pandas as pd
dd=pd.DataFrame({"c":pd.Series([1,1,2], dtype="category"), "i":[1,2,2], "x":[1, 2, 3]})
gr=dd.groupby(["c", "i"], as_index=False, observed=True)["x"].sum()
gr   # will contain a NaN for c=2, i=1

It will only work as expected if I aggregate over all DataFrame columns, i.e. leaving out ["x"].

INSTALLED VERSIONS ------------------ commit: None python: 3.6.7.final.0 python-bits: 64 OS: Windows OS-release: 10 machine: AMD64 processor: Intel64 Family 6 Model 78 Stepping 3, GenuineIntel byteorder: little LC_ALL: None LANG: None LOCALE: None.None

pandas: 0.23.4
pytest: 3.10.1
pip: 18.1
setuptools: 40.5.0
Cython: 0.29
numpy: 1.15.4
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 7.1.1
sphinx: None
patsy: 0.5.1
dateutil: 2.7.5
pytz: 2018.7
blosc: None
bottleneck: 1.2.1
tables: None
numexpr: 2.6.8
feather: None
matplotlib: 3.0.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.3
html5lib: 1.0.1
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[chris-b1]

Seems to be an interaction with as_index - investigation/PR welcome!

In [120]: dd.groupby(["c", "i"], observed=True)["x"].sum()
Out[120]: 
c  i
1  1    1
   2    2
2  2    3
Name: x, dtype: int64

In [121]: dd.groupby(["c", "i"], observed=True)["x"].sum().reset_index()
Out[121]: 
   c  i  x
0  1  1  1
1  1  2  2
2  2  2  3

[Koustav-Samaddar]

@chris-b1 Hi! I just created a PR fixing the bug.

Please ignore #24411 since I had some weird rebasing issue that caused a ton of merge conflicts, due to an out of date fork.