dask groupby could support dropna argument

[jangorecki]

I haven't found any existing request so filling this one.
If there are no plans to add support anytime soon, then I think it make sense to add at least a warning when user provide this argument. This is to prevent to silently giving unexpected (although documented) results.

cat > data.csv <<EOL
id1,id2,v1
a,1,4.5
,2,5.5
b,,
EOL
python

import pandas as pd
import dask
import dask.dataframe as dd

pd.__version__
#'1.1.5'
dask.__version__
#'2020.12.0'

pf = pd.read_csv("data.csv", dtype={"id1":"string", "id2":"Int32", "v1":"Float64"})
pa = pf.groupby(['id1','id2'], dropna=False, as_index=False).agg({'v1':'sum'})
pa
#    id1   id2   v1
#0     a     1  4.5
#1     b  <NA>  0.0
#2  <NA>     2  5.5

df = dd.from_pandas(pf, 1, sort=False)
da = df.groupby(['id1','id2'], dropna=False).agg({'v1':'sum'}).compute()
da.reset_index(inplace=True)
da
#  id1  id2   v1
#0   a    1  4.5

[jsignell]

Hmm groupby does support dropna (I am opening a PR to fix the docstring), but it seems like it is only implemented for the named methods. sum, mean... and apply. I think we missed adding it to the generic agg.

[jangorecki]

That is a good news.

From your PR it also looks that observed argument is also supported already, could you please confirm?

As pandas cannot yet use dropna for categorical columns, pandas-dev/pandas#36327, I assume dask won't be able to do it as well. Once support for that will be implemented in pandas, is it gong to work automatically in dask? or it will have to be added on dask side as well?

[jsignell]

Yeah observed was added pretty recently. I think that dropna will just work when categorical support is added in pandas, but I am not 100% sure.