passing a function to DataFrameGroupBy.agg - confusing documentation/behaviour

[colinmorris]

Problem description

For clarity, I'll split this into 3 sub-issues related to DataFrameGroupBy.agg and its documentation.

1. Not clear what kind of custom function should be provided

The docstring on DataFrameGroupBy.agg says that you can pass in a function, but it's unclear what that function should expect to receive as input, and how what it returns relates to the return value of agg.

After some poking around, it seems to me that the typical pattern is to pass a function that takes a series and returns a scalar, with the dataframe returned by agg having the property that aggregated.loc[group_foo, col_bar] is the result of calling the function on the series that is the column col_bar for rows belonging to group_foo.

If that is the expected behaviour, it should be explained in the docs.

2. Passing an arbitrary kwarg changes the function behaviour

I stumbled on this weirdness while trying to figure out how agg worked:

>>> df = pd.DataFrame(np.random.rand(4,2), columns=['b', 'c'])
>>> df['a'] = [1, 0, 0, 0]
>>> g = df.groupby('a')
>>> g.agg(lambda x: np.product(x.shape))
     b    c
a          
0  3.0  3.0
1  1.0  1.0
>>> g.agg(lambda x, foo=0: np.product(x.shape), foo=0)
   b  c
a      
0  9  9
1  3  3

In other words, by passing in any meaningless kwarg, my function is now called ngroups times with a dataframe for each group, rather than being called with a series ngroups * ncolumns times.

Also, when called in this way, it seems my function can return a list or series of length ncolumns which gets expanded (this doesn't work in the non-kwarg version):

>>> g.agg(lambda x, foo=0: [20, 17], foo=0)
     b   c
a        
0  20  17
1  20  17

3. Note on numpy special casing is confusing

The docstring has this note:

Numpy functions mean/median/prod/sum/std/var are special cased so the default behavior is applying the function along axis=0 (e.g., np.mean(arr_2d, axis=0)) as opposed to mimicking the default Numpy behavior (e.g., np.mean(arr_2d)).

Which is confusing because passing in lambda x: np.mean(x) does seem to give the 'right' answer (the same one as passing lambda x: np.mean(x, axis=0), or np.mean, or 'mean'). This is true whether or not I throw in a kwarg.

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 2.7.6.final.0 python-bits: 32 OS: Linux OS-release: 3.13.0-107-generic machine: i686 processor: i686 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: None.None

pandas: 0.19.2
nose: 1.3.7
pip: 9.0.1
setuptools: 25.1.0
Cython: None
numpy: 1.12.0
scipy: 0.18.1
statsmodels: 0.6.1
xarray: None
IPython: 5.1.0
sphinx: None
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2016.10
blosc: None
bottleneck: None
tables: 3.3.0
numexpr: 2.6.1
matplotlib: 2.0.0
openpyxl: 1.5.7
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.1.3
html5lib: 0.999
httplib2: 0.8
apiclient: None
sqlalchemy: 0.8.0
pymysql: None
psycopg2: None
jinja2: 2.9.4
boto: None
pandas_datareader: None

[jreback]

1. if you'd like to submit a doc PR for 1) that would be great (group.rst and/or doc-string updates / examples).

2. is a violation of the guarantees (IOW, these functions don't take parameters). I suppose it could raise a TypeError, but this is tricky to actually do.

3. not sure what you mean, the docs are clear. but again if you don't think so, then please update.

[colinmorris]

Regarding 3:

What I mean is that it seems to me like no special-casing is necessary. The function I pass to agg is acting on a Series - i.e. a 1-d array. The ordinary behaviour when passing a Series to a function like np.mean is already equivalent to calling that function with axis=0.

Could you give an example of how the behaviour would be different if numpy functions weren't special-cased?

[jorisvandenbossche]

2. is a violation of the guarantees (IOW, these functions don't take parameters). I suppose it could raise a TypeError, but this is tricky to actually do.

This is not very clearly documented (also not that it should work, I agree), but the args/kwargs are in the docstring and I think it is not strange to expect that they are passed (as is done for apply)

Further, this is any case not a violation for SeriesGroupBy (see the args/kwargs being passed in _python_agg_general

pandas/pandas/core/groupby.py

Line 825 in ba05744

f = lambda x: func(x, *args, **kwargs)

), and the following working example:

In [14]: s = pd.Series(np.arange(4))

In [15]: s.groupby(s % 2).agg(lambda group, div: group.sum()/div, div=1)
Out[15]: 
0    2
1    4
dtype: int64

In [16]: s.groupby(s % 2).agg(lambda group, div: group.sum()/div, div=2)
Out[16]: 
0    1
1    2
dtype: int64

3. not sure what you mean, the docs are clear. but again if you don't think so, then please update.

I agree with @colinmorris that the docs are not clear, now I look at it again. Or maybe even wrong.
np.mean(df) does not do a typical numpy mean (as in returning a scalar result), but does the same as df.mean() (because it calls it under the hood, I know). So in that light, passing it to .agg(np.mean), does exactly what it is expected to do.

[jorisvandenbossche]

What I mean is that it seems to me like no special-casing is necessary. The function I pass to agg is acting on a Series - i.e. a 1-d array. The ordinary behaviour when passing a Series to a function like np.mean is already equivalent to calling that function with axis=0.

@colinmorris I think you are right here. It's true that the passed function is first passed to each column and thus acts on a Series. In that sense, the function does nothing special.
And even if it would be called on the 2D dataframe, it would still not be really special cased as I mentioned above.
IMO this full note can be removed.

[jreback]

@jorisvandenbossche your example is not germane.

Further, this is any case not a violation for SeriesGroupBy (see the args/kwargs being passed in

that is NOT applicable to .transform

[jorisvandenbossche]

@jreback sorry, don't understand that comment. This issue is about agg?

[jorisvandenbossche]

@colinmorris I didn't read your point 2 very good first, as you already mention that the reason is that the group is changed from series to dataframe by passing kwarg. But note that this is not for 'meaningless' args. They need to be accepted by the function, otherwise you get a a TypeError as expected. To make it a bit more visual:

In [55]: df = pd.DataFrame(np.random.rand(4,2), columns=['b', 'c'])

In [56]: df['a'] = [1, 0, 0, 0]

In [57]: def func(group, mul=1):
    ...:     print(type(group))
    ...:     return np.product(group.shape) * mul

In [60]: g = df.groupby('a')

In [61]: g.agg(func)
<class 'pandas.core.series.Series'>
<class 'pandas.core.series.Series'>
<class 'pandas.core.series.Series'>
<class 'pandas.core.series.Series'>
Out[61]: 
     b    c
a          
0  3.0  3.0
1  1.0  1.0

In [62]: g.agg(func, mul=2)
<class 'pandas.core.frame.DataFrame'>
<class 'pandas.core.frame.DataFrame'>
Out[62]: 
    b   c
a        
0  18  18
1   6   6

In [63]: g.agg(func, foo=1)
...
TypeError: func() got an unexpected keyword argument 'foo'

But I agree that passing the kwarg or not should not change the passed group from series to frame.

@colinmorris BTW, thanks for your detailed feedback!