BUG: passing kwargs to Series.aggregate raises TypeError: f() got an unexpected keyword argument

[jakeanq]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas. The bug is present on 1.3.0, 1.3.1 and 1.3.2 but not 1.2.5.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

#!/usr/bin/env python

import pandas as pd

def func(a, b, c=2):
    pass

if __name__ == "__main__":
    s = pd.Series([1, 2, 3, 4])

    s.aggregate(func, 0, 99, c=101)

Problem description

Prior to Pandas 1.3.0, it was possible to pass kwargs when using Series.aggregate, however it appears that Pandas now attempts to apply the arguments and keyword arguments twice (first currying them then using them to call the curried function).

Expected Output

Expect output is the program running normally and not throwing an exception.

Actual Output

Traceback (most recent call last):
  File "/home/jake/pandas-env/lib/python3.9/site-packages/pandas/core/apply.py", line 1068, in agg
    result = self.obj.apply(f, *args, **kwargs)
  File "/home/jake/pandas-env/lib/python3.9/site-packages/pandas/core/series.py", line 4357, in apply
    return SeriesApply(self, func, convert_dtype, args, kwargs).apply()
  File "/home/jake/pandas-env/lib/python3.9/site-packages/pandas/core/apply.py", line 1043, in apply
    return self.apply_standard()
  File "/home/jake/pandas-env/lib/python3.9/site-packages/pandas/core/apply.py", line 1099, in apply_standard
    mapped = lib.map_infer(
  File "pandas/_libs/lib.pyx", line 2859, in pandas._libs.lib.map_infer
  File "/home/jake/pandas-env/lib/python3.9/site-packages/pandas/core/apply.py", line 131, in f
    return func(x, *args, **kwargs)
TypeError: f() got an unexpected keyword argument 'c'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jake/./pd_test.py", line 11, in <module>
    s.aggregate(func, 0, 99, c=101)
  File "/home/jake/pandas-env/lib/python3.9/site-packages/pandas/core/series.py", line 4227, in aggregate
    result = op.agg()
  File "/home/jake/pandas-env/lib/python3.9/site-packages/pandas/core/apply.py", line 1070, in agg
    result = f(self.obj, *args, **kwargs)
TypeError: f() got an unexpected keyword argument 'c'

Output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit           : 5f648bf1706dd75a9ca0d29f26eadfbb595fe52b
python           : 3.9.6.final.0
python-bits      : 64
OS               : Linux
OS-release       : 5.13.5-arch1-1
Version          : #1 SMP PREEMPT Sun, 25 Jul 2021 18:02:58 +0000
machine          : x86_64
processor        : 
byteorder        : little
LC_ALL           : None
LANG             : en_AU.UTF-8
LOCALE           : en_AU.UTF-8

pandas           : 1.3.2
numpy            : 1.21.2
pytz             : 2021.1
dateutil         : 2.8.2
pip              : 21.1.3
setuptools       : 56.0.0
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : None
IPython          : None
pandas_datareader: None
bs4              : None
bottleneck       : None
fsspec           : None
fastparquet      : None
gcsfs            : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pyxlsb           : None
s3fs             : None
scipy            : None
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
numba            : None

[simonjayhawkins]

Thanks @jakeanq for the report. It appears that the function is now applied to each value instead of the Series as a whole. (i.e. applied along the wrong axis, The axis keyword is for compatibility with DataFrame and axis=1 raises an error)

first bad commit: [fad5fee] CLN: Move the rest of Series.apply into apply (#39954)

cc @rhshadrach

---

code sample

import pandas as pd

print(pd.__version__)


def func(a, b):
    print(f"func called with {a}")
    return b


s = pd.Series([1, 2, 3, 4])
result = s.aggregate(func, 0, "foo")
print(result)

on 1.2.5

1.2.5
func called with 0    1
1    2
2    3
3    4
dtype: int64
foo

on 1.3.2

1.3.2
func called with 1
func called with 2
func called with 3
func called with 4
0    foo
1    foo
2    foo
3    foo
dtype: object

[rhshadrach]

It appears that the function is now applied to each value instead of the Series as a whole. (i.e. applied along the wrong axis, The axis keyword is for compatibility with DataFrame and axis=1 raises an error)

Series.agg will first try row-by-row using apply. If this fails, it will fallback to aggregating the entire Series. I believe this is the behavior we need to maintain (but am currently working on removing it for future versions).

[rhshadrach]

Expanding on my previous comment, the example @simonjayhawkins posted is failing in the attempt to use apply because of a bug. The call is:

result = self.apply(func, *args, **kwargs)

But apply accepts the argument convert_dtype before *args. As a result, b in the example is unspecified and fails, so agg fallback to f(self, *args, **kwargs) where self is the Series in question. You can see this by specifying a default to b or calling it as a kwarg:

def func(a, b="foo"):
    print(f"func called with {a}")
    return b


s = pd.Series([1, 2, 3, 4])
result = s.aggregate(func, 0, "foo")
print(result)

def func(a, b):
    print(f"func called with {a}")
    return b


s = pd.Series([1, 2, 3, 4])
result = s.aggregate(func, 0, b="foo")
print(result)

Both of these produce on 1.2.x:

func called with 1
func called with 2
func called with 3
func called with 4
0    foo
1    foo
2    foo
3    foo
dtype: object

While it's possible to preserve the erroneous convert_dtype behavior, I think it's reasonable to fix both the regression as well as the convert_dtype bug for 1.3.x.

[simonjayhawkins]

While it's possible to preserve the erroneous convert_dtype behavior, I think it's reasonable to fix both the regression as well as the convert_dtype bug for 1.3.x.

Thanks @rhshadrach for investigating this further. My code sample should maybe have been a separate issue.

using semver, we can indeed also fix bugs on 1.3.x, especially if it makes sense to fix these issues together.