DEPR: Special casing of NumPy and Python builtin functions

[rhshadrach]

Currently various code paths detects whether a user passes e.g. Python's builtin sum or np.sum and replaces the operation with the pandas equivalent. I think we shouldn't alias a potentially valid UDF to the pandas version; this prevents users from using the NumPy version of the method if that is indeed what they want. This can lead to some surprising behavior: for example, NumPy ddof defaults to 0 whereas pandas defaults to 1. We also only do the switch when there are no args/kwargs provided. Thus:

df.groupby("a").agg(np.sum)  # Uses pandas sum with a numeric_only argument
df.groupby("a").agg(np.sum, numeric_only=True)  # Uses np.sum; raises since numeric_only is not an argument

It can also lead to different floating point behavior:

np.random.seed(26)
size = 100000
df = pd.DataFrame(
    {
        "a": 0,
        "b": np.random.random(size),
    }
)
gb = df.groupby("a")
print(gb.sum().iloc[0, 0])
# 50150.538337372185
print(df["b"].to_numpy().sum())
# 50150.53833737219

However, using the Python builtin functions on DataFrames can give some surprising behavior:

ser = pd.Series([1, 1, 2], name="a")
print(max(ser))
# 2

df = pd.DataFrame({'a': [1, 1, 2], 'b': [3, 4, 5]})
print(max(df))
# b

If we remove special casing of these builtins, then apply/agg/transform will also exhibit this behavior. In particular, whether a DataFrame is broken up into Series internally to compute the operation will determine whether we get a result like 2 or b above.

However, I think this is still okay to do as the change to users is straightforward: use "max" instead of max.

I've opened #53426 to see what impact this would have on our test suite.

cc @topper-123 @mroeschke @jbrockmendel

[topper-123]

A few questions to start the discussion:

• Will this emit warnings for a lot of current user code? I see that np.std will behave differently, but how about other numpy functions?
• Will supplying numpy functions work when the data is masked arrays and arrow arrays?

[rhshadrach]

Will this emit warnings for a lot of current user code? I np.std will behave differently, but how about other numpy functions?

I hope not - as a pandas user it had never occurred to me to pass np.sum or sum until I started contributing and seeing it in the test suite. That said, the process of either keeping current behavior or adopting future behavior is straight foward: (current) np.sum -> "sum" (future) np.sum -> lambda x: np.sum(x)

The OP of #53426 demonstrates different floating point results with np.sum. I believe that will occur across all functions. But also users doing ser.agg(np.sum, arg1) must be passing pandas arguments Update: currently we do not do the switch to pandas methods when arguments are provided.

Will supplying numpy functions work when the data is masked arrays and arrow arrays?

No - using the POC I get:

Edit: The example below fails because it uses np.random.random. See #53425 (comment).

Code

np.random.seed(26)
size = 100000
df = pd.DataFrame(
    {
        "a": 0,
        "b": np.random.random(size),
    },
    dtype="Int64",
)
df.apply(np.sum)

TypeError: cannot safely cast non-equivalent object to int64

[jbrockmendel]

I think #52042 means the DataFrameGroupBy cases here will be emitting FutureWarnings anyway, so affected users will need to update their code regardless.

[mroeschke]

I would be +1 for not doing this special casing

[rhshadrach]

I was a bit incorrect in #53425 (comment); I've updated that comment (see the crossed out line) and added this as another example of not-great current behavior to the OP.

[topper-123]

I think I can remember that this may have been done for performance reasons, i.e. the builtin methods are faster. Could you check on that?

[rhshadrach]

Yes - the builtin methods are quite a bit faster. But users can still use them by passing e.g. "sum" instead of sum or np.sum.

[topper-123]

No - using the POC I get:

I tried this:

>>> df = pd.DataFrame({"a": [1,1,1,2,2], "b": [4,5,6,4,pd.NA]}, dtype="int32[pyarrow]")
>>> df.agg(np.sum, axis=0)
a     7
b    19
dtype: int64

Seems to work after removing calls to com.get_cython_func. Can you explain how you'd get it to fail?

I think IK'm probably ok with it being deprecated though.

[rhshadrach]

Seems to work after removing calls to com.get_cython_func. Can you explain how you'd get it to fail?

It fails using the POC branch (#53426), not main.

Ack! My original example used np.random.random instead of np.random.randint! The failure I reported was on the DataFrame construction. Thanks for catching this.

[topper-123]

Cool 👍 .

[topper-123]

After this deprecation has been enforced: df.agg([np.sum]) will not aggregate, but transform. Also df.agg([np.sum, lambda x: np.sum(x)]) will fail.

Just a heads up that this needs to be fixed before v3.0. Implementing #54747 v2 (I will push today) will fix this though, but you should probably be aware of this just in case something stops #54747.

[rhshadrach]

@topper-123 - how are you concluding this? If I enforce the deprecation here as well as the one in #53325, then I get:

df = pd.DataFrame({'a': [1, 1, 2], 'b': [3, 4, 5]})
print(df.agg([np.sum]))
#      a   b
# sum  4  12
print(df.agg([np.sum, lambda x: np.sum(x)]))
#           a   b
# sum       4  12
# <lambda>  4  12

Am I doing something wrong?