Implement DataFrame.__array_ufunc__

[TomAugspurger]

For some cases, this will preserve extension types of arrays by calling
the ufunc blockwise.

In [1]: import pandas as pd; import numpy as np
In [2]: df = pd.DataFrame({"A": pd.array([0, 1], dtype="Sparse")})

In [3]: np.sin(df).dtypes
Out[3]:
A    Sparse[float64, nan]
dtype: object

Implementation-wise, this was done by moving Series.__array_ufunc__ to NDFrame and making it generic for Series / DataFrame. The DataFrame implementation goes through BlockManager.apply(ufunc).

We don't currently handle the multi-input case well for dataframes (aside from ufuncs that are implemented as dunder ops like np.add). For these, we fall back to the old implementation of converting to an ndarray and wrapping the result. This loses extension types.

We also don't currently handle multi-output ufuncs (like np.modf). This would require a BlockManager.apply that returns a Tuple[BlockManager], nout per input block. Maybe someday, but that's low priority.

closes #23743

[jreback]

lgtm some questions.

[jbrockmendel]

needs rebase, otherwise i think pretty ready

[jreback]

can you merge master

[jbrockmendel]

@TomAugspurger needs rebase, otherwise i think good to go

[jreback]

ideally can you split this function up a bit (if easy)

[jbrockmendel]

presrves -> preserves

[jreback]

can you update :->

[jbrockmendel]

if we were to implement __array_function__ (after dropping numpy 1.16 i guess) is that a "instead of" or an "in addition to" __array_ufunc__ thing?

does this make __array_wrap__ unnecessary?

[TomAugspurger]

__array_ufunc__ is separate from __array_function__. If there's ever a specialized protocol (like __array_ufunc__) it takes priority over __array_function__.

IIRC I tried removing __array_wrap__ but there may have been a test failure. Apparently it's used by things other than __array_ufunc__.

[jbrockmendel]

@TomAugspurger can you merge master? i tried doing it myself but am getting some really weird-looking errors

[TomAugspurger]

Gonna be a bit probably.

…

On Nov 20, 2020, at 2:02 PM, jbrockmendel ***@***.***> wrote: @TomAugspurger <https://github.com/TomAugspurger> can you merge master? i tried doing it myself but am getting some really weird-looking errors — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#36955 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOISEIYDAE3HVNRKCPODSQ3DMRANCNFSM4SHYJLYQ>.

[jreback]

ideally we could do this for 1.2 if can merge master

[jbrockmendel]

@jreback i merged master 2 days ago, this should be OK

[jreback]

ok 2 comments i think should do here (these are typos / doc-string). my comments about splitting up the actual function can be done later. ping on green.

[jreback]

can you update :->

[jreback]

thanks @TomAugspurger very nice

[TomAugspurger]

Thanks Brock.

[jorisvandenbossche]

The fact that we now align is a breaking change, but it seems one that could easily be detected (when the operands are not yet aligned) and done with a deprecation cycle?

[TomAugspurger]

Did we face a similar issue with Series implementing __array_ufunc__ in #23293? I don't see anything in the change log?

I vaguely recall a discussion around np.add(a, b) aligning, but, say, np.logaddexp(a, b) not, being considered a bug.

[jorisvandenbossche]

Yes, we did a similar change back then for Series I think, and it seems we didn't discuss it much at the time either. And I also don't recall (m)any issues about it that were raised after doing the change. Which is certainly a sign that this might be indeed OK to simply change.

I would say the main differences is that for Series we changed this in 0.25, but now for 1.x, we said to do changes using a deprecation cycle where possible (and this seems a case where it is easily possible)

[TomAugspurger]

Right, the 1.0 vs. 0.25 distinction is the important part. I'd prefer to see a deprecation warning for the alignment, but I don't know if I'll have time to do it for this release.

One complication just noticed: there isn't an easy way to silence the warning, like with a keyword. We'd need to only warn when the inputs are unaligned, and require users to call .align() ahead of time.

[jorisvandenbossche]

One complication just noticed: there isn't an easy way to silence the warning, like with a keyword. We'd need to only warn when the inputs are unaligned, and require users to call .align() ahead of time.

Indeed, aligning manually would be the way to get around the warning if you want the new behaviour, otherwise the solution is to first convert to numpy arrays before calling the ufunc, in case you want the old non-aligning behaviour.