API: New callable indexing makes storing functions in a Series difficult

[evanpw]

Example of something that previously worked (before 7bbd031):

>>> s = pd.Series([lambda x: x] * 10)
>>> s[s.index > 5] = lambda x: x + 1

But now the second line tries to apply the function in the rhs to the elements of the Series, rather than assigning them (and throws an exception). This is very counter-intuitive when using __setitem__ rather than calling Series.where directly.

[jreback]

this is quite odd to store callables in a Series

what is the purpose of doing that?

[jreback]

hmm, this is simple to fix (we just don't evaluate a callable for other in .where).

@sinhrks do we have a valid use of other as a callable?

[evanpw]

We're choosing a function based on some columns in a DataFrame, and then applying them on other column values / external data. Since there might be a different function for each row, it's convenient to store them in the same place.

[jreback]

@evanpw I think you might be able to just remove the _maybe_callable(other...) and this will work. Not really sure of the original purpose of testing for a callable on the other (e.g. it works, but IMHO not that useful).

If you want to give this a shot and see what it looks like would be great.

[evanpw]

Yes, deleting this line from where in generic.py fixes the problem for me:

other = com._apply_if_callable(other, self)

[sinhrks]

The usage in my mind was chaining like:

pd.DataFrame({'A': [1, 2, 1, 2], 'B': [1, 2, 3, 4]}).groupby('A').sum().where(lambda x: x < 5, lambda x: -x)
#    B
# A   
# 1  4
# 2 -6

I think it is more likely rather than storing callable.

CC: @TomAugspurger

[evanpw]

I think that makes sense when calling where directly. What about making it configurable, and turned off when using __setitem__?

[jreback]

yes that is a bug I think.