A map method to work with many columns

[oricou]

Hi,

I would like to map a function expecting N arguments to a DataFrame of N columns. Today to do that I do:

df = pd.DataFrame({"A": [1, 2, 3, 4], "B": [5, 6, 7, 8]})
def f(x,y):
     return x*y

df[['A', 'B']].apply(lambda x: f(*x), axis=1) 

So the map method for DataFrame would be:

def map(self, f):
    return self.apply(lambda x: f(*x), axis=1)

if I am right. Then df.map(f) would produce a Serie.

Note that with 1 colummn and a 1 argument function map would work as numpy.map does.

Thanks,

Olivier.

ps: I edited this post after the talk I had with Tom, so it must seem strange now 😃

[TomAugspurger]

That would break compatibility.

[oricou]

What!? Is that a magic answer to close issues? I am sorry but you should explain how it would break compatibility and as far as I know breaking compatibility has been done before.

From my understanding all previous code will still work if you add this feature.

[TomAugspurger]

and as far as I know breaking compatibility has been done before.

Sorry, I posted before finishing my comment. It'd break compat and I don't see the benefit. I could be missing something though. Could you maybe explain a bit further what you're trying to achieve with a runnable example?

From my understanding all previous code will still work if you add this feature.

How would that be? Currently the function passed to f expects a column or row as it's first argument. With your proposal it would have to expect a positional argument for each row or column. Or am I misunderstanding?

[TomAugspurger]

For example, suppose we had

In [10]: df = pd.DataFrame({"A": [1, 2, 3, 4], "B": [5, 6, 7, 8]})

In [11]: def f(row):
    ...:     return row.A * row.B

IIUC (which I may not) your proposal would break the following

In [12]: df.apply(f, axis=1)
Out[12]:
0     5
1    12
2    21
3    32
dtype: int64

Since f would be called with two arguments instead of 1.

[oricou]

I want to apply a 2 arguments function to 2 columns of my DataFrame. As you said apply expect a vector (a column or a row) which means the function given to apply should have one argument. Imagine you have functions with N arguments and you want to map it on all the rows for the N columns you have selected. The best way I have found to do that is the use of a lambda function to call my function (as explained in my first post).

Now thinking about it, it may be more logic to use map instead of apply to do what I want. Map a function expecting N arguments to N columns would run row by row the function with the N values. It would behave the same than map with a 1 argument function applied to a Serie.

Example:

df = pd.DataFrame({'A':[1,2], 'B':[3,4]})
df.apply(lambda x,y: x+2*y, axis=1)   # does not work

# what about having?
df.map(lambda x,y: x+2*y)

Since map does not exist for DataFrame, you are not going to break anything.

[oricou]

About your example, apply would detect that f expect 1 argument and there it would behave as it does now.

[TomAugspurger]

Thanks for the clarification. Perhaps you could update your original post with your new proposal.

Since map does not exist for DataFrame, you are not going to break anything.

Indeed, this would be preferable. Then we'll have to decide whether the additional method is worth implementing, & maintaining. Especially since your workaround is relatively straightforward.

About your example, apply would detect that f expect 1 argument and there it would behave as it does now.

In general, we try to avoid magic behavior like this. It often leads to surprising / difficult to debug behavior in unanticipated cases.

[oricou]

ok, I update the original post

[oricou]

BTW I think my workaround is not that easy, it implies you know *x and understand apply and axis. Many users know how to use numpy.map and would understand df.map(f) but not beeing able to do my workaround.