ENH: add to cython groupby functions

[jreback]

• shift here and here
• cumsum
• cumprod

[jtratner]

@jreback can I take this? If it's for 0.13 you have some time and I'd be really interested in a chance to dig into more of the Cython internals.

[jreback]

go for it! you will need to add a function template in src/generate_code.py and add the appropriate places in groupby.py

[jtratner]

@jreback cool - thanks :) I'm looking forward to figuring all that out

[jtratner]

@jreback were you thinking this would cover time series shifting too?

[jreback]

yes the index type actually doesn't matter though its based on position shiftting (well the time-series stuff happens at a high level and is just translated to positions to move anyhow)

[jtratner]

@jreback - if it's already translate to positions to move, then that makes
it much simpler. Thanks!

On Tue, Jul 9, 2013 at 7:34 PM, jreback notifications@github.com wrote:

yes the index type actually doesn't matter though its based on position
shiftting (well the time-series stuff happens at a high level and is just
translated to positions to move anyhow)

—
Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/issues/4095#issuecomment-20712874
.

[jreback]

yep...look at pandas/core/frame/shift....

[jtratner]

I've let this slide... I will try to circle back to this when I have a chance, but if someone else wants to take it go go go

[iwschris]

I've started looking at this one.

[iwschris]

@jreback I have a good start here on shift, but I need a little guidance to make sure that I don't reinvent something that's already in place. There's a fair amount of logic in groupby.py that expects the output of a groupby op to be compressed. However shift and cumsum shouldn't compress their result. The output length will be the same as the input length.

Currently I've done this by creating a list of ops that won't compress (currently just shift and cumsum I think) and then everywhere that the shape is expected to get compressed, I've added logic to keep it from doing so. Is that the right approach or has this already been done with some other groupby op that I'm not aware of?

[jreback]

these are kind of like (and I think should be implemented like) transform ops

another example is fillna
these return a same sized object as the input

can u show a branch that u have so far?

[iwschris]

Alright, that helps. Let me get this cleaned up a little bit and I'll start a PR so that we can look at real code.

[chris-b1]

cumprod

[jreback]

closed by #10901

[Oleg-Krivosheev]

Looks like there might be a problem http://stackoverflow.com/questions/33903624/python-pandas-memoryerror-skiplist-failed-when-using-rolling-median-and-apply

[randomgambit]

@jreback sorry to revive this but in my dataframe, a simple

df['duration']=df.groupby('prop', sort=False).sale_dt.transform(lambda x: x.shift(-1)-x)

takes just forever. Is this just because the data is big, or this is related to the old problems that are mentioned here? Happy to help if I can!

[jreback]

anything with a lambda function will by definition be slow its basically a python loop.

but what you are doing is NOT a transform, which must return a scalar per group.

[jreback]

you probably want

df.sale_dt - df.groupby('prop').sale_dt.transform('shift', -1)

[randomgambit]

Thanks Jeff

but what you are doing is NOT a transform, which must return a scalar per group.

Wait but my understanding is that transformis either

• one scalar per group (I compute the mean of x for every group, I and I want that mean merged back to the original data),
• OR some variation of the same data (aka some transformation, such as standardizing the values in the dataframe by demeaning and dividing by the standard deviation). Have I missed something?

[jreback]

that's what I said
you can use apply if you want
but using a lambda will simply be slow always

[randomgambit]

got it thanks.

by the way df.sale_dt - df.groupby('prop').sale_dt.transform('shift', -1) is perfect

[randomgambit]

I think I will write a book soon:

the 100 most common errors every Pandas user has to make

bestseller on amazon for sure