ENH: Limited window function support for pandas #1083

cpcloud · 2017-07-31T19:31:35Z

No description provided.

cpcloud · 2017-08-05T20:54:00Z

ibis/sql/postgres/compiler.py

+    ops.LastValue: fixed_arity(sa.func.first_value, 1),
+    ops.NthValue: fixed_arity(sa.func.nth_value, 2),
+    ops.Lead: fixed_arity(sa.func.lead, 1),
+    ops.Lag: fixed_arity(sa.func.lag, 1),


I'm going to revert these changes because there are no tests. I'll add in a follow up PR.

cpcloud · 2017-08-08T00:14:31Z

@wesm can you review this when you get a chance? most important things here are the changes in ibis/pandas/execution.py and ibis/pandas/tests/test_operations.py.

cpcloud · 2017-08-08T00:17:52Z

ibis/pandas/execution.py

 @execute_node.register(ops.Selection, pd.DataFrame)
-def execute_selection_dataframe(op, data, scope=None):
+def execute_selection_dataframe(op, data, scope=None, **kwargs):


I need to refactor this function a bit after this PR is merged. It's getting a bit too big for its britches.

wesm · 2017-08-08T00:59:52Z

Wow, big PR! I will take some time to review, am going to try to make a little more progress on the remaining Arrow 0.6.0 issues, and then I can get to this

cpcloud · 2017-08-09T00:49:15Z

One thing to note here is that for the groupby.transform style operations (OVER (PARTITION BY) in SQL parlance), we execute this in the fastest way possible: by executing each transform separately, rather than executing the operation per group.

Take z-score for example. There are two ways we could implement this.

1. Call `execute` in a `lambda` that executes the operation on each group.

df.groupby('foo').bar.transform(lambda x: (x - x.mean()) / x.std())

This will be very slow for high cardinality grouping keys, since pandas can't know that our lambda is composed of a bunch of function calls that are very fast when done individually. Also, the left operand of the subtraction in this case can be taken from the parent DataFrame. This avoids any indexing that happens when doing groupby.apply.

2. Execute each aggregation in a separate transform.

gb = df.groupby('foo').bar
result = (df.x - gb.transform('mean')) / gb.transform('std')

This can be up to 500x (!) faster than the former for certain operations.

This patch always uses the latter when executing any non trivial operations using t.groupby().mutate() expressions.

cpcloud · 2017-08-10T16:31:43Z

GCS keeps intermittently failing, rebuilding.

cpcloud · 2017-08-10T17:07:10Z

I'm going to pluck out the scope containing op instead of expr changes here

cpcloud · 2017-08-15T00:29:11Z

ibis/pandas/tests/conftest.py

+def batting_df():
+    path = os.environ.get('BATTING_CSV', 'batting.csv')
+    df = pd.read_csv(path, index_col=None, sep=',')
+    five_percent = int(0.01 * len(df))


This should be renamed

cpcloud · 2017-08-15T00:29:33Z

ci/pgload.sql

@@ -0,0 +1,69 @@
+CREATE EXTENSION IF NOT EXISTS file_fdw;


This whole file should be removed

cpcloud · 2017-08-15T00:33:08Z

ibis/pandas/execution.py

+        raise ValueError(
+            'More than one operation name found in {} class'.format(typename)
+        )
+    return getattr(data.expanding(), operation_name.lower())()


We can use a context here to handle this attribute stuff

cpcloud · 2017-08-15T14:17:32Z

ibis/pandas/execution.py

+
+@execute_node.register(ops.StandardDev, SeriesGroupBy, SeriesGroupBy)
+def execute_reduction_series_groupby_mask_std(op, data, mask, **kwargs):
+    return data.apply(lambda x, mask=mask.obj: x[mask[x.index]].std())


These both should use context.

cpcloud · 2017-08-15T14:17:58Z

ibis/pandas/execution.py

+
+@execute_node.register(ops.CumulativeMax, pd.Series)
+def execute_series_cummax(op, data, **kwargs):
+    return data.cummax()


Need to add a test to make sure these do or don't work on SeriesGroupBy as well.

wesm · 2017-08-15T15:18:34Z

Reviewing this now. Sorry about the delay

wesm · 2017-08-15T16:04:44Z

ibis/pandas/context.py

+
+
+@six.add_metaclass(abc.ABCMeta)
+class Context(object):


I'm a bit confused about terminology. What is Context, maybe WindowType?

Yeah this isn't named very well.

The idea this encapsulates is really something like AggregationContext.

For any particular aggregation such as sum, mean, etc we need to decide based on the presence or absence of other expressions like group_by and order_by whether we should call groupby(...).transform, groupby(...).apply, groupby(...).expanding().<method>, groupby(...).rolling().<method> or just groupby(...).<method> on the data.

Gotcha. The name doesn't matter too much for now as long as it's explained what the thing is.

I'll add documentation to explain this module and it's classes

wesm · 2017-08-15T17:22:35Z

ibis/pandas/tests/test_operations.py

+    result = expr.execute()
+
+    columns = ['G', 'yearID']
+    more_values = batting_df[columns].sort_values('yearID').G.rolling(5).sum()


Do you think this will be able to easily handle temporal expressions like "5 days" here?

I think we can extend ibis.window to accept time intervals (probably implemented as an interval type) which would be translated to pandas objects and passed into rolling which accepts offsets as of 0.19.0

wesm · 2017-08-15T17:31:22Z

LGTM modulo comment about internal terminology

cpcloud · 2017-08-15T19:35:50Z

Merging on green.

cpcloud self-assigned this Jul 31, 2017

cpcloud added feature Features or general enhancements pandas The pandas backend labels Jul 31, 2017

cpcloud added this to the 0.11.3 milestone Jul 31, 2017

cpcloud force-pushed the window-funcs branch 2 times, most recently from 46e09db to f5f5837 Compare August 4, 2017 18:38

cpcloud changed the title ~~WIP: Pandas window functions~~ ENH: Limited window function support for pandas Aug 4, 2017

cpcloud force-pushed the window-funcs branch 3 times, most recently from 7953f4c to 9b41812 Compare August 5, 2017 02:42

cpcloud mentioned this pull request Aug 5, 2017

REF/TST: Refactor sqlite tests to use functions #1084

Closed

cpcloud force-pushed the window-funcs branch 5 times, most recently from 0b97191 to 935ff51 Compare August 5, 2017 20:53

cpcloud commented Aug 5, 2017

View reviewed changes

cpcloud force-pushed the window-funcs branch 13 times, most recently from 1b2b77c to 77a446b Compare August 7, 2017 23:40

cpcloud commented Aug 8, 2017

View reviewed changes

cpcloud force-pushed the window-funcs branch from 77a446b to 5bfe3c5 Compare August 8, 2017 22:29

cpcloud force-pushed the window-funcs branch 2 times, most recently from b625bdf to 1404f92 Compare August 10, 2017 06:25

cpcloud mentioned this pull request Aug 10, 2017

ENH: Pass ops in scope instead of exprs #1109

Closed

cpcloud force-pushed the window-funcs branch 7 times, most recently from aacb757 to 0831a4f Compare August 15, 2017 14:10

cpcloud commented Aug 15, 2017

View reviewed changes

wesm reviewed Aug 15, 2017

View reviewed changes

ENH: Limited window function support for pandas

10cf761

cpcloud force-pushed the window-funcs branch from 02ee5a0 to 10cf761 Compare August 15, 2017 19:27

cpcloud closed this in e7d6353 Aug 15, 2017

cpcloud deleted the window-funcs branch August 15, 2017 19:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Limited window function support for pandas #1083

ENH: Limited window function support for pandas #1083

cpcloud commented Jul 31, 2017

cpcloud Aug 5, 2017 •

edited

Loading

cpcloud commented Aug 8, 2017

cpcloud Aug 8, 2017

wesm commented Aug 8, 2017

cpcloud commented Aug 9, 2017

cpcloud commented Aug 10, 2017

cpcloud commented Aug 10, 2017

cpcloud Aug 15, 2017

cpcloud Aug 15, 2017

cpcloud Aug 15, 2017

cpcloud Aug 15, 2017

cpcloud Aug 15, 2017

wesm commented Aug 15, 2017

wesm Aug 15, 2017

cpcloud Aug 15, 2017

wesm Aug 15, 2017

cpcloud Aug 15, 2017

wesm Aug 15, 2017

wesm Aug 15, 2017

cpcloud Aug 15, 2017 •

edited

Loading

wesm commented Aug 15, 2017

cpcloud commented Aug 15, 2017



		@six.add_metaclass(abc.ABCMeta)
		class Context(object):

ENH: Limited window function support for pandas #1083

ENH: Limited window function support for pandas #1083

Conversation

cpcloud commented Jul 31, 2017

cpcloud Aug 5, 2017 • edited Loading

Choose a reason for hiding this comment

cpcloud commented Aug 8, 2017

Choose a reason for hiding this comment

wesm commented Aug 8, 2017

cpcloud commented Aug 9, 2017

1. Call execute in a lambda that executes the operation on each group.

2. Execute each aggregation in a separate transform.

cpcloud commented Aug 10, 2017

cpcloud commented Aug 10, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wesm commented Aug 15, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud Aug 15, 2017 • edited Loading

Choose a reason for hiding this comment

wesm commented Aug 15, 2017

cpcloud commented Aug 15, 2017

cpcloud Aug 5, 2017 •

edited

Loading

1. Call `execute` in a `lambda` that executes the operation on each group.

cpcloud Aug 15, 2017 •

edited

Loading