ENH: Faster grouped rolling and expanding operations in the pandas backend #1549

cpcloud · 2018-07-24T03:58:56Z

Also improves support for time series based rolling and expanding operations
Includes behind the scenes handling of indexes, so that operations
that need them get them, but users never have to interact with them.
Includes a performance improvement for grouped and ordered operations
Forces joins to be materialized (this is a bugfix)
~~Calls apply_to on results in the pandas backend~~ This is complicated by the use of nan as a missing value. Operations whose output type is nullable integral, will produce a float64 array in the presence of nans. We have to think about to handle this in a reasonable way.

kszucs · 2018-07-24T14:04:46Z

@cpcloud should I review?

cpcloud · 2018-07-24T14:07:17Z

@kszucs Please do, whenever you have a chance.

This reverts commit bf3f3d9.

kszucs · 2018-07-24T14:22:51Z

ibis/expr/types.py

@@ -731,6 +731,9 @@ class SortExpr(Expr):
    def _type_display(self):
        return 'array-sort'

+    def get_name(self):


Is this name propagation only used for Sort expressions? It suggests that somewhere We don't have a clear distinction between expressions and operations.

No, get_name is a method on Expr.

self.op().expr.get_name() in this case

I'll change to not use resolve_name and make an issue to move resolve_name into Node (instead of just in ValueOp where it is now).

Created #1552 to track this.

kszucs · 2018-07-24T14:26:17Z

ibis/pandas/aggcontext.py

    return lambda data, function=function, args=args, kwargs=kwargs: (
        function(data, *args, **kwargs)
    )


 class Summarize(AggregationContext):
-
    __slots__ = ()


AFAIK __slots__ are not inherited, must be defined in each child class

Yep, I believe that is the case in this module.

kszucs · 2018-07-24T14:30:31Z

ibis/pandas/aggcontext.py

-                    type(self).__name__
+        group_by = self.group_by
+
+        if not group_by:


Could You use a couple of one liner comments? There are four, not necessarily straightforward agg branches.

kszucs · 2018-07-24T14:37:04Z

ibis/pandas/aggcontext.py

+        keys = group_by + order_by
+        frame = self.parent.obj
+        name = grouped_data.obj.name
+        indexed_series = frame[keys + [name]].set_index(


This block is pretty terse, comments? :)

kszucs · 2018-07-24T14:42:26Z

ibis/pandas/client.py

@@ -360,7 +360,13 @@ def execute(self, query, params=None, limit='default', async=False):
            )

        assert isinstance(query, ir.Expr)
-        return execute(query, params=params)
+        result = execute(query, params=params)
+        if isinstance(result, pd.DataFrame):


A comment about removing the indices?

kszucs · 2018-07-24T14:42:43Z

ibis/pandas/client.py

+        result = execute(query, params=params)
+        if isinstance(result, pd.DataFrame):
+            schema = query.schema()
+            return result.reset_index()[schema.names]


Schema.apply_to()?

I reverted this commit because it introduce a whole host of other issues related to casting and nullability of integer data (pandas casts integers with null values to float64)

kszucs · 2018-07-24T14:45:07Z

ibis/pandas/decimal.py

@@ -52,7 +52,7 @@ def execute_decimal_log2(op, data, **kwargs):
        return decimal.Decimal('NaN')


-@execute_node.register(ops.UnaryOp, decimal.Decimal)
+@execute_node.register((ops.UnaryOp, ops.Negate), decimal.Decimal)


Isn't Negate a UnaryOp?

Yep. I was playing around with some performance improvements related to multipledispatch 0.5.0, but they aren't strictly necessary. I'll remove this.

I mean We should prefer the dict lookups, just could mention it in a comment or at the top of the modules. It's more explicit too.

kszucs · 2018-07-24T14:49:10Z

ibis/pandas/execution/generic.py

@@ -139,17 +150,33 @@ def execute_cast_series_date(op, data, type, **kwargs):
    raise TypeError("Don't know how to cast {} to {}".format(from_type, type))


+@execute_node.register(ops.SortKey, pd.Series, bool)
+def execute_sort_key_series_bool(op, data, ascending, **kwargs):
+    return data


Is it supposed to actually sort the series?

No. This dispatch exists so that I don't have to special case in ibis/pandas/execution/util.py.

kszucs · 2018-07-24T15:05:25Z

ibis/pandas/execution/generic.py

+@execute_node.register(
+    (ops.Comparison, ops.Add, ops.Multiply), six.string_types, pd.Series)
+@execute_node.register(
+    (ops.Comparison, ops.Add), six.string_types, six.string_types)


Nice set of rules!

kszucs · 2018-07-24T15:18:27Z

ibis/pandas/execution/temporal.py

@@ -73,31 +74,31 @@ def convert_to_offset(n):
    return data.apply(convert_to_offset)


-@execute_node.register(ops.TimestampAdd, datetime.datetime, datetime.timedelta)
+@execute_node.register(ops.TimestampAdd, timestamp_types, timedelta_types)


If the timestamp and timedelta literals are properly converted to pd.Timestamp and pd.Timedelta beforehand, then defining an execute_node for BinaryOp (parent of all temporal binary ops) might be enough - not sure though.

Yep, this was related to the perf optimization. I can remove it.

kszucs · 2018-07-24T15:23:12Z

ibis/pandas/execution/tests/test_window.py

+    expected = df[['plain_datetimes_naive', 'dup_strings']].set_index(
+        'plain_datetimes_naive').squeeze().tshift(
+                freq=execute(-range_offset)).reindex(
+                df.plain_datetimes_naive).reset_index(drop=True)


Multiple expressions?

kszucs · 2018-07-24T15:27:12Z

ibis/pandas/execution/util.py

    try:
+        if isinstance(by, six.string_types):


How can it be a string?

If it's a group by key. Sorting group by and order by keys before is the key thing here that makes this much faster.

kszucs · 2018-07-24T15:32:18Z

ibis/pandas/execution/window.py

@@ -104,28 +122,51 @@ def execute_window_op(op, data, window, scope=None, context=None, **kwargs):
        factory=OrderedDict,
    )

+    # figure out what the dtype of the operand is
+    operand_type = operand.type()
+    if isinstance(operand_type, dt.Integer) and operand_type.nullable:


Perhaps it's the responsibility of to_pandas()

The problem there is that the dtypes would have to know about the operations. One option is to convert all nullable integer columns to float64, but that seems really disgusting.

This reverts commit b63936d.

…d or not

cpcloud added 5 commits July 23, 2018 23:41

ENH: Faster moving and better support for time series

5e482e2

Clean up test

e6af6d8

Give SortExpr and SortKey a name

7906a34

Make SortKey trivially executable

7519bf9

Move name

6273ae9

cpcloud added this to the 0.14 milestone Jul 24, 2018

cpcloud added feature Features or general enhancements expressions Issues or PRs related to the expression API pandas The pandas backend performance Issues related to ibis's performance labels Jul 24, 2018

Need eager ordering for datatypes module for the dtype function

3a03ea4

cpcloud added 3 commits July 24, 2018 10:10

Make sure we get the name from both groupby and non group by objects

805524d

Try without reordering levels

bf3f3d9

Revert "Try without reordering levels"

732ee3b

This reverts commit bf3f3d9.

kszucs reviewed Jul 24, 2018

View reviewed changes

Remove extraneous index levels not needed by the input

4f8351d

kszucs reviewed Jul 24, 2018

View reviewed changes

cpcloud added 5 commits July 25, 2018 10:02

Dup execute_node dispatch registration

ef3a31e

Fixed width and timedelta only for negate

85150bd

Consolidate array collect implementations

0fc7b13

Do not reorder levels if we do not need to

05a0734

Add failing array collect rolling test

291bb61

cpcloud mentioned this pull request Jul 25, 2018

Move resolve_name into Node #1552

Closed

cpcloud added 16 commits July 25, 2018 11:42

Comments about why we are not using UnaryOp

ba0c3a3

Aggcontext comments

ad6a957

Apply to with schema

b63936d

Move decimal to execution module

5c0bf8e

More decimal moves

7317c95

Revert "Apply to with schema"

4cdc7f2

This reverts commit b63936d.

Remove whitespace

3d330a9

Not true if we are calling cumsum etc

c117848

Comments about time shifting

7e52bd9

Fix comment

cbb5287

Add grouped rolling benchmarks

641117a

Use standard pandas

91b0756

Fix asv deps

e0fbb54

Fix tests

7ddfebf

Do not reindex

5f6f96c

Reset indices on pieces with a multiindex

285a589

cpcloud force-pushed the faster-moving branch from 04a107d to 285a589 Compare July 27, 2018 16:01

cpcloud added 4 commits July 27, 2018 16:45

Deal with windows missing things

dd7c9dc

Fix error message

d42f662

Reuse expressions where possible in selections

617f485

Make sure we have then name regardless of whether the data are groupe…

4c0cbb1

…d or not

cpcloud closed this in 9624af8 Jul 29, 2018

cpcloud deleted the faster-moving branch July 29, 2018 21:50

cpcloud mentioned this pull request Jul 30, 2018

ENH/REF: Use post_execute on Series/DataFrame to call reset_index #1446

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Faster grouped rolling and expanding operations in the pandas backend #1549

ENH: Faster grouped rolling and expanding operations in the pandas backend #1549

cpcloud commented Jul 24, 2018 •

edited

Loading

kszucs commented Jul 24, 2018

cpcloud commented Jul 24, 2018

kszucs Jul 24, 2018

cpcloud Jul 24, 2018

kszucs Jul 24, 2018

cpcloud Jul 25, 2018

cpcloud Jul 25, 2018

kszucs Jul 24, 2018

cpcloud Jul 24, 2018

kszucs Jul 24, 2018 •

edited

Loading

cpcloud Jul 25, 2018

kszucs Jul 24, 2018

cpcloud Jul 25, 2018

kszucs Jul 24, 2018

cpcloud Jul 25, 2018

kszucs Jul 24, 2018

cpcloud Jul 25, 2018

cpcloud Jul 25, 2018

kszucs Jul 24, 2018

cpcloud Jul 24, 2018

kszucs Jul 25, 2018 •

edited

Loading

kszucs Jul 24, 2018

cpcloud Jul 24, 2018

kszucs Jul 24, 2018

kszucs Jul 24, 2018

cpcloud Jul 24, 2018

kszucs Jul 24, 2018

kszucs Jul 24, 2018

cpcloud Jul 25, 2018

kszucs Jul 24, 2018

cpcloud Jul 25, 2018

ENH: Faster grouped rolling and expanding operations in the pandas backend #1549

ENH: Faster grouped rolling and expanding operations in the pandas backend #1549

Conversation

cpcloud commented Jul 24, 2018 • edited Loading

kszucs commented Jul 24, 2018

cpcloud commented Jul 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kszucs Jul 24, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kszucs Jul 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud commented Jul 24, 2018 •

edited

Loading

kszucs Jul 24, 2018 •

edited

Loading

kszucs Jul 25, 2018 •

edited

Loading