ENH/DEV: Make ibis Node instances hashable #1611

cpcloud · 2018-09-05T20:24:23Z

Closes #890

cpcloud · 2018-09-05T20:28:56Z

ibis/sql/compiler.py

            return False

        this_exprs = self._all_exprs()
        other_exprs = other._all_exprs()

        if self.limit != other.limit:
-            cache[(self, other)] = False
+            cache[self, other] = False
            return False

        for x, y in zip(this_exprs, other_exprs):


This needs to check the length of this_exprs and other_exprs otherwise we'll hit the same bug we have for equality.

kszucs · 2018-09-05T22:20:11Z

ibis/expr/operations.py

            return False

        if len(self.args) != len(other.args):
-            cache[(self, other)] = False
+            cache[self, other] = False
            return False

        for left, right in zip(self.args, other.args):


You could use simply:

all_equal(self.args, other.args, cache=cache)

But the length of the arguments needs to be checked here as well, because map zips its arguments.

The equality PR will address this length issue. I will refactor this and the code near your other comment to use all_equal once that is PR is merged.

ibis/expr/visualize.py

ibis/pandas/execution/selection.py

ibis/sql/compiler.py

kszucs · 2018-09-05T22:39:45Z

ibis/sql/compiler.py


-        cache[(self, other)] = True
-        return True
+        cache[self, other] = len(this_exprs) == len(other_exprs) and all(


Use all_equals instead?

Will address after #1600.

kszucs

The code got much much cleaner and better!

We were incorrectly using weakref.WeakValueDictionary to hold temp table instances that mapped to themselves. This defeats the purpose of using a weakref container because the WeakValueDictionary will hold a strong reference to the key and weak reference to the value. In this scenario we will forever hold at least one strong reference per key and leak memory. By using weakref.WeakSet we now only weak references on the table instance.

cpcloud · 2018-09-07T17:42:29Z

@kszucs arg it looks like there was a backwards incompatible change in nbconvert from 5.3.1 to 5.4.0 :(

ibis/impala/client.py

cpcloud · 2018-09-10T17:29:51Z

@kszucs Can you review this again?

kszucs · 2018-09-10T17:33:55Z

Sure, reviewing now

kszucs · 2018-09-10T17:43:04Z

ibis/expr/analysis.py

-    substitutor = Substitutor()
-    return substitutor.substitute(expr, mapping)
+    """Substitute subexpressions in `expr` with expression to expression
+    mapping `substitutions`.


I got sweaty. Surprisingly rhythmical and symmetric sentence :)

Yes, now that you've pointed it out it's making my head hurt.

cpcloud · 2018-09-10T17:45:00Z

There's a large performance regression introduced by this PR that I'm looking into.

kszucs · 2018-09-10T17:50:51Z

ibis/util.py

@@ -291,3 +249,36 @@ def get_logger(name, level=None, format=None, propagate=False):
            logging, os.environ.get('LOGLEVEL', 'WARNING').upper()))
    logger.addHandler(handler)
    return logger
+
+
+def safe_get_name(expr):


IMO these would be nicer as Expr methods.

kszucs · 2018-09-10T17:52:44Z

ibis/expr/analysis.py

+    ----------
+    expr : ibis.expr.types.Expr
+        An Ibis expression
+    substitutions : Mapping[ibis.expr.types.Expr, ibis.expr.types.Expr]


Based on the dict comprehension below, substitutions seems like a list of 2-tuples.

kszucs · 2018-09-10T17:54:01Z

ibis/expr/analysis.py

@@ -61,38 +60,38 @@ def _substitute(self, expr, mapping):
        Parameters
        ----------
        expr : ibis.expr.types.Expr
-        mapping : Dict, OrderedDict
+        mapping : Mapping[ibis.expr.types.Expr, ibis.expr.types.Expr]


Mapping[ibis.expr.types.Node, ibis.expr.types.Expr]?

kszucs · 2018-09-10T17:57:47Z

ibis/expr/analysis.py

+            unchanged = True
+            for i, arg in enumerate(new_args):
+                if isinstance(arg, ir.Expr):
+                    new_arg = self.substitute(arg, mapping)


This recursion seems like a task for lin.traverse. Issue?

kszucs · 2018-09-10T18:12:54Z

ibis/expr/operations.py

+            self._hash = hash(
+                (type(self),) + tuple(
+                    element.op() if isinstance(element, ir.Expr) else element
+                    for element in self.flat_args()


We should get rid of flat_args and prevent having any non-expressions in the hierarchy, except the literal leaves.

cpcloud · 2018-09-10T19:14:47Z

Ok, figured out what perf problem is: I wasn't short circuiting with self is other in Node.equals.

cpcloud · 2018-09-10T19:42:42Z

@kszucs Fixed the docs, and added those functions as private methods.

cpcloud added this to the Next Release milestone Sep 5, 2018

cpcloud added feature Features or general enhancements expressions Issues or PRs related to the expression API labels Sep 5, 2018

cpcloud commented Sep 5, 2018

View reviewed changes

kszucs reviewed Sep 5, 2018

View reviewed changes

ibis/expr/visualize.py Outdated Show resolved Hide resolved

kszucs reviewed Sep 5, 2018

View reviewed changes

ibis/pandas/execution/selection.py Show resolved Hide resolved

kszucs reviewed Sep 5, 2018

View reviewed changes

ibis/sql/compiler.py Show resolved Hide resolved

kszucs reviewed Sep 5, 2018

View reviewed changes

kszucs requested changes Sep 5, 2018

View reviewed changes

cpcloud added 5 commits September 6, 2018 17:13

ENH/DEV: Make ibis Node instances hashable

d184eab

remove irrelevant comment

a31b1e3

Do not use key=id anymore

be9b1e7

Clean up pandas code and remove use of id

c4aac06

Clean up equality of Select DML

1c3db1b

cpcloud force-pushed the hashable-ops branch from 42cbdc0 to 1c3db1b Compare September 6, 2018 21:13

cpcloud added 9 commits September 6, 2018 17:20

Clean up equality checking code

c46ea0a

Bug

2376ea1

Redefine __nonzero__ in ListExpr for python2

0c3959f

Fix docs

b56f1b0

Add hashing benchmarks

d52f129

Try to build the docs in sequence rather than parallel

ac46f4e

Show the conda env used to build docs

55c55b5

Show docs

e0c76a5

cpcloud commented Sep 7, 2018

View reviewed changes

ibis/impala/client.py Show resolved Hide resolved

cpcloud added 3 commits September 10, 2018 10:19

Remove py27 benchmarks

5327f62

Quote benchmarks

e60e0dd

Add hashing benchmarks

c96ccaf

cpcloud added 5 commits September 10, 2018 10:21

Remove format.py license

d058330

Use expr_key

03ced57

Move functions around

7d293a0

Use util.expr_key where possible

7256641

Cleanup

a1b02e9

cpcloud force-pushed the hashable-ops branch from c0e67b7 to a1b02e9 Compare September 10, 2018 14:29

cpcloud added 2 commits September 10, 2018 10:51

Clean up analysis substitution class

922ad14

docstrings

d0ae9be

kszucs reviewed Sep 10, 2018

View reviewed changes

kszucs approved these changes Sep 10, 2018

View reviewed changes

cpcloud added 2 commits September 10, 2018 15:10

Try to fix perf

d669f49

Use is in Select DML equality as well

3515133

cpcloud added 6 commits September 10, 2018 15:18

Fix docstring

951cfc5

Make _safe_name a property instead of util method

d0db870

Make expr_key a _key property on Expr

90d66d9

Cleanup

a617ad7

is first

db83c80

revert

466a36b

cpcloud closed this in 2cfd385 Sep 10, 2018

cpcloud deleted the hashable-ops branch September 10, 2018 21:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH/DEV: Make ibis Node instances hashable #1611

ENH/DEV: Make ibis Node instances hashable #1611

cpcloud commented Sep 5, 2018

cpcloud Sep 5, 2018

kszucs Sep 5, 2018

cpcloud Sep 6, 2018

kszucs Sep 5, 2018

cpcloud Sep 6, 2018

kszucs left a comment

cpcloud commented Sep 7, 2018

cpcloud commented Sep 10, 2018

kszucs commented Sep 10, 2018

kszucs Sep 10, 2018 •

edited

Loading

cpcloud Sep 10, 2018

cpcloud commented Sep 10, 2018

kszucs Sep 10, 2018

kszucs Sep 10, 2018

kszucs Sep 10, 2018

kszucs Sep 10, 2018

kszucs Sep 10, 2018

cpcloud commented Sep 10, 2018

cpcloud commented Sep 10, 2018

ENH/DEV: Make ibis Node instances hashable #1611

ENH/DEV: Make ibis Node instances hashable #1611

Conversation

cpcloud commented Sep 5, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kszucs left a comment

Choose a reason for hiding this comment

cpcloud commented Sep 7, 2018

cpcloud commented Sep 10, 2018

kszucs commented Sep 10, 2018

kszucs Sep 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud commented Sep 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud commented Sep 10, 2018

cpcloud commented Sep 10, 2018

kszucs Sep 10, 2018 •

edited

Loading