[hail][feature] Super array expression #5913

danking · 2019-04-18T23:19:43Z

First step towards treating CollectionExpressions exactly like non-distributed tables. Works for arrays and sets.

>>> a = [hl.struct(b=[hl.struct(inner=1),
...                   hl.struct(inner=2)]),
...      hl.struct(b=[hl.struct(inner=3)])]
>>> hl.eval(a.b)
[[hl.struct(inner=1), hl.struct(inner=2)], [hl.struct(inner=3)]]
>>> hl.eval(a.b.inner)
[[1, 2], [3]]
>>> hl.eval(hl.flatten(a.b).inner)
[1, 2, 3]
>>> hl.eval(hl.flatten(a.b.inner))
[1, 2, 3]

konradjk · 2019-04-18T23:28:57Z

hail/python/hail/expr/expressions/typed_expressions.py

+            if "Cannot use 'collection.foo' or" not in str(err):
+                raise
+        raise TypeError(
+            f"Cannot use 'collection.foo' or 'collection['foo']' on "


Do you actually want foo in here? Could you use item, or does the recursion mess with this?

oh sure yeah item

patrick-schultz · 2019-04-19T12:30:55Z

Should we support array['field name'] for when field name isn't a valid python identifier? We'd have to disambiguate that from array indexing, but one is always a python string and one always (convertible to) a numeric expression, right?

danking · 2019-04-19T15:03:44Z

@patrick-schultz I am one step ahead of you sir:

https://github.com/hail-is/hail/pull/5913/files#diff-8a90eaa51bf133eaef5d7f12c0d4decfR2736

If you pass a python string, I use the array projection approach, if you pass anything else I try to convert it to an integer.

danking · 2019-04-19T15:04:09Z

Every time I see this PR I get giddy. This is so in line with the Hail I want.

tpoterba · 2019-04-19T13:45:12Z

hail/python/hail/expr/expressions/typed_expressions.py

+        etype = self.dtype.element_type
+        if isinstance(etype, hl.tstruct):
+            return hl.struct(
+                **{k: self.map(lambda x: x[k]) for k in etype}).__getitem__(item)


I don't understand this bit. Shouldn't this be map(lambda x: x[item])?

You also can't use hl.struct here -- that can change missingness patterns

Design comment - We use specialized expression classes for expression types that have unique behavior. Shouldn't we use that model here? A NestedStructCollectionExpression type of thing?

This converts from the row format to the column format, then it performs the get item. I suppose it can be directly written as:

return self.map(lambda x: x[item])

Conceptually though, I think of myself as operating on the column formatted data. I'll change to self.map.

tpoterba · 2019-04-19T15:07:24Z

hail/python/hail/expr/expressions/typed_expressions.py

+                **{k: self.map(lambda x: x[k]) for k in etype}).__getitem__(item)
+        try:
+            if isinstance(etype, (hl.tarray, hl.tset)):
+                return self.map(lambda x: x.__getitem__(item))


using __getitem__ in this way is bad style -- use x[item]

addressed

tpoterba

few last comments

tpoterba · 2019-04-22T22:17:56Z

hail/python/hail/expr/expressions/typed_expressions.py

    if isinstance(type, tndarray) and is_numeric(type.element_type):
        return NDArrayNumericExpression(ir, type, indices, aggregations)
-    elif type in scalars:
+    if type in scalars:


why this change? it was clearer before, no?

oh, for consistency. I'd prefer to change the others to elif instead

pylint doesn't like the else elifs if not necessary. at first I hated it but now I prefer it.

do you have a PEP reference?

It's not in PEP-8, which quietly demonstrates both styles in an unrelated recommendation. In pylint it's called no-else-return. We can add this to our standard pylint rules if you prefer this style. At first I hated it. Now I've come to prefer the uniformity of if and the lack of indentation on the else case.

I suppose if all of hail is written using the else-return style we should stay consistent.

For what it's worth, I think standard C++ style is also to never put an else clause on an if statement that returns.

I'm happy with either, but do have a preference for elif in this case. I'm fine with going with style guidelines, though.

I think I have somewhat mixed preferences, too. I'd prefer:

def foo(*args): if take_other_path: return blah(*args) ... lots of code ... return other_thing

to

def foo(*args): if take_other_path: return blah(*args) else: lots of code ... return other_thing

However, in code with several short conditionals, I prefer

def foo(a, b, c, ..., z): if a < 0: return a elif b < 0: return b ... else: return z

to

def foo(a, b, c, ..., z): if a < 0: return a if b < 0: return b ... return z

, since if you forget one of the "return" keywords, you get a piece of code that's much harder to debug.

tpoterba · 2019-04-22T22:20:42Z

hail/python/hail/expr/expressions/typed_expressions.py

@@ -420,12 +420,13 @@ def __getitem__(self, item):
        """
        if isinstance(item, slice):
            return self._slice(self.dtype, item.start, item.stop, item.step)
-        else:


can revert this change, right?

yeah but python style prefers no else or elif when each branch returns and it felt weird to move the else to my new if.

tpoterba · 2019-04-22T22:21:40Z

hail/python/hail/expr/expressions/typed_expressions.py

+        """
+
+        if isinstance(item, str):
+            return self.map(lambda x: x[item])


nice, super clean.

tpoterba · 2019-04-23T00:15:04Z

this is super

x

tpoterba

request clarification on PEP style w.r.t. returns

danking · 2019-04-25T19:36:38Z

@tpoterba @patrick-schultz Ok, I switched to elif style and made fixes so that the surrounding code also followed the elif style. I'd like to merge this and push off any further discussions to another PR. IMO, the no-else-return style is almost always nicer when I want to:

bind a variable half way through an if chain, or
have a complex condition (see the one with a while loop around 3400) that demands a nested if (I have to duplicate the common fall-through case)

x

tpoterba · 2019-04-26T15:14:35Z

hail/python/hail/expr/expressions/typed_expressions.py

+        if isinstance(etype, hl.tstruct):
+            return ArrayStructExpression(ir, type, indices, aggregations)
+        else:
+            raise NotImplementedError(type)


these not implemented errors are wrong. You want these to get to line 3440 below to go through typ_to_expr.

with this in mind, the no-elif model is way nicer since you don't need to duplicate that logic!

fixed

danking assigned tpoterba Apr 18, 2019

danking changed the title ~~Super array expression~~ [hail][feature] Super array expression Apr 18, 2019

konradjk reviewed Apr 18, 2019

View reviewed changes

tpoterba previously requested changes Apr 19, 2019

View reviewed changes

tpoterba previously requested changes Apr 23, 2019

View reviewed changes

danking force-pushed the super-array-expression branch from a3e5540 to f8e9624 Compare April 25, 2019 19:28

tpoterba previously requested changes Apr 26, 2019

View reviewed changes

tpoterba approved these changes Apr 26, 2019

View reviewed changes

danking force-pushed the super-array-expression branch 2 times, most recently from e52474d to 5928e94 Compare May 7, 2019 17:16

danking added 10 commits May 9, 2019 13:21

[hail] collection expression acts like a table

5792479

boom

5272368

remove reuqirements changes

0da2647

fixes

7dd9706

fix

fa1517d

fix

c973ca4

fml

9be107c

fixes

b6c90c2

fix

11679ab

fix

acad005

danking added 9 commits May 9, 2019 13:22

fix

c9f6136

no getitem, no column format

1359e85

expressions!

0c4fd32

fix array expression getitem

83156e8

elif style

6d6a640

elif style

f5e47be

fix

bf62431

fix

65777b0

fix two bugs

d15c85a

danking force-pushed the super-array-expression branch from 589b14b to d15c85a Compare May 9, 2019 17:22

danking merged commit 256109d into hail-is:master May 9, 2019

danking deleted the super-array-expression branch December 18, 2019 19:59

nawatts mentioned this pull request Jan 22, 2020

Change exception type for __getattr__ on struct collection expressions #7948

Merged

[hail][feature] Super array expression #5913

[hail][feature] Super array expression #5913

Uh oh!

Conversation

danking commented Apr 18, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

patrick-schultz commented Apr 19, 2019

Uh oh!

danking commented Apr 19, 2019

Uh oh!

danking commented Apr 19, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tpoterba left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tpoterba commented Apr 23, 2019

Uh oh!

tpoterba left a comment

Choose a reason for hiding this comment

Uh oh!

danking commented Apr 25, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!