[hail][feature] Super array expression

[danking]

First step towards treating CollectionExpressions exactly like non-distributed tables. Works for arrays and sets.

cc: @konradjk

>>> a = [hl.struct(b=[hl.struct(inner=1),
...                   hl.struct(inner=2)]),
...      hl.struct(b=[hl.struct(inner=3)])]
>>> hl.eval(a.b)
[[hl.struct(inner=1), hl.struct(inner=2)], [hl.struct(inner=3)]]
>>> hl.eval(a.b.inner)
[[1, 2], [3]]
>>> hl.eval(hl.flatten(a.b).inner)
[1, 2, 3]
>>> hl.eval(hl.flatten(a.b.inner))
[1, 2, 3]

[konradjk]

Do you actually want foo in here? Could you use item, or does the recursion mess with this?

[danking]

oh sure yeah item

[danking]

done

[patrick-schultz]

Should we support array['field name'] for when field name isn't a valid python identifier? We'd have to disambiguate that from array indexing, but one is always a python string and one always (convertible to) a numeric expression, right?

[danking]

@patrick-schultz I am one step ahead of you sir:

https://github.com/hail-is/hail/pull/5913/files#diff-8a90eaa51bf133eaef5d7f12c0d4decfR2736

If you pass a python string, I use the array projection approach, if you pass anything else I try to convert it to an integer.

[danking]

Every time I see this PR I get giddy. This is so in line with the Hail I want.

[tpoterba]

I don't understand this bit. Shouldn't this be map(lambda x: x[item])?

[tpoterba]

You also can't use hl.struct here -- that can change missingness patterns

[tpoterba]

Design comment - We use specialized expression classes for expression types that have unique behavior. Shouldn't we use that model here? A NestedStructCollectionExpression type of thing?

[danking]

This converts from the row format to the column format, then it performs the get item. I suppose it can be directly written as:

return self.map(lambda x: x[item])

Conceptually though, I think of myself as operating on the column formatted data. I'll change to self.map.

[tpoterba]

using __getitem__ in this way is bad style -- use x[item]

[tpoterba]

few last comments

[tpoterba]

few last comments

[tpoterba]

why this change? it was clearer before, no?

[tpoterba]

oh, for consistency. I'd prefer to change the others to elif instead

[danking]

pylint doesn't like the else elifs if not necessary. at first I hated it but now I prefer it.

[tpoterba]

do you have a PEP reference?

[danking]

It's not in PEP-8, which quietly demonstrates both styles in an unrelated recommendation. In pylint it's called no-else-return. We can add this to our standard pylint rules if you prefer this style. At first I hated it. Now I've come to prefer the uniformity of if and the lack of indentation on the else case.

[danking]

I suppose if all of hail is written using the else-return style we should stay consistent.

[patrick-schultz]

For what it's worth, I think standard C++ style is also to never put an else clause on an if statement that returns.

[tpoterba]

I'm happy with either, but do have a preference for elif in this case. I'm fine with going with style guidelines, though.

I think I have somewhat mixed preferences, too. I'd prefer:

def foo(*args):
    if take_other_path:
        return blah(*args)
    ...
    lots of code
    ...
    return other_thing

to

def foo(*args):
    if take_other_path:
        return blah(*args)
    else:
        lots of code
        ...
        return other_thing

However, in code with several short conditionals, I prefer

def foo(a, b, c, ..., z):
    if a < 0:
        return a
    elif b < 0:
        return b
    ...
    else:
        return z

to

def foo(a, b, c, ..., z):
    if a < 0:
        return a
    if b < 0:
        return b
    ...
    return z

, since if you forget one of the "return" keywords, you get a piece of code that's much harder to debug.

[tpoterba]

can revert this change, right?

[danking]

yeah but python style prefers no else or elif when each branch returns and it felt weird to move the else to my new if.

[tpoterba]

nice, super clean.

[tpoterba]

this is super

[tpoterba]

request clarification on PEP style w.r.t. returns

[tpoterba]

request clarification on PEP style w.r.t. returns

[danking]

@tpoterba @patrick-schultz Ok, I switched to elif style and made fixes so that the surrounding code also followed the elif style. I'd like to merge this and push off any further discussions to another PR. IMO, the no-else-return style is almost always nicer when I want to:

• bind a variable half way through an if chain, or
• have a complex condition (see the one with a while loop around 3400) that demands a nested if (I have to duplicate the common fall-through case)

[tpoterba]

these not implemented errors are wrong. You want these to get to line 3440 below to go through typ_to_expr.

with this in mind, the no-elif model is way nicer since you don't need to duplicate that logic!