New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generic utility for expression traversal #1336

Closed
wants to merge 4 commits into
base: master
from

Conversation

Projects
2 participants
@kszucs
Member

kszucs commented Feb 6, 2018

It could be further improved to support more specific cases, e.g. with passing callable as control variable. Probably we should avoid specific scenarios (like _get_args) and refactor the underlying structure instead.

self.validator = ExprValidator([self.parent.table])
self.valid = True

This comment has been minimized.

@kszucs

kszucs Feb 6, 2018

Member

Was this for caching purposes? If so then I won't remove.

@cpcloud cpcloud self-requested a review Feb 6, 2018

@cpcloud cpcloud self-assigned this Feb 6, 2018

@cpcloud cpcloud added this to To do in Refactoring via automation Feb 6, 2018

@cpcloud cpcloud added this to the 0.13 milestone Feb 6, 2018

# these could be callables instead
proceed = True

This comment has been minimized.

@cpcloud

cpcloud Feb 7, 2018

Member

Let 's make these be functions with the following signature:

def default_proceed(expr, result):
    return True


def default_halt(expr, result):
    return not default_proceed(expr, result)

The the signature of traverse becomes:

def traverse(fn, expr, type=ir.Expr, proceed=default_proceed, halt=default_halt, container=Stack):
    pass

This comment has been minimized.

@cpcloud

cpcloud Feb 7, 2018

Member

result = fn(expr) in my function signatures above.

This comment has been minimized.

@kszucs

kszucs Feb 7, 2018

Member

I'm not sure what You're thinking of, but I've sketched the new versions in the comments.
Do I understand it correctly?

if isinstance(expr, ir.TableExpr):
return lin.stop, expr
else:
return lin.proceed, None

This comment has been minimized.

@kszucs

kszucs Feb 7, 2018

Member

So this would look like the following?

def finder(expr):
    if isinstance(expr, ir.TableExpr):
        return expr

def halt(expr, result):
    return isinstance(expr, ir.TableExpr):

lin.traverse(finder, expr.op().flat_args(), halt=halt)

This comment has been minimized.

@cpcloud

cpcloud Feb 8, 2018

Member

Hm, now I'm not so sure I like my suggestion. This duplicates the condition which isn't good.

This comment has been minimized.

@cpcloud

cpcloud Feb 8, 2018

Member

Let's keep your API for now.

This comment has been minimized.

@kszucs

kszucs Feb 8, 2018

Member

Ok, should I rename stop to halt?

This comment has been minimized.

@cpcloud

cpcloud Feb 8, 2018

Member

Yep, then will merge

if isinstance(expr.op(), ops.And):
return lin.proceed, None
else:
return lin.stop, expr

This comment has been minimized.

@kszucs

kszucs Feb 7, 2018

Member
lin.traverse(
    lambda expr: expr if not isinstance(expr.op(), ops.And) else None, 
    expr, 
    proceed=lambda expr, result: isinstance(expr.op(), ops.And),
    type=ir.BooleanColumn
)
data = None
return lin.proceed, data
return collections.OrderedDict(lin.traverse(finder, expr))

This comment has been minimized.

@kszucs

kszucs Feb 7, 2018

Member
def fn(expr):
    op = expr.op()
    if hasattr(op, 'source'):
        return (op, op.source.dictionary.get(op.name, None))
    elif isinstance(op, ir.Literal):
        return (op, op.value)

collections.OrderedDict(lin.traverse(fn, expr))
op = expr.op()
if isinstance(op, ops.TableColumn):
return lin.proceed, self._validate_column(expr)
return lin.proceed, None

This comment has been minimized.

@kszucs

kszucs Feb 7, 2018

Member

This one would be simpler.

def validate(expr):
    op = expr.op()
    if isinstance(op, ops.TableColumn):
        return self._validate_column(expr)

lin.traverse(validate, expr, type=ir.ValueExpr)

This comment has been minimized.

@cpcloud
if isinstance(expr, ir.TableExpr):
return lin.stop, expr
else:
return lin.proceed, None

This comment has been minimized.

@kszucs

kszucs Feb 7, 2018

Member
def finder(expr):
    if isinstance(expr, ir.TableExpr):
        return expr

def proceed(expr, result):
    return not isinstance(expr, ir.TableExpr)

lin.traverse(finder, expr, proceed=proceed)

This comment has been minimized.

@cpcloud

cpcloud Feb 8, 2018

Member

Yep, that looks good to me.

This comment has been minimized.

@kszucs

kszucs Feb 8, 2018

Member

Ehh, You need to convince me about the advantages :)
I favor the previous one.

This comment has been minimized.

@kszucs

kszucs Feb 8, 2018

Member

In case of lineage function the callable must return the next traversable expressions.

@cpcloud

This comment has been minimized.

Member

cpcloud commented Feb 8, 2018

@kszucs can you rebase?

@kszucs kszucs force-pushed the kszucs:traverse branch from 2078526 to dda25e4 Feb 8, 2018

@kszucs

This comment has been minimized.

Member

kszucs commented Feb 8, 2018

@cpcloud rebase done

@cpcloud

cpcloud approved these changes Feb 9, 2018

@cpcloud cpcloud closed this in e1928ed Feb 9, 2018

Refactoring automation moved this from To do to Done Feb 9, 2018

@cpcloud

This comment has been minimized.

Member

cpcloud commented Feb 9, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment