Generic utility for expression traversal

[kszucs]

It could be further improved to support more specific cases, e.g. with passing callable as control variable. Probably we should avoid specific scenarios (like _get_args) and refactor the underlying structure instead.

[kszucs]

Was this for caching purposes? If so then I won't remove.

[cpcloud]

Let 's make these be functions with the following signature:

def default_proceed(expr, result):
    return True


def default_halt(expr, result):
    return not default_proceed(expr, result)

The the signature of traverse becomes:

def traverse(fn, expr, type=ir.Expr, proceed=default_proceed, halt=default_halt, container=Stack):
    pass

[cpcloud]

result = fn(expr) in my function signatures above.

[kszucs]

I'm not sure what You're thinking of, but I've sketched the new versions in the comments.
Do I understand it correctly?

[kszucs]

So this would look like the following?

def finder(expr):
    if isinstance(expr, ir.TableExpr):
        return expr

def halt(expr, result):
    return isinstance(expr, ir.TableExpr):

lin.traverse(finder, expr.op().flat_args(), halt=halt)

[cpcloud]

Hm, now I'm not so sure I like my suggestion. This duplicates the condition which isn't good.

[cpcloud]

Let's keep your API for now.

[kszucs]

Ok, should I rename stop to halt?

[cpcloud]

Yep, then will merge

[kszucs]

lin.traverse(
    lambda expr: expr if not isinstance(expr.op(), ops.And) else None, 
    expr, 
    proceed=lambda expr, result: isinstance(expr.op(), ops.And),
    type=ir.BooleanColumn
)

[kszucs]

def fn(expr):
    op = expr.op()
    if hasattr(op, 'source'):
        return (op, op.source.dictionary.get(op.name, None))
    elif isinstance(op, ir.Literal):
        return (op, op.value)

collections.OrderedDict(lin.traverse(fn, expr))

[kszucs]

This one would be simpler.

def validate(expr):
    op = expr.op()
    if isinstance(op, ops.TableColumn):
        return self._validate_column(expr)

lin.traverse(validate, expr, type=ir.ValueExpr)

[cpcloud]

Yep.

[kszucs]

def finder(expr):
    if isinstance(expr, ir.TableExpr):
        return expr

def proceed(expr, result):
    return not isinstance(expr, ir.TableExpr)

lin.traverse(finder, expr, proceed=proceed)

[cpcloud]

Yep, that looks good to me.

[kszucs]

Ehh, You need to convince me about the advantages :)
I favor the previous one.

[kszucs]

In case of lineage function the callable must return the next traversable expressions.

[cpcloud]

@kszucs can you rebase?

[kszucs]

@cpcloud rebase done

[cpcloud]