ParseTree to AST transformation with alternative rules

[mar9000]

In a test grammar I have a rule composed of only alternatives and one of this alternative is also composed of only alternatives. Like

abstractRuleA: abstractRuleB | ruleC;
abstractRuleB: abstractRuleD | ruleE;

You can also look at the Python parsing example, see the stmt and simple_stmt rules.
it is possible we have other levels. The implementation of ParseTreeToASTTransformer and registerNodeFactory() is not possible, AFAIK because there are levels where we can't return a Node.
There is a workaroud when we have only one level that is by using alternatives labels.

But I was not able to figure it out when we have 2 or more levels.

The tutorial/example on how to parse Python in fact as the same problem (see the stmt and simple_stmt rules) but it does not uses ParseTreeToASTTransformer.

[mar9000]

I've found it... indeed was pretty simple: override transform() to just skip the pure alternative rules plus altrnative labels:

abstractRuleA:
    abstractRuleB  #abstractRuleBAlias
    | ruleC #ruleCAlias
    ;
abstractRuleB: 
    abstractRuleD #abstractRuleDAlias
    | ruleE #ruleEAlias
    ;

override fun transform(source: Any?, parent: Node?): Node? {
        return when (source) {
            is AbstractRuleBAliasContext -> transform(source.abstractRuleB(), parent)
            // other rules...
            else -> super.transform(source, parent)
        }
}```

[ftomassetti]

Thank you Marco for sharing this. I think this can be interesting also for @alessiostalla

[alessiostalla]

Hi, I'm not sure I understand. Where's the Python example you're talking about? Anyway, sure you can override transform but that doesn't scale if you have many such cases. A registerNodeFactory should be sufficient.

[mar9000]

AFAIK is not possible to use registerNodeFactory because for several rules there is nothing you can construnct/return, and if you calculate it, the child description in withChild() become cumbersome.

The Python example I was speaking about is this:
https://github.com/Strumenta/kolasu-tutorial/blob/master/src/main/kotlin/com/strumenta/python3parser/parsetreetoast/conversions.kt

From the grammar:

stmt: simple_stmt | compound_stmt;
simple_stmt: small_stmt (';' small_stmt)* (';')? NEWLINE;
small_stmt: (expr_stmt | del_stmt | pass_stmt | flow_stmt |
             import_stmt | global_stmt | nonlocal_stmt | assert_stmt);
expr_stmt: testlist_star_expr (annassign | augassign (yield_expr|testlist) |
                     ('=' (yield_expr|testlist_star_expr))*);
flow_stmt: break_stmt | continue_stmt | return_stmt | raise_stmt | yield_stmt;

Here you have several examples. For instance you can label small_stmt rule, indeed registering a factory for label alias, but then for flow_stmtAlias you cannot return a Node. You have just to skip it and continue with break_stmt, or continue_stmt, etc.

It's in this case I thought to resolve the problem with:

override fun transform(source: Any?, parent: Node?): Node? {
        return when (source) {
            is Flow_StmtContext -> 
                var flowStmtChild = <one of source.break_stmt or one of source.continue_stmt or...>
                transform(flowStmtChild, parent)
            // other rules...
            else -> super.transform(source, parent)
        }
}

The mentioned Python parser uses kolasu but not the ParseTreeToAstTrasformer and node factory. I pretty sure if you try to port that parser to the same technic used for example in the kolasu-entity-parser you will face this problem.
Nevertheless the code in the overriden transform() is very small, it has only to skip "pure alternative" rules like flow_stmt.

With the use of alternative labels it's even less required: it's necessary only when a rule of a pure alternative rules (small_stmt) reference another pure alternative rule (flow_stmt).

[alessiostalla]

OK, if I understand it, in those cases we use a registerNodeFactory overload that takes a function, like so (untested):

transformer.registerNodeFactory(Flow_StmtContext) { ctx -> 
    when (ctx) {
        is Flow_StmtContext -> ...
        else -> ... //throw, or return a special error node
    }
}

It's the same logic, really, only it doesn't require overriding the transform method.

[mar9000]

Ok, but as I said above AFAIK, this is ok if you do not have to use withChild(). continue_stmt and break_stmt do not have children luckly, but the other ones have children.
Because flow_stmt is let's say "generic" you have to use withChild() on that factory at least 3 times, one time for each flow_stmt alternative with children.

transformer.registerNodeFactory(Flow_StmtContext) { ctx -> 
    when (ctx) {
        is Break_StmtContext -> ...
        is Continue_StmtContext -> ...
        else -> ... //throw, or return a special error node
    }
}
.withChild(when ctx is Return_StmtContext get/set return value)
.withChild(when ctx is Yield_StmtContext get/set yield value)
.withChild(when ctx is Raise_StmtContext get/set raise value)

And this is a simple case because return, yield and raise have only one child. Take compound_stmt and try to write the registerNodeFactory() call complete with the withChild() calls.

Of course it can be written but it's probably more concise to substitute compound_stmt with its alternative in the context overriding the transform() method.