Add support for "|" inside productions

[plafer]

Cleans up the use case where one has many small productions. For example,

@pg.production("prod: TOKEN_1")
# ... <24 lines>
@pg.production("prod: TOKEN_26")
def func(p):
    pass

Would be simplified to

@pg.production("prod: TOKEN_1 | ... | TOKEN_26")
def func(p):
    pass

More specifically, I need this right now, as I need to accept (and throwaway) all tokens until I see a newline.

I'd be happy to submit a PR if you guys think it's a valuable feature to have.

[alex]

Seems reasonable to me בתאריך יום ג׳, 4 באוג׳ 2020, 16:45, מאת Philippe Laferrière ‏< notifications@github.com>:

…

Cleans up the use case where one has many small productions. For example, @pg.production("prod: TOKEN_1") # ... <24 lines> @pg.production("prod: TOKEN_26") def func(p): pass Would be simplified to @pg.production("prod: TOKEN_1 | ... | TOKEN_26") def func(p): pass More specifically, I need this right now, as I need to accept (and throwaway) all tokens until I see a newline. I'd be happy to submit a PR if you guys think it's a valuable feature to have. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#100>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBHFK3T77XN5C6X72T3R7BXOXANCNFSM4PUZBPEQ> .

[nobodxbodon]

We have a production rule bin_expr : expr | expr where | stands for BitOr, which was fine with 0.7.7 but broken on 0.7.8. It seems to be caused by this change. I wonder if there's a way to get around? Some kind of escaping or else?

[leahcornelius]

Perhaps \ escapes it? If not then add a lexer rule to match | to VERTICLE_PIPE or similar and use that? @nobodxbodon

[nobodxbodon]

@leocornelius thanks for the suggestion. Just tried this test case and it does seem to work:

    def test_arithmetic(self):
        lg = LexerGenerator()
        lg.add("NUMBER", r"\d+")
        lg.add("BITOR", r"\|")

        pg = ParserGenerator(["NUMBER", "BITOR"], precedence=[
            ("left", ["BITOR"]),
        ])

        @pg.production("main : expr")
        def main(p):
            return p[0]

        @pg.production("expr : expr BITOR expr")
        def expr_binop(p):
            return BoxInt(operator.or_(p[0].getint(), p[2].getint()))

        @pg.production("expr : NUMBER")
        def expr_num(p):
            return BoxInt(int(p[0].getstr()))

        lexer = lg.build()
        parser = pg.build()

        assert parser.parse(lexer.lex("1|8")) == BoxInt(9)

[leahcornelius]

See #101, it should now be fixed :)