Recognizing consecutive tokens with no whitespace in between? (the >> problem)

[osa1]

Sorry if this was discussed already, but I can't figure out how to parse two consecutive tokens with no whitespace in between, to handle cases like >> in expression context while also handling T1<T2<T3>> and T1<T2<T3> > in type context.

Is this possible in lalrpop? Does anyone have any example code that does this?

Edit: Another use case that just came up while working on Fir is this:

Type: () = {
    Symbol "[" Type "]",     // Vec[I32]
    "[" Type "]",            // [I32]
    Type Type                // I32 I32
}

Here I want Vec [I32] (note the space) to be parsed as production 3, not as 1.

[dburgener]

I think the standard way to handle this is to consistently treat ">" as a token, and then parse ">" ">" at that parser level. You have to be careful about avoiding ambiguities where you use ">>" in something inside a "<" ">" pair. Like if if your T1<T2<T3>> example, if T3 could actually be T4>>T5, then we have an ambiguity when we see T1<T2<T4>, since we don't know whether we're closing the T2 brackets or doing a >>. But that's not unique to the >> problem or to lalrpop, it's just that a grammar where T3 can be an expression using >> isn't LR(1).

Anyways, the below toy example is valid lalrpop that handles both T1<T2<T3>> and >> in different contexts. The whitespace aspect is just handled by having ">" be a valid token in the lexer and ">>" not be one.

grammar;

pub Expr: () = {
        Type,
        Expr ">" ">" Symbol,
        Symbol,
}

Type: () = {
        Symbol "<" Type ">",
}

Symbol: () = {
        "[a-zA-Z0-9]*"
}

[osa1]

Your parser parses x > > y (note the whitespace) as an expr. My question was how to accept consecutive >s only if they're next to each other in input (no whitespace in between). x > > y isn't valid expression but x > y and x >> y are.

[dburgener]

Oh, yep. You are absolutely right. That's a good point.

I think based on that, I'm not aware of a way to do this today without writing a custom lexer than can be context aware. #673 would be a cleaner approach if that ever lands, but I don't believe it's currently being actively worked on. But without that, I suspect that the answer here is that you need to write a custom context-aware lexer to handle this in lalrpop?