Relaxed antiquotation parsing

[Kha]

This is an incomplete implementation of an idea I mentioned in the Typed Macros PR: #1251 (comment) (the first half). In short, when encountering an antiquotation $x while running a parser p, we do not immediately assume that the entire antiquotation is for p but check if we p accepts $x and further tokens behind it. Thus the example `(structInstField| $id := $val) is interpreted as `(structInstField| $id:ident := $val:term), not `(structInstField| $id:structInstField <ERROR: expected ')'>.

Unfortunately, there are some regressions. One of them is defensible I think , but the other would likely require us to adapt the Lean grammar; see below.

[Kha]

I think this one makes sense: how many people would read the original line and conclude that $seq is not the first of two elements of do, but the entire content of it? With the relaxed antiquotation parser, we do now get seq : TSyntax `doElem in the modified line.
The fix only makes this marginally clearer; I would have liked (do $seq); ... better, which does interpret seq as a doSeq (or, with the second part of the comment linked above, as TSyntax [`doSeq, `doElem]). However, the parens terminate e.g. mutable variables, which is undesirable.

[gebner]

To be fair, I can empathize with Lean here. I've also found the zoo of do-notation nonterminals a bit confusing at times.

Indentation would be another way to disambiguate this example:

`(doElem|
  repeat
    do $seq
    if $cond then break)

Excessive use of braces helps too:

`(doElem| repeat do { do $seq }; if $cond then break)

We could also remove the slightly misleading syntax for nested do and replace it by braces. (I think it's misleading because it's not a boundary for return like the normal do.)

syntax "{" doSeq "}" : doElem
-- then this should work
`(doElem| repeat { $seq }; if $cond then break)

A coercion from doSeq to doSeqItem could also help (but I guess you don't want a coercion to insert new tokens).

[Kha]

Yeah, it might be good to clean up the grammar there. And yes introducing tokens in a coercion probably isn't a good idea due to lack of access to getRef (though I even thought about a monadic coercion class that is automatically used by antiquotations).

[Kha]

I'm going ahead and merging this as it should be a strict improvement now and I've been confused multiple times about why it didn't work on master yet.