[WIP] Macro rules backtracking #33840

LeoTestard · 2016-05-24T14:54:41Z

This fixes macro_rules! to implement the wanted backtracking semantics: if an arm fails to match, the next arm is tried, and so on.

This was previously not the case because of hard errors emitted during parsing of non-terminals. Now that the parser code has been fixed not to panic but to return a DiagnosticBuilder in case of error, it's easy to catch those errors and signal that the arm failed to match.

This PR probably contains a bit of hackery, so it's flagged as WIP so that people can review it.

rust-highfive · 2016-05-24T14:54:55Z

r? @sfackler

(rust_highfive has picked a reviewer for you, use r? to override)

LeoTestard · 2016-05-24T14:55:04Z

r? @nrc

nagisa · 2016-05-25T00:47:54Z

Potential O(n²)?

LeoTestard · 2016-05-25T09:40:40Z

No. The parsing algorithm is the same. There is actually no « backtracking » as you may understand it. The parsing algorithm works on a single arm and, on failure, will try the next arms with the same tokens. This PR only make sure all errors encountered during the parsing for an arm are not fatal but properly reported. There's no change in complexity here. In fact, the parsing algorithm could already be called on every arm if no non-terminal parsing errors occured.

… of increasing it.

This allows to cancel an error with only an immutable access to the `Handler`. Mutable access is already not required to emit an error anyway.

nrc · 2016-06-02T08:10:35Z

src/libsyntax/ext/tt/macro_parser.rs

    match name {
        "tt" => {
            p.quote_depth += 1; //but in theory, non-quoted tts might be useful
            let res: ::parse::PResult<'a, _> = p.parse_token_tree();
-            let res = token::NtTT(P(panictry!(res)));
+            let res = token::NtTT(P(try!(res)));


nit: prefer ? to try!

LeoTestard · 2016-06-07T20:16:36Z

Well I think I can remove the ‶WIP″ marker and submit this PR for review. cc @pnkfelix @nrc

nrc · 2016-06-08T12:41:05Z

lgtm with a cursory read over the code, but r? @pnkfelix for a detailed review

LeoTestard · 2016-06-09T00:13:31Z

src/libsyntax/ext/tt/macro_parser.rs

+                        let nt = match parse_nt(&mut rust_parser, span,
+                                                &ident.name.as_str()) {
+                            Ok(nt) => Rc::new(MatchedNonterminal(nt)),
+                            Err(diag) => return Failure(diag)


Note to myself: we must not fail here, but rather just remove the current parsing item from the stack.

LeoTestard · 2016-06-09T11:32:54Z

Note for myself and for others: Allright we have a problem here. As I noted in the comment above, we currently only backtrack from an arm, not from a parsing item. That means that if you have a macro like this:

macro_rules! foo(
    ( $( $e:expr ),* some_other_tokens ) => ...
)

If we provide it input that doesn't match expr, you'd expect it to skip the sequence repetition, since it can appear 0 times, and instead match it against the following tokens. This is not the case, since, as I outlined in the line note, it will just fail and continue with the next arm.

This sounds easy to fix: just remove the current parsing item from the stack.
But. If we do this, we have a future-proofing issue. We thought future-proofing analysis was solved for single-arm macros. It was only because the backtracking semantics were not implemented. Now consider the following macro:

macro_rules! foo(
    ( $( $e:expr , )* $( $i:ident : $t:ty , )* ) => ...
)

Before type ascription syntax, foo!(a: i32,) should skip the first sequence repetition, since a: i32 is not an expr, and bind a to $i.
After type ascription syntax, it will bind a: i32 to $e and skip the second sequence repetition.
The thing is that, based on the current FOLLOW sets analysis for single arm macros, we can't reject this because all requirements are met: : is in FOLLW(ident), ident and expr both are in FOLLOW(,), etc.

This was not a problem before because failure to match a NT (such as expr) inside the sequence repetition would abort the parsing of the macro. But those are not the desired parser semantics.

This problem is in fact the very same as the one with multiple-arm macros:

macro_rules! foo(
    $( $e:expr ) => ... ;
    $( $i:ident : $t:ty ) => ...
)

So I believe that my analysis for multiple-arm macros could be applied to fix the issue with the current future-proofing analysis for single-arm macros. But until we sorted that out, we have two solutions for this PR:

Don't fix it to backtrack inside a single arm, keep it that way, and merge it. We'll fix this issue later.
Don't merge it, and wait for the future-proofing issue I described to be fixed, and then make a true fix for backtracking that addresses this issue.

(By fixing this issue, I mean: making sure the parser backtracks out of a sequence repetition.)

cc @nikomatsakis @pnkfelix

nikomatsakis · 2016-06-24T16:18:18Z

@LeoTestard as we said in person, this is indeed a drag, though I guess it just ups the priority of investigating the other future-proofing problem. :(

bors · 2016-06-28T04:46:13Z

☔ The latest upstream changes (presumably #34424) made this pull request unmergeable. Please resolve the merge conflicts.

alexcrichton · 2016-07-19T04:27:52Z

ping, @LeoTestard looks like this needs a rebase?

LeoTestard · 2016-07-19T10:09:22Z

@alexcrichton Yes indeed, but it's blocked anyway until we have a fix for the future-proofing issue. I don't know what we should do with this PR. Maybe we can just close it and reopen it later?

alexcrichton · 2016-07-19T15:37:58Z

Ah either way is fine by me, this is now active in the last 5 hours rather than 21 days, so I'm happy at least :)

brson · 2016-11-09T01:24:21Z

This is quite stalled. @pnkfelix @LeoTestard is there any chance of this PR landing? Should we close?

LeoTestard · 2016-12-21T23:30:18Z

@brson Yes, there is still a chance. I thought again about this problem after I found the problem I mentioned above, and I no longer think it's a problem. It turns out that I did not understood how the parser handled ambiguity at the time. I should be able to sum up the situation in more detail.

steveklabnik · 2017-01-03T16:23:25Z

@LeoTestard

I should be able to sum up the situation in more detail.

Any chance of this happening? I am tempted to just close this PR until you or someone else is ready to move forward here; this is going to be extremely out of date, right?

nikomatsakis · 2017-01-03T18:18:48Z

Seems reasonable to close the PR in the meantime, though @LeoTestard I'd like to hear more about what you have in mind.

LeoTestard · 2017-01-05T14:33:20Z

@steveklabnik The code is probably quite out of date indeed, I agree that the PR can be closed in the meantime.

@nikomatsakis Sure. I promise I'll take the time to write about it at some point, but I'm quite busy these days.

rust-highfive assigned sfackler May 24, 2016

rust-highfive assigned nrc and unassigned sfackler May 24, 2016

LeoTestard force-pushed the macro-rules-backtracking branch from 07a1b18 to dde0e88 Compare May 24, 2016 15:05

LeoTestard force-pushed the macro-rules-backtracking branch from dde0e88 to ce31cb7 Compare May 25, 2016 09:43

LeoTestard added 2 commits May 31, 2016 12:10

Fix errors::Handler::cancel() to *decrease* the error count instead…

096ed00

… of increasing it.

Make errors::Handler::cancel() take an immutable reference.

97839c0

This allows to cancel an error with only an immutable access to the `Handler`. Mutable access is already not required to emit an error anyway.

LeoTestard force-pushed the macro-rules-backtracking branch 2 times, most recently from b9eb282 to 0d8dae4 Compare May 31, 2016 10:22

LeoTestard mentioned this pull request Jun 1, 2016

[wip] reference: rework Macros section #31990

Closed

9 tasks

nrc reviewed Jun 2, 2016
View reviewed changes

[WIP] Remove panic! in macro parsing.

be53b20

LeoTestard force-pushed the macro-rules-backtracking branch from 0d8dae4 to be53b20 Compare June 2, 2016 15:45

LeoTestard mentioned this pull request Jun 2, 2016

Add a literal fragment specifier to macro_rules!. rust-lang/rfcs#1576

Merged

LeoTestard force-pushed the macro-rules-backtracking branch from 67c1790 to 13a8c8d Compare June 8, 2016 12:40

rust-highfive assigned pnkfelix and unassigned nrc Jun 8, 2016

LeoTestard reviewed Jun 9, 2016
View reviewed changes

Use a public field instead of methods.

24b9e39

LeoTestard force-pushed the macro-rules-backtracking branch from 13a8c8d to 24b9e39 Compare June 9, 2016 14:37

durka mentioned this pull request Aug 13, 2016

Fragment specifiers in macro are not taken into account #35650

Closed

alexcrichton mentioned this pull request Oct 31, 2016

[WIP] Check future proofing of macros with multiple arms using FIRST sets. #34140

Closed

LeoTestard mentioned this pull request Dec 21, 2016

Check future-proofing of macro_rules! using FIRST sets. rust-lang/rfcs#1746

Closed

pnkfelix closed this Jan 4, 2017

pnkfelix added A-macros Area: All kinds of macros (custom derive, macro_rules!, proc macros, ..) T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Jan 4, 2017

kennytm mentioned this pull request Jun 29, 2017

Only match a fragment specifier if it starts with certain tokens. #42913

Merged

ehuss mentioned this pull request Apr 17, 2021

Odd behaviour coming from macros #84259

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Macro rules backtracking #33840

[WIP] Macro rules backtracking #33840

LeoTestard commented May 24, 2016

rust-highfive commented May 24, 2016

LeoTestard commented May 24, 2016

nagisa commented May 25, 2016

LeoTestard commented May 25, 2016

nrc Jun 2, 2016

LeoTestard commented Jun 7, 2016 •

edited

nrc commented Jun 8, 2016

LeoTestard Jun 9, 2016

LeoTestard commented Jun 9, 2016 •

edited

nikomatsakis commented Jun 24, 2016

bors commented Jun 28, 2016

alexcrichton commented Jul 19, 2016

LeoTestard commented Jul 19, 2016

alexcrichton commented Jul 19, 2016

brson commented Nov 9, 2016

LeoTestard commented Dec 21, 2016

steveklabnik commented Jan 3, 2017

nikomatsakis commented Jan 3, 2017

LeoTestard commented Jan 5, 2017

[WIP] Macro rules backtracking #33840

[WIP] Macro rules backtracking #33840

Conversation

LeoTestard commented May 24, 2016

rust-highfive commented May 24, 2016

LeoTestard commented May 24, 2016

nagisa commented May 25, 2016

LeoTestard commented May 25, 2016

nrc Jun 2, 2016

Choose a reason for hiding this comment

LeoTestard commented Jun 7, 2016 • edited

nrc commented Jun 8, 2016

LeoTestard Jun 9, 2016

Choose a reason for hiding this comment

LeoTestard commented Jun 9, 2016 • edited

nikomatsakis commented Jun 24, 2016

bors commented Jun 28, 2016

alexcrichton commented Jul 19, 2016

LeoTestard commented Jul 19, 2016

alexcrichton commented Jul 19, 2016

brson commented Nov 9, 2016

LeoTestard commented Dec 21, 2016

steveklabnik commented Jan 3, 2017

nikomatsakis commented Jan 3, 2017

LeoTestard commented Jan 5, 2017

LeoTestard commented Jun 7, 2016 •

edited

LeoTestard commented Jun 9, 2016 •

edited