Prevent bumping the parser past the EOF. #32479

eddyb · 2016-03-25T14:48:58Z

Makes Parser::bump after EOF into an ICE, forcing callers to avoid repeated EOF bumps.
This ICE is intended to break infinite loops where EOF wasn't stopping the loop.

For example, the handling of EOF in parse_trait_items' recovery loop fixes #32446.
But even without this specific fix, the ICE is triggered, which helps diagnosis and UX.

This is a [breaking-change] for plugins authors who eagerly eat multiple EOFs.
See docopt/docopt.rs#171 for such an example and the necessary fix.

nikomatsakis · 2016-03-25T15:30:50Z

@bors r+

bors · 2016-03-25T15:30:51Z

📌 Commit 4b00d0c has been approved by nikomatsakis

nikomatsakis · 2016-03-25T15:31:38Z

@bors r-

nikomatsakis · 2016-03-25T15:31:45Z

starting a crater run

nagisa · 2016-03-25T18:11:46Z

src/libsyntax/parse/parser.rs

-                            if p.token == token::Semi {
+
+                            // Don't bump past the EOF.
+                            if p.token == token::Eof {


To me it feels like at this point converting this chain to a match would be better here.

I wish loop-match was a bit more first-class, but I agree.

nikomatsakis · 2016-03-25T19:50:38Z

Crater report: https://gist.github.com/nikomatsakis/416a719bf29fb0fa8028

4431 crates tested: 2819 working / 1508 broken / 7 regressed / 1 fixed / 96 unknown.

haven't investigated yet.

eddyb · 2016-03-25T19:55:29Z

Looks like this breaks syntax extensions (regex, peg and docopt, at least).

nikomatsakis · 2016-03-26T09:20:14Z

cc @BurntSushi @kevinmehall -- curious to get your take here. In response to various bugs where the parser was looping infinitely, @eddyb implemented a change that caused the bump method to panic if you attempt to bump past EOF. This breaks some of your syntax extension crates. We're debating whether to land it and under what conditions. (One option might be to modify your crates.)

@eddyb, I know you were attempting to modify the PR to only abort if we bumped past EOF multiple times, what was the outcome of those experiments?

eddyb · 2016-03-26T12:31:18Z

@nikomatsakis Actually, that's the current state, but parse_unspanned_seq is giving me trouble because it's unconditionally bumping a token, even if it's not the ket and the travis failure was because that changes how errors are reported (presumably due to the "expected tokens" list which check, used by eat, pushes onto).

I realize right now that I can just handle the error case by not bumping, and I just pushed that change.

EDIT: Hah, it's not that easy, you can have both errors and a } to bump.

BurntSushi · 2016-03-26T13:58:54Z

@nikomatsakis I'm totally fine with breakage of plugins if there's a path to fixing them.

With that said, what part of the parsing code would break? Regex's interaction with the parser is quite small: https://github.com/rust-lang-nursery/regex/blob/master/regex_macros/src/lib.rs#L567

I guess it's clearer where it breaks in docopt: https://github.com/docopt/docopt.rs/blob/master/docopt_macros/src/macro.rs#L234

eddyb · 2016-03-26T14:01:56Z

@BurntSushi The original changes would make parser.eat(&token::Eof) always ICE - but that seemed impractical so I'm trying something which will only trigger on the second EOF bump, not the first one.

kevinmehall · 2016-03-26T15:36:13Z

peg uses parser.eat to obtain the string literal from a peg!("grammar goes here") invocation and check that no additional arguments are passed. If there's a better way to do that, I'm happy to change it. Unstable interfaces are unstable, after all.

eddyb · 2016-03-26T15:47:45Z

@kevinmehall Changing that shouldn't be necessary with my latest attempt - just waiting for travis to confirm everything's good with the tests before starting another crater run.

eddyb · 2016-03-26T20:04:13Z

@nrc See my changes to compile-fail tests. I believe that if we switch away from parsing everything as TTs first we can handle ; closing (, for example:

{ option.map(|some| 42; }

It currently parses as the following TTs:

{ option.map(|some| 42;) }

It accidentally gave relatively sane errors because parse_unspanned_seq was bumping the ; as if it was ) for the method call, resulting in the same effect as this:

{ option.map(|some| 42)) }

I fixed that so it behaves as it should, given what TTs we end up with, but if we didn't parse it to TTs, we could recover a statement based on ; and parse as:

{ option.map(|some| 42); }

Which is arguably what the intention was, and save for a single parse error, the compilation could continue unhindered.

nikomatsakis · 2016-03-27T09:33:28Z

Sounds good

On Sat, Mar 26, 2016 at 05:31:54AM -0700, Eduard-Mihai Burtescu wrote:

@nikomatsakis Actually, that's the current state, but parse_unspanned_seq is giving me trouble because it's unconditionally bumping a token, even if it's not the ket and the travis failure was because that changes how errors are reported (presumably due to the "expected tokens" list which check, used by eat, pushes onto).

I realize right now that I can just handle the error case by not bumping, and I just pushed that change.

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#32479 (comment)

nrc · 2016-03-28T19:29:29Z

lgtm

eddyb · 2016-03-28T21:21:39Z

@bors r=nikomatsakis

bors · 2016-03-28T21:21:41Z

📌 Commit 221d0fb has been approved by nikomatsakis

bors · 2016-03-29T03:50:43Z

⌛ Testing commit 221d0fb with merge a111297...

Prevent bumping the parser past the EOF. Makes `Parser::bump` after EOF into an ICE, forcing callers to avoid repeated EOF bumps. This ICE is intended to break infinite loops where EOF wasn't stopping the loop. For example, the handling of EOF in `parse_trait_items`' recovery loop fixes #32446. But even without this specific fix, the ICE is triggered, which helps diagnosis and UX. This is a `[breaking-change]` for plugins authors who eagerly eat multiple EOFs. See docopt/docopt.rs#171 for such an example and the necessary fix.

bors · 2016-03-29T06:06:07Z

nikomatsakis added beta-nominated Nominated for backporting to the compiler in the beta channel. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Mar 25, 2016

nagisa reviewed Mar 25, 2016
View reviewed changes

eddyb force-pushed the eof-not-even-twice branch 2 times, most recently from 88c6a99 to 27a3e24 Compare March 26, 2016 01:06

eddyb force-pushed the eof-not-even-twice branch from 27a3e24 to 4b1db08 Compare March 26, 2016 12:30

eddyb force-pushed the eof-not-even-twice branch 2 times, most recently from 8012a49 to 64936b9 Compare March 26, 2016 13:58

eddyb force-pushed the eof-not-even-twice branch from 64936b9 to 140210b Compare March 26, 2016 15:11

nagisa mentioned this pull request Mar 26, 2016

Flood of unexpected token: <eof> errors triggered by foo(|_|) #32505

Closed

eddyb added 2 commits March 26, 2016 21:03

syntax: Prevent bumping the parser EOF to stop infinite loops.

6abab49

syntax: Stop the bump loop for trait items at } and EOF.

221d0fb

eddyb force-pushed the eof-not-even-twice branch from 140210b to 221d0fb Compare March 26, 2016 19:53

eddyb mentioned this pull request Mar 28, 2016

Use parse_seq_to_before_end to keep the end token. docopt/docopt.rs#171

Merged

bors merged commit 221d0fb into rust-lang:master Mar 29, 2016

eddyb deleted the eof-not-even-twice branch March 29, 2016 06:07

pmarcelll mentioned this pull request Mar 30, 2016

macro! compilation is broken #32609

Closed

nrc added the beta-accepted Accepted for backporting to the compiler in the beta channel. label Mar 31, 2016

pnkfelix mentioned this pull request Apr 7, 2016

Parser backports for beta #32795

Merged

alexcrichton removed the beta-nominated Nominated for backporting to the compiler in the beta channel. label Apr 7, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prevent bumping the parser past the EOF. #32479

Prevent bumping the parser past the EOF. #32479

eddyb commented Mar 25, 2016

nikomatsakis commented Mar 25, 2016

bors commented Mar 25, 2016

nikomatsakis commented Mar 25, 2016

nikomatsakis commented Mar 25, 2016

nagisa Mar 25, 2016

eddyb Mar 25, 2016

nikomatsakis commented Mar 25, 2016

eddyb commented Mar 25, 2016

nikomatsakis commented Mar 26, 2016

eddyb commented Mar 26, 2016

BurntSushi commented Mar 26, 2016

eddyb commented Mar 26, 2016

kevinmehall commented Mar 26, 2016

eddyb commented Mar 26, 2016

eddyb commented Mar 26, 2016

nikomatsakis commented Mar 27, 2016

nrc commented Mar 28, 2016

eddyb commented Mar 28, 2016

bors commented Mar 28, 2016

bors commented Mar 29, 2016

bors commented Mar 29, 2016

Prevent bumping the parser past the EOF. #32479

Prevent bumping the parser past the EOF. #32479

Conversation

eddyb commented Mar 25, 2016

nikomatsakis commented Mar 25, 2016

bors commented Mar 25, 2016

nikomatsakis commented Mar 25, 2016

nikomatsakis commented Mar 25, 2016

nagisa Mar 25, 2016

Choose a reason for hiding this comment

eddyb Mar 25, 2016

Choose a reason for hiding this comment

nikomatsakis commented Mar 25, 2016

eddyb commented Mar 25, 2016

nikomatsakis commented Mar 26, 2016

eddyb commented Mar 26, 2016

BurntSushi commented Mar 26, 2016

eddyb commented Mar 26, 2016

kevinmehall commented Mar 26, 2016

eddyb commented Mar 26, 2016

eddyb commented Mar 26, 2016

nikomatsakis commented Mar 27, 2016

nrc commented Mar 28, 2016

eddyb commented Mar 28, 2016

bors commented Mar 28, 2016

bors commented Mar 29, 2016

bors commented Mar 29, 2016