async move breaks doc tests

[ghost]

Steps to reproduce:

1. Create a library project with the following contents in lib.rs:

//! ```
//! #![feature(async_await)]
//!
//! fn foo() {
//!     drop(async move {});
//! }
//! ```

2. Run tests:

repro master $ cargo --version
cargo 1.35.0-nightly (0e35bd8af 2019-03-13)
repro master $ rustc --version
rustc 1.35.0-nightly (3eb4890df 2019-03-19)
repro master $ cargo test
    Finished dev [unoptimized + debuginfo] target(s) in 0.00s
     Running target/debug/deps/repro-0e401502cf52c662

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

   Doc-tests repro

running 1 test
test src/lib.rs -  (line 1) ... FAILED

failures:

failures:
    src/lib.rs -  (line 1)

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '--doc'

3. Now delete the move inside async move {}:

//! ```
//! #![feature(async_await)]
//!
//! fn foo() {
//!     drop(async {});
//! }
//! ```

4. Run tests again:

repro master $ cargo test
   Compiling repro v0.1.0 (/home/stjepan/work/repro)
    Finished dev [unoptimized + debuginfo] target(s) in 0.25s
     Running target/debug/deps/repro-0e401502cf52c662

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

   Doc-tests repro

running 1 test
test src/lib.rs -  (line 1) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

[nikomatsakis]

This seems to work outside of rustdoc

[nikomatsakis]

Marking as blocking for async await stabilization pending some investigation.

cc @rust-lang/rustdoc -- can you all give any tips to try and gain insight into what error is occurring etc?

[GuillaumeGomez]

Can you give a try to:

//! #![feature(async_await)]
//!
//! # fn main() {
//! fn foo() {
//!     drop(async move {});
//! }
//! # foo();
//! # }

But beyond that, I'm wondering if it isn't just a missing flag or something along the line?

Maybe you have other ideas @QuietMisdreavus ?

[QuietMisdreavus]

I spent some time yesterday poking at this, and it seems like it's failing somewhere in the pre-parsing of the doctest. Some time around the part where it parses the move keyword, it just... dies. I'm having trouble pinning down exactly where (and how) it fails, since it's not panicking, and setting -Z treat-err-as-bug (which i have to modify the code to do since we usually don't pass that into doctests) makes it fail at an earlier spot.

[nikomatsakis]

I've sometimes seen problems around FatalError.raise()

[QuietMisdreavus]

Funny you should say that... It looks like one of those is being hit when processing this sample:

rust/src/libsyntax/parse/parser.rs

Lines 791 to 792 in 8260e96

	} else if self.last_unexpected_token_span == Some(self.span) {
	FatalError.raise();

It doesn't seem to hit this when compiling regularly, though.

[QuietMisdreavus]

I think i've figured it out. The "pre-parse" step of doctest execution runs the parser separately from the later execution of it. In this run, the "default edition" is never set, so libsyntax assumes everything is being run under Rust 2015. This causes it to fail when parsing the async move block. What's confusing is how this has never been a problem until now... 🤔

[QuietMisdreavus]

I've got a PR posted that should fix this: #60065

[Nemo157]

Hmm, I was going to suggest that might be the issue, but I tried a test containing async { let foo = 5; } (which fails to compile on edition 2015) and that works fine (unsurprisingly since futures use a lot of doc-tests containing something similar).

[QuietMisdreavus]

Plain async blocks probably pass in the current nightly because they fail after the initial parse, or perhaps they bubble up a single error from parse_item instead of emitting it directly, which causes rustdoc to stop parsing and continue creating the doctest. The difference with async move blocks is that they use a keyword that exists in 2015, which causes it to attempt parsing a closure (instead of a struct expression), so it totally fails to parse instead of just creating a real expression that would fail a later check.