Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

async/await: awaiting inside a match block captures borrow too eagerly #57017

Open
Tracked by #69663
seanmonstar opened this issue Dec 20, 2018 · 25 comments · Fixed by #94309
Open
Tracked by #69663

async/await: awaiting inside a match block captures borrow too eagerly #57017

seanmonstar opened this issue Dec 20, 2018 · 25 comments · Fixed by #94309
Labels
A-async-await Area: Async & Await A-lifetimes Area: lifetime related A-mir Area: Mid-level IR (MIR) - https://blog.rust-lang.org/2016/04/19/MIR.html AsyncAwait-Triaged Async-await issues that have been triaged during a working group meeting. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@seanmonstar
Copy link
Contributor

seanmonstar commented Dec 20, 2018

If you use await!(some_fut) inside an arm of a match X, the generated future eagerly borrows the value of X, if it not needed.

This may not usually be noticeable, but the issue compounds when the type X contains a trait object, and the future you wish to return is impl Future + Send. This causes a misleading error message that "dyn Trait + Send cannot be shared between threads", which is required to for &X: Send.

Example

Here's a simple struct with a trait object:

struct Client(Box<Any + Send>);

Consider a function like this:

impl Client {
    fn status(&self) -> u16 {
        200
    }
}

You could consider using a match to determine what kind of future to await (or what arguments to pass):

async fn get() {
}

pub fn wat() -> impl Future + Send {
    let client = Client(Box::new(true));
    async move {
        match client.status() {
            200 => {
                let _x = await!(get());
            },
            _ => (),
        }
    }
}

If the await is moved out of the match block, all is well:

pub fn ok() -> impl Future + Send {
    let client = Client(Box::new(true));
    async move {
        if client.status() == 200 {
            let _x = await!(get());
        }
    }
}

The wat function causes this compilation error:

error[E0277]: `(dyn std::any::Any + std::marker::Send + 'static)` cannot be shared between threads safely
  --> src/main.rs:21:17
   |
21 | pub fn wat() -> impl Future + Send {
   |                 ^^^^^^^^^^^^^^^^^^ `(dyn std::any::Any + std::marker::Send + 'static)` cannot be shared between threads safely
   |
   = help: the trait `std::marker::Sync` is not implemented for `(dyn std::any::Any + std::marker::Send + 'static)`
   = note: required because of the requirements on the impl of `std::marker::Sync` for `std::ptr::Unique<(dyn std::any::Any + std::marker::Send + 'static)>`
   = note: required because it appears within the type `std::boxed::Box<(dyn std::any::Any + std::marker::Send + 'static)>`
   = note: required because it appears within the type `Client`
   = note: required because of the requirements on the impl of `for<'r> std::marker::Send` for `&Client`
   = note: required because it appears within the type `for<'r> {Client, &'r Client, u16, impl std::future::Future, ()}`
   = note: required because it appears within the type `[static generator@src/main.rs:23:16: 30:6 client:Client for<'r> {Client, &'r Client, u16, impl std::future::Future, ()}]`
   = note: required because it appears within the type `std::future::GenFuture<[static generator@src/main.rs:23:16: 30:6 client:Client for<'r> {Client, &'r Client, u16, impl std::future::Future, ()}]>`
   = note: required because it appears within the type `impl std::future::Future`
   = note: the return type of a function must have a statically known size

Playground: https://play.rust-lang.org/?version=nightly&mode=debug&edition=2018&gist=2a9dbea32d31457d50d40b99c52ee214 (updated to latest syntax -Niko)

@Centril Centril added the A-async-await Area: Async & Await label Dec 20, 2018
@cramertj
Copy link
Member

This is just an artifact of the compiler holding the borrow of client in your match expression for the duration of the match condition-- there's nothing async/await!-specific going on here. There's a potential fix to be made to eliminate the borrow of client sooner, but that's a general MIR representation issue.

@cramertj cramertj added A-mir Area: Mid-level IR (MIR) - https://blog.rust-lang.org/2016/04/19/MIR.html A-lifetimes Area: lifetime related labels Dec 21, 2018
@cramertj
Copy link
Member

("liveness" is really the tag I want, but we don't have one for that ;) )

@jonas-schievink
Copy link
Contributor

This might be a duplicate of #46525?

@seanmonstar
Copy link
Contributor Author

I'm not certain it's only that; it does seem related to await!. The compiler seems to be fine with give up the borrow if I call a self or &mut self method inside the match.

@nikomatsakis nikomatsakis added the AsyncAwait-Triaged Async-await issues that have been triaged during a working group meeting. label Mar 5, 2019
@nikomatsakis
Copy link
Contributor

Marking this as deferred -- the problem here (discussed here on Zulip) has to do with how we use an "overapproximation" to decide what data may be inside a generator, at least initially. This is based on the HIR. Only later do we get more precise results when generating MIR.

I think we could resolve this by making that over-appoximation more precise, but @Zoxc expressed a desire to do away with it altogether. This would obviously be better, if we can make it work, though I'm a bit skeptical. (See the Zulip topic for more details.)

@nikomatsakis nikomatsakis added T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. AsyncAwait-Unclear labels Apr 16, 2019
@nikomatsakis
Copy link
Contributor

I'm adding the AsyncAwait-Unclear label, because I'd like to revisit this question and try to make some progress here before we stabilize, but I'm not willing to call it blocking yet =)

@sdroege
Copy link
Contributor

sdroege commented Jun 5, 2019

Another instance of this from #61211. Doesn't only affect what is borrowed there but also the trait bounds of the generated future.

    let foo = Mutex::new("123");

    let foo: Box<dyn futures::future::Future<Output = ()> + Send> = Box::new(async move {
        // In a separate block works
        // {
        let bar = foo.lock().unwrap();
        drop(bar);
        // }
        futures::future::lazy(|_| ()).await;
    });

Fails with

error[E0277]: `std::sync::MutexGuard<'_, &str>` cannot be sent between threads safely

@Mark-Simulacrum
Copy link
Member

FWIW it took me ~30 minutes or so to decide that this was likely a compiler "bug" with async/await, after spending 10-15 minutes reducing the amount of code I had added to just one match statement. The errors are long -- and largely unintelligible. I'm not sure if I'd call this a blocker, but I would say that if it's not fixed, we should definitely include it in a FAQ section of "common errors" or something along those lines.

Here's a gist of the error. There is no clear mention of the actual source of the error, which is in a different crate (this snippet points at main.rs, which is a thin shim over the larger crate rooted at main.rs) that compiled fine(!) -- that means that this might be introduced accidentally in a library and not discovered until a client attempts to use said function in a manner requiring Sync/Send, for example.

@inv2004
Copy link

inv2004 commented Jul 12, 2019

@Mark-Simulacrum
Copy link
Member

My understanding is yes; but I can't be certain.

@nikomatsakis nikomatsakis added AsyncAwait-OnDeck AsyncAwait-Polish Async-await issues that are part of the "polish" area and removed AsyncAwait-Triaged Async-await issues that have been triaged during a working group meeting. AsyncAwait-OnDeck labels Oct 1, 2019
@nikomatsakis
Copy link
Contributor

@rustbot claim

I'm assigning this to myself not to necessarily fix but to organize a call where we can dig a bit into what it would take to fix this, especially if we can move the MIR construction earlier.

@eholk
Copy link
Contributor

eholk commented Jul 26, 2021

I'm looking into this a little now. I noticed there was a difference between the match and if versions of the wat and ok functions and decided to try if let:

pub fn wat() -> impl Future + Send {
    let client = Client(Box::new(true));
    async move {
        if let 200 = client.status() {
            get().await;
        }
    }
}

This also fails. Maybe this isn't too informative though, since if let desugars into a match.

@eholk
Copy link
Contributor

eholk commented Aug 2, 2021

@rustbot claim

@eholk
Copy link
Contributor

eholk commented Aug 16, 2021

Here's a shorter test case that illustrates the issue. This one is based around yield, so it avoids some of the extra syntax introduced by desugaring await. It makes the logs a lot easier to follow.

#![feature(generators, generator_trait, negative_impls)]

struct Client;

impl !Sync for Client {}

fn status(_client_status: &Client) -> i16 {
    200
}

fn assert_send<T: Send>(_thing: T) {}

fn main() {
    let g = move || match status(&Client) {
        _status => yield,
    };
    assert_send(g);
}

@tmandry tmandry moved this from Blocked to In progress in wg-async work Aug 20, 2021
eholk added a commit to eholk/rust that referenced this issue Sep 16, 2021
@tmandry tmandry removed the AsyncAwait-Polish Async-await issues that are part of the "polish" area label Sep 20, 2021
@eholk eholk moved this from In progress (current sprint) to In Progress (long term) in wg-async work Sep 30, 2021
@eholk eholk moved this from In Progress (long term) to On deck in wg-async work Oct 14, 2021
@eholk eholk removed their assignment Oct 28, 2021
@estebank
Copy link
Contributor

estebank commented Dec 9, 2021

Current output:

error: future cannot be sent between threads safely
  --> src/main.rs:19:17
   |
19 | pub fn wat() -> impl Future + Send {
   |                 ^^^^^^^^^^^^^^^^^^ future created by async block is not `Send`
   |
   = help: the trait `Sync` is not implemented for `(dyn Any + Send + 'static)`
note: future is not `Send` as this value is used across an await
  --> src/main.rs:24:26
   |
22 |         match client.status() {
   |               ------ has type `&Client` which is not `Send`
23 |             200 => {
24 |                 let _x = get().await;
   |                          ^^^^^^^^^^^ await occurs here, with `client` maybe used later
...
28 |     }
   |     - `client` is later dropped here
help: consider moving this into a `let` binding to create a shorter lived borrow
  --> src/main.rs:22:15
   |
22 |         match client.status() {
   |               ^^^^^^^^^^^^^^^

Following the suggestion solves the problem with the code, but the match block borrowing should still support the original code.

@nikomatsakis
Copy link
Contributor

So excited this is closed! Nice work @eholk

flip1995 pushed a commit to flip1995/rust that referenced this issue Mar 24, 2022
[generator_interior] Be more precise with scopes of borrowed places

Previously the generator interior type checking analysis would use the nearest temporary scope as the scope of a borrowed value. This ends up being overly broad for cases such as:

```rust
fn status(_client_status: &Client) -> i16 {
    200
}

fn main() {
    let client = Client;
    let g = move || match status(&client) {
        _status => yield,
    };
    assert_send(g);
}
```

In this case, the borrow `&client` could be considered in scope for the entirety of the `match` expression, meaning it would be viewed as live across the `yield`, therefore making the generator not `Send`.

In most cases, we want to use the enclosing expression as the scope for a borrowed value which will be less than or equal to the nearest temporary scope. This PR changes the analysis to use the enclosing expression as the scope for most borrows, with the exception of borrowed RValues which are true temporary values that should have the temporary scope. There's one further exception where borrows of a copy such as happens in autoref cases also should be ignored despite being RValues.

Joint work with `@nikomatsakis`

Fixes rust-lang#57017

r? `@tmandry`
bors added a commit to rust-lang-ci/rust that referenced this issue May 20, 2022
…-guard, r=nikomatsakis

generator_interior: Count match pattern bindings as borrowed for the whole guard expression

The test case `yielding-in-match-guard.rs` was failing with `-Zdrop-tracking` enabled. The reason is that the copy of a local (`y`) was not counted as a borrow in typeck, while MIR did consider this as borrowed.

The correct thing to do here is to count pattern bindings are borrowed for the whole guard. Instead, what we were doing is to record the type at the use site of the variable and check if the variable comes from a borrowed pattern. Due to the fix for rust-lang#57017, we were considering too small of a scope for this variable, which meant it was not counted as borrowed.

Because we now unconditionally record the borrow, rather than only for bindings that are used, this PR is also able to remove a lot of the logic around match bindings that was there before.

r? `@nikomatsakis`
@eholk
Copy link
Contributor

eholk commented Jul 7, 2022

Ugh, so it looks like this isn't actually fixed yet. Playground

use std::any::Any;
use std::future::Future;

struct Client(Box<dyn Any + Send>);

impl Client {
    fn status(&self) -> u16 {
        200
    }
}

async fn get() {
}

pub fn wat() -> impl Future + Send {
    let client = Client(Box::new(true));
    async move {
        match client.status() {
            200 => {
                let _x = get().await;
            },
            _ => (),
        }
    }
}

Output:

   Compiling playground v0.0.1 (/playground)
error: future cannot be sent between threads safely
  --> src/lib.rs:15:17
   |
15 | pub fn wat() -> impl Future + Send {
   |                 ^^^^^^^^^^^^^^^^^^ future created by async block is not `Send`
   |
   = help: the trait `Sync` is not implemented for `(dyn Any + Send + 'static)`
note: future is not `Send` as this value is used across an await
  --> src/lib.rs:20:31
   |
18 |         match client.status() {
   |               ------ has type `&Client` which is not `Send`
19 |             200 => {
20 |                 let _x = get().await;
   |                               ^^^^^^ await occurs here, with `client` maybe used later
...
24 |     }
   |     - `client` is later dropped here
help: consider moving this into a `let` binding to create a shorter lived borrow
  --> src/lib.rs:18:15
   |
18 |         match client.status() {
   |               ^^^^^^^^^^^^^^^

error: could not compile `playground` due to previous error

@wesleywiser wesleywiser reopened this Jul 8, 2022
wg-async work automation moved this from Done to On deck Jul 8, 2022
@danielhenrymantilla
Copy link
Contributor

I'm a bit confused by what is the part considered problematic and deemed fix-worty.

match / if let semantics state that temporaries be dropped / "go out of scope" at the end of the match / if let block.

So match client.status() {… becomes:

{ let temporary = &client;
    match temporary.status() { 200 => {}}
} // <- temporary dropped here

which can be further simplified down to:

let temporary = &client;
if temporary.status() == 200 {/* stuff, such as a yield or await point */
}

So the problem in this issue becomes that of temporary above crossing a yield/.await. This, thus, by definition, requires that they be captured by the generator, but since this is just an "automatic out-of-scope" operation, it is semantically meaningless, according to NLL:

  • Incidentally, this NLL observation is what allows, in both this simplification, and the original match code, for other borrows to occur, or even for the borrowee to be moved / dropped. The temporary binding, technically StorageLive-allocated in the "stack", ends up "stack"-deallocated / StorageDead, and yet this does not require the borrow to still be usable at that point.

So I guess the whole issue is about "making generator captures NLL-aware", of sorts?

@eholk
Copy link
Contributor

eholk commented Jul 11, 2022

So I guess the whole issue is about "making generator captures NLL-aware", of sorts?

I think that's a good way of looking at it.

As you point out, @danielhenrymantilla, technically temporarily does live across the await point. But users find this surprising and it's not observable (other than in the generator type) if we end the borrow earlier, so we'd like to have rules that allow this program to work.

@jyn514
Copy link
Member

jyn514 commented Apr 27, 2023

This will be fixed by -Zdrop-tracking, I think. #97331

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-async-await Area: Async & Await A-lifetimes Area: lifetime related A-mir Area: Mid-level IR (MIR) - https://blog.rust-lang.org/2016/04/19/MIR.html AsyncAwait-Triaged Async-await issues that have been triaged during a working group meeting. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
Status: On deck