Tracking issue for incorrect lifetime bound errors in async

[tmandry]

Reported issues

Beta Give feedback

1. Async function leads to a "more general type" error #71723
   A-async-await A-typesystem AsyncAwait-Triaged C-bug T-compiler +5
   
2. Bad Error Message: error[E0277]: the trait bound T: Generator<ResumeTy> is not satisfied #79648
   A-async-await A-diagnostics AsyncAwait-Triaged C-bug T-compiler +5
3. Async fn disagrees on its own return type ("one of the expected opaque types" vs "one of the found opaque types") #82921
   A-async-await AsyncAwait-Triaged C-bug T-compiler +4
4. Problem with GATs, async, and Send-bounds #90696
   A-associated-items A-async-await A-lifetimes A-traits AsyncAwait-Triaged C-bug F-generic_associated_types GATs-triaged T-compiler requires-nightly +10
5. Lifetime bounds in auto trait impls prevent trait from being implemented on generators #64552
   A-async-await A-coroutines A-traits AsyncAwait-Triaged C-bug P-medium T-compiler T-types +8
   
6. implementation of Debug is not general enough when making async block into &dyn Future + Send #87425
   A-async-await A-lifetimes AsyncAwait-Triaged C-bug T-compiler WG-async fixed-by-next-solver +7
7. Implementation of trait is not general enough when boxing future returning <T as Foo<'a>>::Foo #92415
   A-async-await AsyncAwait-Polish AsyncAwait-Triaged C-bug +4
8. GAT Self: 'a bounds break some async blocks with impl Trait #92096
   C-bug F-generic_associated_types GATs-triaged +3
9. Bogus higher-ranked lifetime error in an async block #102211
   A-async-await A-lifetimes AsyncAwait-Triaged C-bug T-compiler +5
10. Wrong Send constraint when using a trait with lifetime + associated type in async #60658
    A-async-await A-diagnostics AsyncAwait-Triaged C-enhancement T-compiler +5
11. Unhelpful error "one type is more general than the other" in async code #64650
    A-async-await A-diagnostics AsyncAwait-Triaged C-enhancement D-confusing D-newcomer-roadblock T-compiler +7
12. Why implementation of iterator is not generic enough in async context? #71671
    A-async-await A-diagnostics AsyncAwait-Triaged C-bug D-confusing T-compiler +6
13. "Implementation not general enough" with async and projection, fixed by wrapping with async move #110339
    A-async-await A-coroutines AsyncAwait-Triaged T-compiler T-types +5
14. async block with futures::stream::Buffered is not Send #104382
    +0
15. RPITIT with Send trait marker breaks borrow checker #111105
    A-borrow-checker C-bug F-return_position_impl_trait_in_trait +3
16. Higher ranked lifetime error when tryng to see that a future is send. #114046
    A-async-await AsyncAwait-Triaged C-bug T-compiler T-types +5

Cause

We erase lifetime relations of types inside generators, making it impossible in some cases to prove bounds or normalize associated types.

See #64552 (comment) for a more detailed explanation.

Implementation history

• Correctly check auto traits on generator interiors #92449

Notes

Please keep discussion in this thread at a "meta-level" and open new issues for code that does not compile.

These might not all have the same cause, though it seems plausible that they do. My primary motivation is to try to group all the related issues together so they're easier to keep track of. If we can split them into sub-groups, all the better.

Many of the issues in this list are cribbed from #92449, thanks to @compiler-errors for coming up with that list.

[compiler-errors]

For the record, the approach laid out in #64552 (comment) is not sufficient.

It's essentially what I attempted in #92449, which did not fix all of the issues listed, especially after I actually made it (kinda) sound in "attempt 2" in the PR by using placeholder types rather than region infer vars. Some additional discussion lives in https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Generator.20interior.20auto.20traits.

It's also not clear that the approach in my PR is even sound (#92449 (comment)). Any attempt to fix this issue should probably run their approach by T-types before wasting much time on it, unless you want to be like me and waste a ton of time on it 😅

[compiler-errors]

These might not all have the same cause, though it seems plausible that they do.

Yes and no.

They all have the same "root cause", which is that we erase lifetime information from generator witnesses, which causes us to no longer be able to prove certain lifetime requirements that are needed to prove that generators implement certain auto traits.

This is a very general problem, and does not have a single solution to fix it, IMO. That's because the way that these lifetime requirements actually come about are incredibly diverse, though. Revisiting exactly how each of these issues manifests will be necessary for any solution to this issue.

[tmandry]

Thanks for your comments @compiler-errors, very helpful!

That's because the way that these lifetime requirements actually come about are incredibly diverse, though. Revisiting exactly how each of these issues manifests will be necessary for any solution to this issue.

Do you think sorting issues into camps like "failed normalization", "requirement from a trait impl" would help? (I guess a single error might satisfy multiple of these.)

[safinaskar]

Attempt to simulate async lambdas using lambdas, which return futures, leads to lifetime errors, as explained in this Niko's post: https://smallcultfollowing.com/babysteps/blog/2023/03/29/thoughts-on-async-closures/

[emersonford]

It wasn't immediately clear to me how async + generators + auto traits + HRTBs tie together to result in this bug, so wanted to give a shot at documenting it for folks not as familiar with rustc's type internals as this seems to somewhat common in async programming that requires Send futures (and please correct me if I'm wrong!).

Async fns are effectively desugared into generators, so in something like

fn foo() -> impl Future<Output = i64> { // effectively the same signature as `async fn foo() -> i64 { ... }`
    async {
        ready(10).await
    }
}

the impl Future<Output = i64> opaque return type will be resolved as some concrete generator type (similar to the SumGenerator struct in the blog post). You can see this in this playground using the "Show HIR" functionality.

So when you want the future to be send (e.g. using -> impl Future<Output = i64> + Send as the return type or having T: Future<Output = i64> + Send as the parameter type as found in tokio::spawn), you need to prove that the opaque generator type is also Send. Since Send is an auto trait, the type checker will automatically try to prove that the opaque generator type is Send by inspecting the internals of the generator. The bug occurs at this step, specifically when some internal value of a generator requires a trait implementation with some lifetime bound. For example, using @danielhenrymantilla's example here:

trait Trait { type Assoc; }

struct Foo<T : Trait>(T::Assoc);

impl Trait for fn(&'static ()) {
    type Assoc = ();
}

fn main() {
    let sendable_future: &dyn Send = &async {
        let s = Foo::<fn(&'static ())>(());
        async{}.await;
    };
}

taking it step by step:

1. To prove that the opaque generator type for sendable_future is Send, you need to prove the internals are also Send, i.e. you need to prove that s is Send.
2. To prove a given struct is Send, all of its member fields must be Send. Thus for Foo, you must prove that T::Assoc is Send.
3. In order to do this though, you must first prove the T: Trait bound of struct Foo<T: Trait>(...);. Thus the first step of proving Send for s = Foo::<fn(&'static ())>(()) is proving that Trait is implemented for fn(&'static ()).
4. However, lifetimes in generator interiors are currently being erased(?), so after region replacement, the type checker attempts to prove the bound for<'a> fn(&'static ()): Trait. This obviously doesn't hold, so you end up with the error message

   = note: `fn(&'0 ())` must implement `Trait`, for any lifetime `'0`...
   = note: ...but `Trait` is actually implemented for the type `fn(&'static ())`

You run into the same issue with something like

struct Foo<T>(T);

unsafe impl Send for Foo<&'static str> {}

fn main() {
    let sendable_future: &dyn Send = &async {
        let s = Foo::<&'static str>("");
        async{}.await;
    };
}

which gives you the implementation of 'Send' is not general enough error message as the type checker attempts to prove for<'a> Foo<&'static str>: Send.

So one non-trivial fix here is to fix step 4 and not erase lifetimes inside of generator interiors.

---

What's not immediately clear to me though is why this is so specific to auto traits + generators. Even if you don't need to prove Send for the generator, don't you still need to prove that Trait is implemented for fn(&'static ()) inside of the generator interior? I.e., why does

use std::future::Future;

trait Trait { type Assoc; }

struct Foo<T : Trait>(T::Assoc);

impl Trait for fn(&'static ()) {
    type Assoc = ();
}

fn main() {
    let f: &dyn Future<Output = ()> = &async {
        let s = Foo::<fn(&'static ())>(());
        async{}.await;
    };
}

compile just fine?

[emersonford]

In terms of workarounds:

For something like

trait Trait { type Assoc; }

struct Foo<T : Trait>(T::Assoc);

impl Trait for fn(&'static ()) {
    type Assoc = ();
}

fn main() {
    let sendable_future: &dyn Send = &async {
        let s = Foo::<fn(&'static ())>(());
        async{}.await;
    };
}

you can insert a drop like so:

trait Trait { type Assoc; }

struct Foo<T : Trait>(T::Assoc);

impl Trait for fn(&'static ()) {
    type Assoc = ();
}

fn main() {
    let sendable_future: &dyn Send = &async {
        let s = Foo::<fn(&'static ())>(());
        std::mem::drop(s);
        async{}.await;
    };
}

and enable -Zdrop-tracking, which will omit checking Send for s as part of the generator Send check (since drop tracking will correctly identify that the generator does not hold s across yield points).

---

For something like

let _: &dyn Send = &async {
    let _it = [()].iter().map(|it| None::<()>).flatten();
    async {}.await;
};

you should collect the iterator, e.g.

let _: &dyn Send = &async {
    let collect = [()].iter().map(|it| None::<()>).flatten().collect::<Vec<_>>();
    async {}.await;
};