Generator support #43076

Zoxc · Jul 5, 2017

This adds experimental support for generators intended to land once rust-lang/rfcs#2033 is approved.

This is not yet ready to be merged. Things to do:

Make closure arguments on generators an error
Spot FIXMEs
Pass make tidy
Write tests
Document the current syntax and semantics for generators somewhere
Use proper error message numbers
~~Make the implicit argument type default to ()~~

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

rust-highfive · Jul 5, 2017

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

cc @rust-lang/compiler, I suspect a number of you may be interested in this!

alexcrichton · Jul 5, 2017

cc @rust-lang/compiler, I suspect a number of you may be interested in this!

☔️ The latest upstream changes (presumably #42727) made this pull request unmergeable. Please resolve the merge conflicts.

bors · Jul 6, 2017

☔️ The latest upstream changes (presumably #42727) made this pull request unmergeable. Please resolve the merge conflicts.

nikomatsakis · Jul 7, 2017

nikomatsakis reviewed Jul 7, 2017

View changes

nikomatsakis · Jul 7, 2017

src/librustc/hir/mod.rs

@@ -1059,6 +1071,12 @@ pub enum Expr_ {
    /// For example, `[1; 5]`. The first expression is the element
    /// to be repeated; the second is the number of times to repeat it.
    ExprRepeat(P<Expr>, BodyId),
+
+    /// A suspension point for generators


It would be good to give an example of Rust syntax; this corresponds to a yield, right?

In an older version this was separate from yield, now it's the same and it could use an renaming.

nikomatsakis · Jul 7, 2017

nikomatsakis reviewed Jul 7, 2017

View changes

nikomatsakis · Jul 7, 2017

src/librustc/hir/mod.rs

+    ExprSuspend(P<Expr>),
+
+    /// The argument to a generator
+    ExprImplArg(NodeId),


This corresponds to a gen arg, right? I am wondering if we should think about removing this from the PR, since it is not needed for async-await, and I find it hard to imagine this syntax ultimately being stabilized (it just seems rather unlike any other syntax we have in the language). I'm not sure how much actual simplicity would result though, maybe very little.

I am also against the concept as a whole - it's there to get rid of thread-local state but I do not think it's a satisfactory solution.

Skimming through the PR, it seems like this would be a reasonably nice simplification.

@eddyb

I am also against the concept as a whole - it's there to get rid of thread-local state but I do not think it's a satisfactory solution.

Does this mean you are also against yield returning a value, sort of like let x = yield 22?

Ah, I see what you're hinting at. In my hand-desugared examples I ended up having the generator equivalent of IntoIterator produce the first Yield/CoResult/etc. alongside the generator itself. The other options are multiple entry points, taking Option<Input>, or maybe a different suspend model. Maybe a yield without a value and produce the first input.

Anyway, what I'm against is any such scheme used with async IO, when it can be avoided.

Having explicitly named arguments would be an alternative which is just as expressive and could probably reuse most of the code dealing with regular arguments. One of the problems with this is that if we allow yield inside function and closure bodies in addition to generator literal bodies we need syntax for another set of arguments. This is less of a problem if we're committed to using compiler plugins for ergonomic async I/O.

I have to admit that I am finding these last 2 comments a bit confusing. e.g., @eddyb, when you write:

The other options are multiple entry points

I don't really understand what you are referring to. The other options for.. what exactly? In any case, it seems like neither iterators nor futures require the ability to provide "feedback" during execution, so I would personally be happy to "defer" that part for later PRs.

@Zoxc

Having explicitly named arguments would be an alternative which is just as expressive and could probably reuse most of the code dealing with regular arguments.

I would think so -- those regular arguments wind up being assigned to standard locals in MIR, iirc, almost immediately upon function entry. In fact, I thikn it would "just work" just fine -- unless you meant that, after a yield, the function arguments would be updated "in place" with the new values?

unless you meant that, after a yield, the function arguments would be updated "in place" with the new values?

This is indeed what I meant.

@nikomatsakis By "multiple entry points" I meant that you can also have yield produce a value by providing multiple ways to resume a given generator, the one taking no value for the very first time, and then using the one requiring a value. Using the wrong one would "just" panic. If you do have multiple entry points, though, you can also start doing crazier things, like having multiple types that yield can produce, similar to what ends up being done in dynamic language. However, while quite flexible, the dynamic nature of it makes me uneasy and I'm very glad that yield producing no value back to the generator can be used for async I/O.

nikomatsakis · Jul 7, 2017

nikomatsakis reviewed Jul 7, 2017

View changes

nikomatsakis · Jul 7, 2017

src/librustc/hir/mod.rs

@@ -1014,7 +1026,7 @@ pub enum Expr_ {
    /// A closure (for example, `move |a, b, c| {a + b + c}`).
    ///
    /// The final span is the span of the argument block `|...|`
-    ExprClosure(CaptureClause, P<FnDecl>, BodyId, Span),
+    ExprClosure(CaptureClause, P<FnDecl>, BodyId, Span, Option<GeneratorClause>),


It's worth adding a comment -- what is an Option<GeneratorClause>?

(Also, I wonder if we should convert this to a struct variant at some point.)

nikomatsakis · Jul 7, 2017

nikomatsakis reviewed Jul 7, 2017

View changes

nikomatsakis · Jul 7, 2017

src/librustc/hir/print.rs

+                if gen.is_some() {
+                    self.head("gen")?;
+                    space(&mut self.s)?;
+                }


This isn't the syntax that users use, right? In that case, we probably shouldn't either. Also, it seems weird that we don't print if this is "movable" or "immovable" -- iirc from the last time I looked, that stuff isn't really used in this branch, maybe we should remove it from this PR too.

We need syntax to mark a generator as movable or immovable. I haven't come up with any good ideas for that. The best idea I got is using static for immovable generators. The only syntax that seems to be nailed down is yield

Isn't this idea of movable vs immovable a kind of separate, orthogonal proposal? That is, at the moment, I thought we would be starting with only the "movable" kind...

Immovable types is a separate orthogonal proposal. Immovable generators would be a extension to generators which depend on immovable types. Immovable generators are very important for async I/O experiments however.

I do intend to implement immovable generators, but landing that in master would block on the immovable types RFC.

nikomatsakis · Jul 7, 2017

nikomatsakis reviewed Jul 7, 2017

View changes

nikomatsakis · Jul 7, 2017

src/librustc/ty/sty.rs

@@ -150,6 +150,9 @@ pub enum TypeVariants<'tcx> {
    /// `|a| a`.
    TyClosure(DefId, ClosureSubsts<'tcx>),

+    /// The anonymous type of a generator. Pairs with a TyClosure for closure generators.


I think this comment could be expanded. For example, what does "pairs with" mean? My guess is that this is the type assigned to an expression like || yield 22, right? In other words, it represents a "resumable" function that, each time it is called, picks up from the last yield point? (Only it works through the generator trait, not the function traits?)

This comment is outdated.

nikomatsakis · Jul 7, 2017

nikomatsakis reviewed Jul 7, 2017

View changes

nikomatsakis · Jul 7, 2017

src/librustc/ty/sty.rs

+        impl Iterator<Item=Ty<'tcx>> + 'tcx
+    {
+        let state = tcx.generator_layout(def_id).fields.iter();
+        let state: Vec<_> = state.map(|d| d.ty.subst(tcx, self.substs)).collect();


Silly question, but is it necessary to collect into an intermediate vec here, given that we're returning an impl Iterator?

Using Vec is simpler than dealing with lifetime errors

There shouldn't be any. Can you paste them here?

This is the error I get if I just return state.map(|d| d.ty.subst(tcx, self.substs)):

error[E0477]: the type `core::iter::Map<core::slice::Iter<'_, mir::LocalDecl<'_>>, [closure@src\librustc\ty\sty.rs:290:19: 290:51 tcx:&ty::context::TyCtxt<'a, 'gcx, 'tcx>, self:&ty::sty::ClosureSubsts<'tcx>]>` does not fulfill the required lifetime --> src\librustc\ty\sty.rs:287:9 | 287 | impl Iterator<Item=Ty<'tcx>> + 'tcx | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | note: type must outlive the lifetime 'tcx as defined on the impl at 282:1 --> src\librustc\ty\sty.rs:282:1 | 282 | / impl<'a, 'gcx, 'tcx> ClosureSubsts<'tcx> { 283 | | /// This returns the types of the MIR locals which had to be stored across suspension points. 284 | | /// It is calculated in rustc_mir::transform::generator::StateTransform. 285 | | /// All the types here must be in the tuple in GeneratorInterior. ... | 302 | | } 303 | | } | |_^ error[E0495]: cannot infer an appropriate lifetime for capture of `tcx` by closure due to conflicting requirements --> src\librustc\ty\sty.rs:290:19 | 290 | state.map(|d| d.ty.subst(tcx, self.substs)) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | note: first, the lifetime cannot outlive the lifetime 'a as defined on the impl at 282:1... --> src\librustc\ty\sty.rs:282:1 | 282 | / impl<'a, 'gcx, 'tcx> ClosureSubsts<'tcx> { 283 | | /// This returns the types of the MIR locals which had to be stored across suspension points. 284 | | /// It is calculated in rustc_mir::transform::generator::StateTransform. 285 | | /// All the types here must be in the tuple in GeneratorInterior. ... | 302 | | } 303 | | } | |_^ note: ...so that the reference type `&ty::context::TyCtxt<'a, 'gcx, 'tcx>` does not outlive the data it points at --> src\librustc\ty\sty.rs:290:9 | 290 | state.map(|d| d.ty.subst(tcx, self.substs)) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ note: but, the lifetime must be valid for the lifetime 'tcx as defined on the impl at 282:1... --> src\librustc\ty\sty.rs:282:1 | 282 | / impl<'a, 'gcx, 'tcx> ClosureSubsts<'tcx> { 283 | | /// This returns the types of the MIR locals which had to be stored across suspension points. 284 | | /// It is calculated in rustc_mir::transform::generator::StateTransform. 285 | | /// All the types here must be in the tuple in GeneratorInterior. ... | 302 | | } 303 | | } | |_^ note: ...so that the type `core::iter::Map<core::slice::Iter<'_, mir::LocalDecl<'_>>, [closure@src\librustc\ty\sty.rs:290:19: 290:51 tcx:&ty::context::TyCtxt<'a, 'gcx, 'tcx>, self:&ty::sty::ClosureSubsts<'tcx>]>` will meet its required lifetime bounds --> src\librustc\ty\sty.rs:287:9 | 287 | impl Iterator<Item=Ty<'tcx>> + 'tcx | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ error: aborting due to previous error(s)

error[E0495]: cannot infer an appropriate lifetime for capture of tcx by closure due to conflicting requirements

You forgot a move.

state.map(move |d| d.ty.subst(tcx, self.substs)) gives this error:

error[E0477]: the type `core::iter::Map<core::slice::Iter<'_, mir::LocalDecl<'_>>, [closure@src\librustc\ty\sty.rs:289:19: 289:56 tcx:ty::context::TyCtxt<'a, 'gcx, 'tcx>, self:ty::sty::ClosureSubsts<'tcx>]>` does not fulfill the required lifetime --> src\librustc\ty\sty.rs:287:9 | 287 | impl Iterator<Item=Ty<'tcx>> + 'tcx { | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | note: type must outlive the lifetime 'tcx as defined on the impl at 282:1 --> src\librustc\ty\sty.rs:282:1 | 282 | / impl<'a, 'gcx, 'tcx> ClosureSubsts<'tcx> { 283 | | /// This returns the types of the MIR locals which had to be stored across suspension points. 284 | | /// It is calculated in rustc_mir::transform::generator::StateTransform. 285 | | /// All the types here must be in the tuple in GeneratorInterior. ... | 301 | | } 302 | | } | |_^ error: aborting due to previous error(s)

You need move and + 'a I think (to capture the 'a lifetime in 'tcx).

I mean, to change the return type to impl Iterator<Item=Ty<'tcx>> + 'a and use a move closure. This error message really needs to be improved.

nikomatsakis · Jul 7, 2017

nikomatsakis reviewed Jul 7, 2017

View changes

nikomatsakis · Jul 7, 2017

src/librustc/ty/sty.rs

@@ -276,6 +279,50 @@ impl<'a, 'gcx, 'acx, 'tcx> ClosureSubsts<'tcx> {
    }
 }

+impl<'a, 'gcx, 'tcx> ClosureSubsts<'tcx> {
+    pub fn state_tys(self, def_id: DefId, tcx: TyCtxt<'a, 'gcx, 'tcx>) ->


I think this function could use a comment. I am guessing it returns the types of local variables that are (potentially) saved/restore in the state?

nikomatsakis · Jul 7, 2017

nikomatsakis reviewed Jul 7, 2017

View changes

nikomatsakis · Jul 7, 2017

src/librustc/ty/sty.rs

+        state.into_iter()
+    }
+
+    pub fn field_tys(self, def_id: DefId, tcx: TyCtxt<'a, 'gcx, 'tcx>) ->


Similarly, what is field_tys? Comment seems good.

nikomatsakis · Jul 7, 2017

nikomatsakis reviewed Jul 7, 2017

View changes

nikomatsakis · Jul 7, 2017

src/librustc_borrowck/borrowck/gather_loans/mod.rs

@@ -201,6 +219,19 @@ fn check_aliasability<'a, 'tcx>(bccx: &BorrowckCtxt<'a, 'tcx>,
    }
 }

+fn check_yields<'a, 'tcx>(bccx: &BorrowckCtxt<'a, 'tcx>,


I would expect this check to be done in the check_loans code -- basically, each time we reach a yield point, we would check that no loans are in scope.

Could you point me to an appropriate location in the check_loans code?

Well, I forgot that check_loans uses the ExprUseVisitor. I would add a "callback" to that interface for "yields". So, specifically the expr_use_visitor::Delegate interface. We could add a fn yield(id: ast::NodeId) method. Then, on each callback, we could check whether there are any loans in scope at that point, e.g. by calling CheckLoans::each_in_scope_loan().

(Sorry, missed this comment till just now.)

I was looking around there and that doesn't seem correct. No loan records are created when we get RestrictionResult::Safe, so I assume that means the loans don't show up with CheckLoans::each_in_scope_loan() leading to unsoundness. We also want to ignore ReEarlyBound and ReFree borrows and it looks like that information is lost by then.

☔️ The latest upstream changes (presumably #42996) made this pull request unmergeable. Please resolve the merge conflicts.

bors · Jul 8, 2017

☔️ The latest upstream changes (presumably #42996) made this pull request unmergeable. Please resolve the merge conflicts.

Zoxc · Jul 10, 2017

Zoxc reviewed Jul 10, 2017

View changes

Zoxc · Jul 10, 2017

src/doc/unstable-book/src/language-features/generators.md

+  types and such.
+
+* Traits like `Send` and `Sync` are automatically implemented for a `Generator`
+  depending on the captured variables of the environment. Note, though, that


Unlike closures, this also depends on the values inside the generator which are live across a suspension point.

apasel422 · Jul 10, 2017

apasel422 reviewed Jul 10, 2017

View changes

apasel422 · Jul 10, 2017

src/doc/unstable-book/src/language-features/generators.md

+  generator progresses.
+
+* Generator literals produce a value with a unique type which implements the
+  `std::ops::Generator` trait. This allows actual execution of the genrator


s/genrator/generator/

apasel422 · Jul 10, 2017

src/libcore/ops/mod.rs

@@ -189,6 +190,13 @@ pub use self::range::{RangeInclusive, RangeToInclusive};
 #[unstable(feature = "try_trait", issue = "42327")]
 pub use self::try::Try;

+#[unstable(feature = "generator_trait", issue = "0")]
+#[cfg(not(stage0))]
+pub use self::generator::State;


core::ops::State on its own has no connection to generators, so we might want to export this with a more descriptive name or introduce a core::generator module, though this can certainly wait until the future RFC that stabilizes this feature.

It is in a generator module. Perhaps we should remove the reexport of it?

Is a simpler to remember syntax like this possible?

fn test() -> impl GeneratorNoReturn<u8> {
   for i in 1 .. 6 {
       yield i
   }
}

leonardo-m · Jul 11, 2017

Is a simpler to remember syntax like this possible?

fn test() -> impl GeneratorNoReturn<u8> {
   for i in 1 .. 6 {
       yield i
   }
}

So I've read over the code again -- not in depth -- and I remain fairly positive. @Zoxc I want to say that this is nice work. =) That said, I still think we could (and should) try to simplify it as much as possible -- it's basically impossible to review a branch of this size, so the more we can do to prune it down (and then add things in separately later) the better, since those additions can then get reviewed in more depth.

The obvious thing for removal remains gen arg -- it feels like it adds a fair bit of complexity into the PR, and it is not needed for most use cases from what I can tell. (Though I think we may want to add some feature like it back at some point -- I don't personally feel confident of just what I think that feature should look like yet.)

The other candidate is the "movable/immovable" stuff, which if I recall is not (yet) deeply integrated. I'd prefer to layer that in separately as well.

So, before r+, I would like to see:

reach consensus about simplifications
- I remain fairly convinced we should remove them, but if for some reason that is infeasible, I'd like to understand that better
I'd like to give one more read over the borrowck and sensitive bits, and review the tests. =)
- But I don't expect those last reviews to take long on my part.
I'd prefer if @eddyb or somebody else on @rust-lang/compiler wanted to chime in and indicate that they've at least given this a preliminary read and don't have any blocking concerns

nikomatsakis · Jul 11, 2017

So I've read over the code again -- not in depth -- and I remain fairly positive. @Zoxc I want to say that this is nice work. =) That said, I still think we could (and should) try to simplify it as much as possible -- it's basically impossible to review a branch of this size, so the more we can do to prune it down (and then add things in separately later) the better, since those additions can then get reviewed in more depth.

The obvious thing for removal remains gen arg -- it feels like it adds a fair bit of complexity into the PR, and it is not needed for most use cases from what I can tell. (Though I think we may want to add some feature like it back at some point -- I don't personally feel confident of just what I think that feature should look like yet.)

The other candidate is the "movable/immovable" stuff, which if I recall is not (yet) deeply integrated. I'd prefer to layer that in separately as well.

So, before r+, I would like to see:

reach consensus about simplifications
- I remain fairly convinced we should remove them, but if for some reason that is infeasible, I'd like to understand that better
I'd like to give one more read over the borrowck and sensitive bits, and review the tests. =)
- But I don't expect those last reviews to take long on my part.
I'd prefer if @eddyb or somebody else on @rust-lang/compiler wanted to chime in and indicate that they've at least given this a preliminary read and don't have any blocking concerns

I'm updating my prototype lib based on coroutines right now. So I wonder if this kind of code is supposed to be working on this branch?

struct A(u32);

fn g1() -> impl Generator<Yield=(), Return=()> {
    move || {
        let a = A(33);
        let inner = a.g2();
        while let State::Yielded(_) = inner.resume(()) {
            yield
        }
    }
}

impl A {
    fn g2<'a>(&'a self) -> impl Generator<Yield=(), Return=()> + 'a {
        move || {
            println!("use it {:?}", self.0);
            yield
        }
    }
}

rozaliev · Jul 11, 2017

I'm updating my prototype lib based on coroutines right now. So I wonder if this kind of code is supposed to be working on this branch?

struct A(u32);

fn g1() -> impl Generator<Yield=(), Return=()> {
    move || {
        let a = A(33);
        let inner = a.g2();
        while let State::Yielded(_) = inner.resume(()) {
            yield
        }
    }
}

impl A {
    fn g2<'a>(&'a self) -> impl Generator<Yield=(), Return=()> + 'a {
        move || {
            println!("use it {:?}", self.0);
            yield
        }
    }
}

@rozaliev That requires !Move because the self that you refer to from the generator in g2 is of type &'a A, pointing into the state of the generator from g1. You could read self.0 before creating the generator in g2, i.e. this should work:

struct A(u32);

fn g1() -> impl Generator<Yield=(), Return=()> {
    move || {
        let a = A(33);
        let inner = a.g2();
        while let State::Yielded(_) = inner.resume(()) {
            yield
        }
    }
}

impl A {
    fn g2(&self) -> impl Generator<Yield=(), Return=()> {
        let x = self.0;
        move || {
            println!("use it {:?}", x);
            yield
        }
    }
}

eddyb · Jul 11, 2017

@rozaliev That requires !Move because the self that you refer to from the generator in g2 is of type &'a A, pointing into the state of the generator from g1. You could read self.0 before creating the generator in g2, i.e. this should work:

struct A(u32);

fn g1() -> impl Generator<Yield=(), Return=()> {
    move || {
        let a = A(33);
        let inner = a.g2();
        while let State::Yielded(_) = inner.resume(()) {
            yield
        }
    }
}

impl A {
    fn g2(&self) -> impl Generator<Yield=(), Return=()> {
        let x = self.0;
        move || {
            println!("use it {:?}", x);
            yield
        }
    }
}

@eddyb that's what I was afraid of, I guess I should wait for !Move, because A::g2(&self) is actually Conn::read<'a,'b>(&'a mut self, &'b mut [u8]) ... and I do need those lifetimes.

rozaliev · Jul 11, 2017

@eddyb that's what I was afraid of, I guess I should wait for !Move, because A::g2(&self) is actually Conn::read<'a,'b>(&'a mut self, &'b mut [u8]) ... and I do need those lifetimes.

@rozaliev Sounds like a problem Tokio doesn't have, although I don't know the specifics.

eddyb · Jul 11, 2017

@rozaliev Sounds like a problem Tokio doesn't have, although I don't know the specifics.

In response to @nikomatsakis's comment above I've prepared a branch which is the current state of this PR (based on 873cb4f) and then removes support for gen arg. Locally this branch is passing all tests.

I'm hoping that this makes an "easy" starting point for perhaps removing the support from this PR, landing this, and then following up with a re-addition of gen arg which could just revert those changes and that commit. @Zoxc how's that sound to you?

alexcrichton · Jul 12, 2017

In response to @nikomatsakis's comment above I've prepared a branch which is the current state of this PR (based on 873cb4f) and then removes support for gen arg. Locally this branch is passing all tests.

I'm hoping that this makes an "easy" starting point for perhaps removing the support from this PR, landing this, and then following up with a re-addition of gen arg which could just revert those changes and that commit. @Zoxc how's that sound to you?

arielb1 · Jul 12, 2017

arielb1 reviewed Jul 12, 2017

View changes

arielb1 · Jul 12, 2017

src/librustc/ty/flags.rs

@@ -85,6 +85,15 @@ impl FlagComputation {
                }
            }

+            &ty::TyGenerator(_, ref substs, ref interior) => {
+                // FIXME: Find out why TyClosure has HAS_TY_CLOSURE
+                // and see if the same reason applies here


It's because trait selection etc. on a closure type depends on the current typeck tables status. Which is also the situation with generators.

arielb1 · Jul 12, 2017

arielb1 reviewed Jul 12, 2017

View changes

arielb1 · Jul 12, 2017

src/librustc/traits/select.rs

+            return Ok(());
+        }
+
+        let self_ty = *obligation.self_ty().skip_binder();


Also copy the comment from assemble_closure_candidate? If it wasn't for Chalk I'll think of doing some merger here.

arielb1 · Jul 12, 2017

arielb1 reviewed Jul 12, 2017

View changes

arielb1 · Jul 12, 2017

src/librustc/traits/select.rs

@@ -2495,6 +2547,40 @@ impl<'cx, 'gcx, 'tcx> SelectionContext<'cx, 'gcx, 'tcx> {
        Ok(VtableFnPointerData { fn_ty: self_ty, nested: obligations })
    }

+    fn confirm_generator_candidate(&mut self,


It would be nice to merge the now-3 confirm_poly_trait_ref functions somehow.

You mean the 3 functions calling confirm_poly_trait_refs?

I would rather see a merge of the 3 locations which deals with trait selection (select.rs, project.rs, probe.rs). Will Chalk things help here?

probe.rs implementing a hacked-together version of trait selection is just a crazy thing that should be fixed. Chalk might make select.rs/project.rs simpler.

still a concern, but not blocking landing

arielb1 · Jul 12, 2017

arielb1 reviewed Jul 12, 2017

View changes

arielb1 · Jul 12, 2017

src/librustc_mir/build/scope.rs

@@ -141,6 +144,9 @@ pub struct Scope<'tcx> {

    /// The cache for drop chain on “normal” exit into a particular BasicBlock.
    cached_exits: FxHashMap<(BasicBlock, CodeExtent), BasicBlock>,
+
+    /// The cache for drop chain on "generator drop" exit.
+    cached_generator_drop: Option<BasicBlock>,


Why do we have like 3 separate ways of caching scope cleanup? There should be exactly 1 way.

There's is 3 ways to exit scopes and all of them generate different code. I do think it would be a good idea to generate them with the same code. However I'm not volunteering to write that...

still a concern, but not blocking landing

arielb1 · Jul 12, 2017

arielb1 reviewed Jul 12, 2017

View changes

arielb1 · Jul 12, 2017

src/librustc_mir/util/liveness.rs

+                    self.defs.add(&local);
+                }
+                LvalueContext::Projection(..) |
+                LvalueContext::Borrow { .. } |


This is definitely incorrect handling of borrows. Luckily, values can't be borrowed across yield points.

OTOH, we can say that borrows are "spot uses" and that values are live if they are either currently borrowed or will have a spot borrow before a "killing def".

This is definitely incorrect handling of borrows. Luckily, values can't be borrowed across yield points.

We do need to do this properly for immovable generators which doesn't have this restriction. I'm wondering if looking at the latest NLL things would give some insights here.

OTOH, we can say that borrows are "spot uses" and that values are live if they are either currently borrowed or will have a spot borrow before a "killing def".

I'm not sure what you mean by this.

arielb1 · Jul 12, 2017

arielb1 reviewed Jul 12, 2017

View changes

arielb1 · Jul 12, 2017

src/librustc_mir/util/liveness.rs

+                        self.uses.add(&local);
+                    }
+                }
+                LvalueContext::StorageLive | LvalueContext::StorageDead => (),


These both are liveness-defs (i.e. they kill the current value). This could make some values that are only used by drops dead. e.g.

loop { let x = { // storagelive yield; Box::new(0) }; // drop; storagedead }

@Zoxc do you have thoughts on the temporary removal of gen arg idea?

alexcrichton · Jul 13, 2017

@Zoxc do you have thoughts on the temporary removal of gen arg idea?

@alexcrichton I think I'll cherry-pick your patch and try to land the majority of this branch just to keep @bors from breaking things

Zoxc · Jul 13, 2017

@alexcrichton I think I'll cherry-pick your patch and try to land the majority of this branch just to keep @bors from breaking things

@Zoxc: 🔑 Insufficient privileges: and not in try users

bors · Jul 13, 2017

@Zoxc: 🔑 Insufficient privileges: and not in try users

@nikomatsakis given your list above I believe the simplification is now in this PR (ce12935). Was wondering if you'd had a chance to tackle the second bullet, reading over the borrowck parts?

alexcrichton · Jul 13, 2017

@nikomatsakis given your list above I believe the simplification is now in this PR (ce12935). Was wondering if you'd had a chance to tackle the second bullet, reading over the borrowck parts?

@bors: r=arielb1

alexcrichton · Aug 28, 2017

@bors: r=arielb1

📌 Commit 9eff522 has been approved by arielb1

bors · Aug 28, 2017

📌 Commit 9eff522 has been approved by arielb1

⌛️ Testing commit 9eff522 with merge 0e3c6e0...

bors · Aug 28, 2017

⌛️ Testing commit 9eff522 with merge 0e3c6e0...

💔 Test failed - status-travis

bors · Aug 28, 2017

💔 Test failed - status-travis

@bors: r=arielb1

alexcrichton · Aug 28, 2017

@bors: r=arielb1

📌 Commit 876278f has been approved by arielb1

bors · Aug 28, 2017

📌 Commit 876278f has been approved by arielb1

⌛️ Testing commit 876278f with merge b1acae9...

bors · Aug 28, 2017

⌛️ Testing commit 876278f with merge b1acae9...

💔 Test failed - status-travis

bors · Aug 28, 2017

💔 Test failed - status-travis

RLS test still failed.

(BTW could some of the commits be squashed? This PR will bring in 127 commits, many of which are "fixing this and that".)

kennytm · Aug 28, 2017

RLS test still failed.

(BTW could some of the commits be squashed? This PR will bring in 127 commits, many of which are "fixing this and that".)

@bors: r=arielb1

At this point the commit history is basically an accurate reflection of this history of this feature...

alexcrichton · Aug 28, 2017

@bors: r=arielb1

At this point the commit history is basically an accurate reflection of this history of this feature...

📌 Commit 98e3ebe has been approved by arielb1

bors · Aug 28, 2017

📌 Commit 98e3ebe has been approved by arielb1

@bors: r=arielb1

alexcrichton · Aug 28, 2017

@bors: r=arielb1

📌 Commit a996d5e has been approved by arielb1

bors · Aug 28, 2017

📌 Commit a996d5e has been approved by arielb1

⌛️ Testing commit a996d5e with merge 9a59d69...

bors · Aug 28, 2017

⌛️ Testing commit a996d5e with merge 9a59d69...

☀️ Test successful - status-appveyor, status-travis
Approved by: arielb1
Pushing 9a59d69 to master...

bors · Aug 28, 2017

☀️ Test successful - status-appveyor, status-travis
Approved by: arielb1
Pushing 9a59d69 to master...

This is super exciting and I can't wait to play around with it on the next nightly

mgattozzi · Aug 28, 2017

This is super exciting and I can't wait to play around with it on the next nightly

Hells yeah, great success! <3

Is there any plan on adding support for coroutines as well? Anyway, ultra good job, I’ll play with it as soon as the nightly lands!

phaazon · Aug 31, 2017

Hells yeah, great success! <3

Is there any plan on adding support for coroutines as well? Anyway, ultra good job, I’ll play with it as soon as the nightly lands!

I’ll play with it as soon as the nightly lands!

What do you mean? This is already in the latest nightly. I've been playing with this since yesterday.

rushmorem · Aug 31, 2017

I’ll play with it as soon as the nightly lands!

What do you mean? This is already in the latest nightly. I've been playing with this since yesterday.

Yeah I just saw that, and I’ve started playing with it as well! I’ve already pushed a patch to rust.vim as well \o/

Congrats to everyone!

phaazon · Aug 31, 2017

Yeah I just saw that, and I’ve started playing with it as well! I’ve already pushed a patch to rust.vim as well \o/

Congrats to everyone!

This does not seem to use stackless coroutines support from LLVM or I am understanding it wrongly?
Seemed like a good fit for this.

lilianmoraru · Sep 5, 2017

This does not seem to use stackless coroutines support from LLVM or I am understanding it wrongly?
Seemed like a good fit for this.

Specifically, the first LLVM RFC(cannot edit my comment on mobile): http://lists.llvm.org/pipermail/llvm-dev/2016-July/102337.html

lilianmoraru · Sep 5, 2017

Specifically, the first LLVM RFC(cannot edit my comment on mobile): http://lists.llvm.org/pipermail/llvm-dev/2016-July/102337.html

@lilianmoraru

You can look at the comments here:

rust-lang/rfcs#1823 (comment)

pftbest · Sep 6, 2017

@lilianmoraru

You can look at the comments here:

rust-lang/rfcs#1823 (comment)

Is there a better way to get a generator which doesn't yield?

let mut generator = || {
    if false {
        yield;
    }
    0xDEAD_BEEF
};

valff · Sep 12, 2017

Is there a better way to get a generator which doesn't yield?

let mut generator = || {
    if false {
        yield;
    }
    0xDEAD_BEEF
};

@lilianmoraru LLVM's coroutine support require heap allocation, so it's not suitable for Rust.

@valff Currently, that is the best way. The if will be optimized away in MIR.

Zoxc · Sep 12, 2017

@lilianmoraru LLVM's coroutine support require heap allocation, so it's not suitable for Rust.

@valff Currently, that is the best way. The if will be optimized away in MIR.

rust-highfive assigned nikomatsakis Jul 5, 2017

alexcrichton referenced this pull request Jul 6, 2017
Closed
Work on nightly, not a fork #1
7 of 7 tasks complete

alexcrichton added the S-waiting-on-review label Jul 6, 2017

aturon referenced this pull request Jul 8, 2017
Merged
eRFC: Experimentally add coroutines to Rust #2033

This was referenced Jul 9, 2017

Open

Tracking issue for RFC 2033: Experimentally add coroutines to Rust #43122

Merged

Support for `yield` expressions dtolnay/syn#186

arielb1 added S-waiting-on-author and removed S-waiting-on-review labels Jul 11, 2017

Ignore a threaded test on emscripten

Loading status checks…

876278f

alexcrichton added a commit to rust-lang-nursery/rls that referenced this pull request Aug 28, 2017

Point to Rustfmt branch with `yield`
cc rust-lang/rust#43076

Loading status checks…

144ce17

Tweak rls submodule again

Loading status checks…

a996d5e

bors merged commit a996d5e into rust-lang:master Aug 28, 2017
2 checks passed

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

homu Test successful
Details

This was referenced Aug 28, 2017

Closed

Revert "Auto merge of #43690 - scalexm:issue-28229, r=nikomatsakis" #44045

Merged

Fix end region emission order #44129

Closed

Store fewer NodeIds in crate metadata. #43887

Merged

Remove the trait selection impl in method::probe #43880

whitequark referenced this pull request Aug 29, 2017
Closed
Consider interactions with real generators #72

Frederick888 referenced this pull request Aug 30, 2017
Closed
Failed to build with rustc 1.21.0-nightly (c11f689d2 2017-08-29) #2002

jonasbb added a commit to jonasbb/rls that referenced this pull request Sep 1, 2017

Point to Rustfmt branch with `yield`
cc rust-lang/rust#43076

4e2a66e

kelseasy referenced this pull request Sep 21, 2017
Merged
feat(rust-mode): add yield to the keywords list #240

zhangsoledad referenced this pull request Oct 8, 2017
Open
rust并发模型 #1

rust-lang/rust

Join GitHub today

Generator support #43076

Conversation

Zoxc commented Jul 5, 2017 • edited Edited 1 time Zoxc edited Jul 15, 2017 (most recent)

rust-highfive assigned nikomatsakis Jul 5, 2017

This comment has been minimized.

rust-highfive commented Jul 5, 2017

This comment has been minimized.

alexcrichton commented Jul 5, 2017

alexcrichton referenced this pull request Jul 6, 2017

This comment has been minimized.

bors commented Jul 6, 2017

alexcrichton added the S-waiting-on-review label Jul 6, 2017

nikomatsakis reviewed Jul 7, 2017 View changes

This comment has been minimized.

This comment has been minimized.

nikomatsakis reviewed Jul 7, 2017 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

nikomatsakis reviewed Jul 7, 2017 View changes

This comment has been minimized.

nikomatsakis reviewed Jul 7, 2017 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

nikomatsakis reviewed Jul 7, 2017 View changes

This comment has been minimized.

This comment has been minimized.

nikomatsakis reviewed Jul 7, 2017 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

arielb1 Aug 1, 2017 • edited Edited 1 time arielb1 edited Aug 1, 2017 (most recent)

This comment has been minimized.

nikomatsakis reviewed Jul 7, 2017 View changes

This comment has been minimized.

nikomatsakis reviewed Jul 7, 2017 View changes

This comment has been minimized.

nikomatsakis Jul 7, 2017 • edited Edited 1 time nikomatsakis edited Jul 7, 2017 (most recent)

nikomatsakis reviewed Jul 7, 2017 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

aturon referenced this pull request Jul 8, 2017

This comment has been minimized.

bors commented Jul 8, 2017

This was referenced Jul 9, 2017

Zoxc reviewed Jul 10, 2017 View changes

This comment has been minimized.

apasel422 reviewed Jul 10, 2017 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Zoxc commented Jul 5, 2017 •

edited

Edited 1 time

Zoxc edited Jul 15, 2017 (most recent)

nikomatsakis reviewed Jul 7, 2017

View changes

nikomatsakis reviewed Jul 7, 2017

View changes

nikomatsakis reviewed Jul 7, 2017

View changes

nikomatsakis reviewed Jul 7, 2017

View changes

nikomatsakis reviewed Jul 7, 2017

View changes

nikomatsakis reviewed Jul 7, 2017

View changes

arielb1 Aug 1, 2017 •

edited

Edited 1 time

arielb1 edited Aug 1, 2017 (most recent)

nikomatsakis reviewed Jul 7, 2017

View changes

nikomatsakis reviewed Jul 7, 2017

View changes

nikomatsakis Jul 7, 2017 •

edited

Edited 1 time

nikomatsakis edited Jul 7, 2017 (most recent)

nikomatsakis reviewed Jul 7, 2017

View changes

Zoxc reviewed Jul 10, 2017

View changes

apasel422 reviewed Jul 10, 2017

View changes