Implement `impl Trait` in return type position by anonymization. #35091

eddyb · Jul 28, 2016

This is the first step towards implementing impl Trait (cc #34511).
impl Trait types are only allowed in function and inherent method return types, and capture all named lifetime and type parameters, being invariant over them.
No lifetimes that are not explicitly named lifetime parameters are allowed to escape from the function body.
The exposed traits are only those listed explicitly, i.e. Foo and Clone in impl Foo + Clone, with the exception of "auto traits" (like Send or Sync) which "leak" the actual contents.

The implementation strategy is anonymization, i.e.:

fn foo<T>(xs: Vec<T>) -> impl Iterator<Item=impl FnOnce() -> T> {
    xs.into_iter().map(|x| || x)
}

// is represented as:
type A</*invariant over*/ T> where A<T>: Iterator<Item=B<T>>;
type B</*invariant over*/ T> where B<T>: FnOnce() -> T;
fn foo<T>(xs: Vec<T>) -> A<T> {
    xs.into_iter().map(|x| || x): $0 where $0: Iterator<Item=$1>, $1: FnOnce() -> T
}

$0 and $1 are resolved (to iter::Map<vec::Iter<T>, closure> and the closure, respectively) and assigned to A and B, after checking the body of foo. A and B are never resolved for user-facing type equality (typeck), but always for the low-level representation and specialization (trans).

The "auto traits" exception is implemented by collecting bounds like impl Trait: Send that have failed for the obscure impl Trait type (i.e. A or B above), pretending they succeeded within the function and trying them again after type-checking the whole crate, by replacing impl Trait with the real type.

While passing around values which have explicit lifetime parameters (of the function with -> impl Trait) in their type should work, regionck appears to assign inference variables in way too many cases, and never properly resolving them to either explicit lifetime parameters, or 'static.
We might not be able to handle lifetime parameters in impl Trait without changes to lifetime inference, but type parameters can have arbitrary lifetimes in them from the caller, so most type-generic usecases (or not generic at all) should not run into this problem.

cc @rust-lang/lang

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

rust-highfive · Jul 28, 2016

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

$0 and $1 are resolved (to iter::Mapvec::Iter<T, closure> and the closure, respectively) and assigned to A and B, after checking the body of foo.

Will the #[must_use] attribute still cause a warning if the return value is unused in this case?

Ryman · Jul 28, 2016

$0 and $1 are resolved (to iter::Mapvec::Iter<T, closure> and the closure, respectively) and assigned to A and B, after checking the body of foo.

Will the #[must_use] attribute still cause a warning if the return value is unused in this case?

@Ryman Not unless the type is normalized with Reveal::All in the lint, no.

eddyb · Jul 28, 2016

@Ryman Not unless the type is normalized with Reveal::All in the lint, no.

☔️ The latest upstream changes (presumably #35143) made this pull request unmergeable. Please resolve the merge conflicts.

bors · Aug 1, 2016

☔️ The latest upstream changes (presumably #35143) made this pull request unmergeable. Please resolve the merge conflicts.

nikomatsakis · Aug 5, 2016

nikomatsakis reviewed Aug 5, 2016

View changes

nikomatsakis · Aug 5, 2016

src/librustc/ty/outlives.rs

@@ -174,6 +174,7 @@ impl<'a, 'gcx, 'tcx> InferCtxt<'a, 'gcx, 'tcx> {
            ty::TyEnum(..) |        // OutlivesNominalType
            ty::TyStruct(..) |      // OutlivesNominalType
            ty::TyBox(..) |         // OutlivesNominalType (ish)
+            ty::TyAnon(..) |        // OutlivesNominalType (ish)


This comment seems wrong. Perhaps "OutlivesProjectionComponents (ish)" -- really this merits a revising of the RFC 1214 rules to include OutlivesAnonType or something, but that's ok.

Probably this code is doing the right thing, but it's a bit more conservative than it might otherwise be. Really there are choices here just like for projections. Consider for example if you had trait Foo<'a,'b>: 'a, then you could in theory derive that impl Foo<'a,'b>: 'a even if 'b: 'a doesn't necessarily hold. Something similar could occur with trait Foo<'a, T>: 'a, where impl Foo<'a, &'b T>: 'a might hold even if 'b: 'a does not hold.

It seems in a way like an odd rule, since the type name that the type system gives includes lifetimes (like 'b) that are not necessarily in scope, but the same thing can occur with projections -- the point is that, after monomorphization, the normalized form of that type is known to outlive 'a, even if all components pre-normalization do not.

/me has that itchy feeling of wanting to revisit a richer, formalized version of this, to try and have a good reference. But nothing here makes me too scared yet. =)

nikomatsakis · Aug 5, 2016

nikomatsakis reviewed Aug 5, 2016

View changes

nikomatsakis · Aug 5, 2016

src/librustc/util/ppaux.rs

@@ -907,6 +907,27 @@ impl<'tcx> fmt::Display for ty::TypeVariants<'tcx> {
            }
            TyTrait(ref data) => write!(f, "{}", data),
            ty::TyProjection(ref data) => write!(f, "{}", data),
+            ty::TyAnon(def_id, substs) => {
+                ty::tls::with(|tcx| {
+                    // Grab the "TraitA + TraitB" from `impl TraitA + TraitB`,


As an aside, I suspect that the grammar ought to be impl (TraitA + TraitB) -- at least, the precedence of impl TraitA + TraitB looks surprising to me. Though maybe I could learn to like it.

I'm imagining things like F: Fn() -> impl Trait + Send <-- how should this be parsed?

The +Trait is eagerly parsed, so F: Fn() -> (impl Trait + Send). However, impl Trait in a bound seems unlikely to me, and definitely not supported at this moment.
The precedence is this way for convenience - unlike trait objects, there's a prefix (impl), which solves all ambiguity save for what you just mentioned.
I suppose -> impl Fn() -> impl Trait + Send parses, as -> impl Fn() -> (impl Trait + Send).

The precedence is this way for convenience - unlike trait objects, there's a prefix (impl), which solves all ambiguity save for what you just mentioned.

I don't think there was an more-or-less "ambiguity" before, it's just that we had a weird set of precedence rules. I think the reason I find the precedence of impl relative to + surprising that, in general, we typically give keyword operators the same precedence as other unary operators and it usually higher than binary operators. Hence box 3 + 2 parses as (box 3) + 2. (Admittedly, though, I have complained about this in the past.)

In any case, I continue to find it inconsistent that the Send is associated with the trait in impl Trait + Send but with the type parameter F in F: Fn() Foo -> Trait + Send. (Even if I agree with your argument that one would not necessarily combine them... yet...)

Seems like maybe we should raise it on the tracking issue?

nikomatsakis · Aug 5, 2016

nikomatsakis reviewed Aug 5, 2016

View changes

nikomatsakis · Aug 5, 2016

src/librustc/traits/fulfill.rs

+pub struct DeferredObligation<'tcx> {
+    pub predicate: ty::PolyTraitPredicate<'tcx>,
+    pub cause: ObligationCause<'tcx>
+}


I haven't read deeply here but I'm a bit dubious about the deferred obligations strategy. Even if we implement things this way, we may want to consider enforcing some kind of inference DAG on uses of impl Trait to give us room to consider later (I guess stabilization makes this less of a concern, actually, never mind).

Anyway, I'm thinking about situations like this one:

trait Foo { type X; } impl<T> Foo for T { default type X = (); } impl<T: Send> Foo for T { type X = u32; }

Now if you have an obligation like <Y as Foo>::X = u32, where Y is an anonymous type, we have to figure out how to resolve it, and to do that we kind of need to know whether Y: Send.

In principle, we could probably defer both obligations (and maybe you do, as I said, I didn't finish reading this commit yet) but I do wonder if there will be other situations (method dispatch?) where it is very difficult to suspend.

AFAIK, even <() as Foo>::X = u32 won't type-check at this moment, so such speculations don't even come into play at this moment - although something like the lattice rule may add them.

AFAIK, even <() as Foo>::X = u32 won't type-check at this moment

Actually, it does: https://is.gd/36phAF

What does not work is something like normalizing <Rc<()> as Foo>::X to (), because we wanted to preserve the ability to specialize Foo for Rc in the future.

Ah, right, I got it mixed up. Still, we don't pick the default, we just don't normalize, so doing the same for impl Trait would be just conservative, not wrong.

doing the same for impl Trait would be just conservative, not wrong.

I think what you mean is: it won't be unsound? As in, we won't normalize to the wrong thing? If so, I agree.

☔️ The latest upstream changes (presumably #35267) made this pull request unmergeable. Please resolve the merge conflicts.

bors · Aug 6, 2016

☔️ The latest upstream changes (presumably #35267) made this pull request unmergeable. Please resolve the merge conflicts.

I don't like the "capture all lifetimes in the impl trait" approach and would prefer something more like lifetime elision.

arielb1 · Aug 7, 2016

I don't like the "capture all lifetimes in the impl trait" approach and would prefer something more like lifetime elision.

☔️ The latest upstream changes (presumably #35166) made this pull request unmergeable. Please resolve the merge conflicts.

bors · Aug 9, 2016

☔️ The latest upstream changes (presumably #35166) made this pull request unmergeable. Please resolve the merge conflicts.

nikomatsakis · Aug 10, 2016

nikomatsakis reviewed Aug 10, 2016

View changes

nikomatsakis · Aug 10, 2016

src/librustc_typeck/rscope.rs

+    /// scope, that anonymized types will close over. This property is
+    /// controlled by the region scope because it's fine-grained enough
+    /// to allow restriction of anonymized types to the syntactical extent
+    /// of a function's return type.


Nit: I can't make heads or tails of this comment :) An example would be great

Are you talking about allowing impl Trait, or the "generics in scope"? The latter would be easy to provide an example from (<'a, T> for fn foo<'a, T>(...) -> impl Trait), but controlling whether impl Trait is legal in a certain position is harder to explain.
For it to make more sense, rscope would have to become one of several properties that get propagated recursively.

At the same time, the ItemContext in typeck::collect may be a better choice now for holding the "generics in scope", as it's the only AstConv implementation which allows impl Trait, but it's still not that uniform (only the return type of a function or inherent impl method supports it, not just any type in the signature of those items).

@eddyb after I read below I figured out what the comment meant. I think something like this would have helped me:

If this scope allows anonymized types, return the generics in scope that anonymized types will close over. For example, if you have a function like:

fn foo<'a, T>() -> impl Trait { ... }

then, for the scope that is used when handling the return type, this function would return 'a and T. This property is controlled by the region scope because it's fine-grained enough to allow restriction of anonymized types to the syntactical extentof a function's return type.

I went through the RFC and made a minimal list of test cases I think would like to see:

positive tests for auto trait inference (something where we need to know that impl Trait: Send for the code to work)
negative tests for auto trait inference (something where the code requires that impl Trait: Send but in fact it does not)
test for interaction of auto trait inference and specialization (as mentioned in the comments)
negative test where choice of regions is ambiguous, e.g., as described in this comment in the final example was one where the code ought to fail
test where there IS a correct choice of regions (e.g. from that same comment); this could be positive or negative, depending on how the code currently behaves
negative tests for attempts to use impl Trait syntax in places where it is not permitted:
- arguments
- traits and impls of traits
- impl header
- fn() -> impl Foo or Fn() -> impl Foo types
- where clauses
negative tests where we try to return values of two distinct types
negative test for recursive fn that uses impl trait as in this example
negative test that shows "true type" is not revealed to the caller (let x: u32 = foo())
positive test that shows that using specialization (or sizeof) you can "reveal" the true type -- but only at trans time
negative test shows an attempt to "leak type" with specialization fails
- e.g., trait Foo { type T; } impl<T> Foo for T { default type T = (); } impl Foo for i32 { default type T = i32; } and then some code that would require <X as Foo>::T = i32 where X is an impl trait type
positive test for return type identity (two calls w/ same arguments yield "equal" types)
negative test for return type identity (two calls w/ a different type argument yield "unequal" types)
optional: test for region return type identity and region inference (because the two types must be the same, a loan is extended?)
feature-gate testing (uses of impl trait where feature gate is NOT enabled)

nikomatsakis · Aug 10, 2016

I went through the RFC and made a minimal list of test cases I think would like to see:

Er, I posted this comment earlier, but in the wrong place:

@eddyb ok so as discussed on IRC, I am in favor of landing this PR, on the assumption that we will address some of the thornier issues later.

One thing I think is missing is negative tests. I'd like to see at least some tests that if you try to return incompatible types that will result in an error. Probably some (more?) tests targeting "Auto trait" inference -- and the limits, e.g. around specialization -- are warranted as well.

One thing I have done in the past -- but didn't do yet for this PR -- is to make a list, based on the RFC, not the code, of things that might need to be tested, and then compare the code against that list. I'm happy to go through and do it, or you can and post it somewhere. :)

(In case it wasn't clear, the code itself seems good to me! Nice work as usual.)

nikomatsakis · Aug 10, 2016

Er, I posted this comment earlier, but in the wrong place:

@eddyb ok so as discussed on IRC, I am in favor of landing this PR, on the assumption that we will address some of the thornier issues later.

One thing I think is missing is negative tests. I'd like to see at least some tests that if you try to return incompatible types that will result in an error. Probably some (more?) tests targeting "Auto trait" inference -- and the limits, e.g. around specialization -- are warranted as well.

One thing I have done in the past -- but didn't do yet for this PR -- is to make a list, based on the RFC, not the code, of things that might need to be tested, and then compare the code against that list. I'm happy to go through and do it, or you can and post it somewhere. :)

(In case it wasn't clear, the code itself seems good to me! Nice work as usual.)

@bors r+

nikomatsakis · Aug 11, 2016

@bors r+

📌 Commit c0b7f97 has been approved by nikomatsakis

bors · Aug 11, 2016

📌 Commit c0b7f97 has been approved by nikomatsakis

☔️ The latest upstream changes (presumably #35592) made this pull request unmergeable. Please resolve the merge conflicts.

bors · Aug 12, 2016

☔️ The latest upstream changes (presumably #35592) made this pull request unmergeable. Please resolve the merge conflicts.

@bors r=nikomatsakis

eddyb · Aug 12, 2016

@bors r=nikomatsakis

📌 Commit 23f0494 has been approved by nikomatsakis

bors · Aug 12, 2016

📌 Commit 23f0494 has been approved by nikomatsakis

⌛️ Testing commit 23f0494 with merge f55ac69...

bors · Aug 12, 2016

⌛️ Testing commit 23f0494 with merge f55ac69...

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-cargotest, auto-linux-64-cross-freebsd, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-opt-no-mir, auto-linux-64-opt-rustbuild, auto-linux-64-x-android-t, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-opt, auto-mac-64-opt-rustbuild, auto-mac-cross-ios-opt, auto-win-gnu-32-opt, auto-win-gnu-32-opt-rustbuild, auto-win-gnu-64-opt, auto-win-msvc-32-opt, auto-win-msvc-64-cargotest, auto-win-msvc-64-opt, auto-win-msvc-64-opt-no-mir, auto-win-msvc-64-opt-rustbuild
Approved by: nikomatsakis
Pushing f55ac69 to master...

bors · Aug 12, 2016

rust-highfive assigned nikomatsakis Jul 28, 2016

eddyb referenced this pull request Jul 31, 2016
Open
Tracking issue for `impl Trait` (RFC 1522, RFC 1951, RFC 2071) #34511
14 of 29 tasks complete

eddyb referenced this pull request Aug 11, 2016
Merged
Remove the ParamSpace separation from formal and actual generics in rustc. #35605

bors merged commit 23f0494 into rust-lang:master Aug 12, 2016
1 of 2 checks passed

1 of 2 checks passed

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details

homu Test successful
Details

ticki referenced this pull request Aug 12, 2016
Open
Detect situations where `impl Trait` is preferred. #1161

eddyb deleted the eddyb:impl-trait branch Aug 12, 2016

TimNN referenced this pull request Aug 28, 2016
Closed
ICE: the type `u8` has an unknown layout #36036

m4rw3r referenced this pull request Aug 28, 2016
Closed
`impl Trait` and making Chomp require nightly #50

Marwes referenced this pull request Aug 30, 2016
Closed
Example of higher order parser generating functions. #66

kennytm referenced this pull request Jan 28, 2018
Open
Incorrect use of traits::SizedReturnType #47830

rust-lang/rust

Join GitHub today

Implement `impl Trait` in return type position by anonymization. #35091

Conversation

eddyb commented Jul 28, 2016 • edited Edited 1 time eddyb edited Jul 30, 2016 (most recent)

rust-highfive assigned nikomatsakis Jul 28, 2016

This comment has been minimized.

rust-highfive commented Jul 28, 2016

This comment has been minimized.

Ryman commented Jul 28, 2016

This comment has been minimized.

eddyb commented Jul 28, 2016

eddyb referenced this pull request Jul 31, 2016

This comment has been minimized.

bors commented Aug 1, 2016

nikomatsakis reviewed Aug 5, 2016 View changes

This comment has been minimized.

This comment has been minimized.

nikomatsakis reviewed Aug 5, 2016 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

nikomatsakis reviewed Aug 5, 2016 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

bors commented Aug 6, 2016

This comment has been minimized.

arielb1 commented Aug 7, 2016

This comment has been minimized.

bors commented Aug 9, 2016

nikomatsakis reviewed Aug 10, 2016 View changes

This comment has been minimized.

This comment has been minimized.

eddyb Aug 10, 2016 • edited Edited 1 time eddyb edited Aug 10, 2016 (most recent)

This comment has been minimized.

This comment has been minimized.

nikomatsakis commented Aug 10, 2016 • edited by eddyb Edited 1 time eddyb edited Aug 10, 2016 (most recent)

This comment has been minimized.

nikomatsakis commented Aug 10, 2016

This comment has been minimized.

nikomatsakis commented Aug 11, 2016

This comment has been minimized.

bors commented Aug 11, 2016

eddyb referenced this pull request Aug 11, 2016

This comment has been minimized.

bors commented Aug 12, 2016

eddyb added some commits Jun 30, 2016

This comment has been minimized.

eddyb commented Aug 12, 2016

This comment has been minimized.

bors commented Aug 12, 2016

This comment has been minimized.

bors commented Aug 12, 2016

bors added a commit that referenced this pull request Aug 12, 2016

This comment has been minimized.

bors commented Aug 12, 2016

Hide details View details bors merged commit 23f0494 into rust-lang:master Aug 12, 2016 1 of 2 checks passed

1 of 2 checks passed

This was referenced Aug 12, 2016

ticki referenced this pull request Aug 12, 2016

eddyb deleted the eddyb:impl-trait branch Aug 12, 2016

TimNN referenced this pull request Aug 28, 2016

m4rw3r referenced this pull request Aug 28, 2016

Marwes referenced this pull request Aug 30, 2016

kennytm referenced this pull request Jan 28, 2018

eddyb commented Jul 28, 2016 •

edited

Edited 1 time

eddyb edited Jul 30, 2016 (most recent)

nikomatsakis reviewed Aug 5, 2016

View changes

nikomatsakis reviewed Aug 5, 2016

View changes

nikomatsakis reviewed Aug 5, 2016

View changes

nikomatsakis reviewed Aug 10, 2016

View changes

eddyb Aug 10, 2016 •

edited

Edited 1 time

eddyb edited Aug 10, 2016 (most recent)

nikomatsakis commented Aug 10, 2016 •

edited by eddyb

Edited 1 time

eddyb edited Aug 10, 2016 (most recent)

bors merged commit `23f0494` into rust-lang:master Aug 12, 2016
1 of 2 checks passed