Implement `impl Trait` in return type position by anonymization.

[eddyb]

This is the first step towards implementing impl Trait (cc #34511).
impl Trait types are only allowed in function and inherent method return types, and capture all named lifetime and type parameters, being invariant over them.
No lifetimes that are not explicitly named lifetime parameters are allowed to escape from the function body.
The exposed traits are only those listed explicitly, i.e. Foo and Clone in impl Foo + Clone, with the exception of "auto traits" (like Send or Sync) which "leak" the actual contents.

The implementation strategy is anonymization, i.e.:

fn foo<T>(xs: Vec<T>) -> impl Iterator<Item=impl FnOnce() -> T> {
    xs.into_iter().map(|x| || x)
}

// is represented as:
type A</*invariant over*/ T> where A<T>: Iterator<Item=B<T>>;
type B</*invariant over*/ T> where B<T>: FnOnce() -> T;
fn foo<T>(xs: Vec<T>) -> A<T> {
    xs.into_iter().map(|x| || x): $0 where $0: Iterator<Item=$1>, $1: FnOnce() -> T
}

$0 and $1 are resolved (to iter::Map<vec::Iter<T>, closure> and the closure, respectively) and assigned to A and B, after checking the body of foo. A and B are never resolved for user-facing type equality (typeck), but always for the low-level representation and specialization (trans).

The "auto traits" exception is implemented by collecting bounds like impl Trait: Send that have failed for the obscure impl Trait type (i.e. A or B above), pretending they succeeded within the function and trying them again after type-checking the whole crate, by replacing impl Trait with the real type.

While passing around values which have explicit lifetime parameters (of the function with -> impl Trait) in their type should work, regionck appears to assign inference variables in way too many cases, and never properly resolving them to either explicit lifetime parameters, or 'static.
We might not be able to handle lifetime parameters in impl Trait without changes to lifetime inference, but type parameters can have arbitrary lifetimes in them from the caller, so most type-generic usecases (or not generic at all) should not run into this problem.

cc @rust-lang/lang

[rust-highfive]

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

[Ryman]

$0 and $1 are resolved (to iter::Mapvec::Iter<T, closure> and the closure, respectively) and assigned to A and B, after checking the body of foo.

Will the #[must_use] attribute still cause a warning if the return value is unused in this case?

[eddyb]

@Ryman Not unless the type is normalized with Reveal::All in the lint, no.

[bors]

☔️ The latest upstream changes (presumably #35143) made this pull request unmergeable. Please resolve the merge conflicts.

[nikomatsakis]

This comment seems wrong. Perhaps "OutlivesProjectionComponents (ish)" -- really this merits a revising of the RFC 1214 rules to include OutlivesAnonType or something, but that's ok.

[nikomatsakis]

Probably this code is doing the right thing, but it's a bit more conservative than it might otherwise be. Really there are choices here just like for projections. Consider for example if you had trait Foo<'a,'b>: 'a, then you could in theory derive that impl Foo<'a,'b>: 'a even if 'b: 'a doesn't necessarily hold. Something similar could occur with trait Foo<'a, T>: 'a, where impl Foo<'a, &'b T>: 'a might hold even if 'b: 'a does not hold.

It seems in a way like an odd rule, since the type name that the type system gives includes lifetimes (like 'b) that are not necessarily in scope, but the same thing can occur with projections -- the point is that, after monomorphization, the normalized form of that type is known to outlive 'a, even if all components pre-normalization do not.

/me has that itchy feeling of wanting to revisit a richer, formalized version of this, to try and have a good reference. But nothing here makes me too scared yet. =)

[nikomatsakis]

As an aside, I suspect that the grammar ought to be impl (TraitA + TraitB) -- at least, the precedence of impl TraitA + TraitB looks surprising to me. Though maybe I could learn to like it.

I'm imagining things like F: Fn() -> impl Trait + Send <-- how should this be parsed?

[eddyb]

The +Trait is eagerly parsed, so F: Fn() -> (impl Trait + Send). However, impl Trait in a bound seems unlikely to me, and definitely not supported at this moment.
The precedence is this way for convenience - unlike trait objects, there's a prefix (impl), which solves all ambiguity save for what you just mentioned.
I suppose -> impl Fn() -> impl Trait + Send parses, as -> impl Fn() -> (impl Trait + Send).

[nikomatsakis]

The precedence is this way for convenience - unlike trait objects, there's a prefix (impl), which solves all ambiguity save for what you just mentioned.

I don't think there was an more-or-less "ambiguity" before, it's just that we had a weird set of precedence rules. I think the reason I find the precedence of impl relative to + surprising that, in general, we typically give keyword operators the same precedence as other unary operators and it usually higher than binary operators. Hence box 3 + 2 parses as (box 3) + 2. (Admittedly, though, I have complained about this in the past.)

In any case, I continue to find it inconsistent that the Send is associated with the trait in impl Trait + Send but with the type parameter F in F: Fn() Foo -> Trait + Send. (Even if I agree with your argument that one would not necessarily combine them... yet...)

Seems like maybe we should raise it on the tracking issue?

[nikomatsakis]

I haven't read deeply here but I'm a bit dubious about the deferred obligations strategy. Even if we implement things this way, we may want to consider enforcing some kind of inference DAG on uses of impl Trait to give us room to consider later (I guess stabilization makes this less of a concern, actually, never mind).

Anyway, I'm thinking about situations like this one:

trait Foo {
    type X;
}

impl<T> Foo for T {
    default type X = ();
}

impl<T: Send> Foo for T {
    type X = u32;
}

Now if you have an obligation like <Y as Foo>::X = u32, where Y is an anonymous type, we have to figure out how to resolve it, and to do that we kind of need to know whether Y: Send.

In principle, we could probably defer both obligations (and maybe you do, as I said, I didn't finish reading this commit yet) but I do wonder if there will be other situations (method dispatch?) where it is very difficult to suspend.

[eddyb]

AFAIK, even <() as Foo>::X = u32 won't type-check at this moment, so such speculations don't even come into play at this moment - although something like the lattice rule may add them.

[nikomatsakis]

AFAIK, even <() as Foo>::X = u32 won't type-check at this moment

Actually, it does: https://is.gd/36phAF

What does not work is something like normalizing <Rc<()> as Foo>::X to (), because we wanted to preserve the ability to specialize Foo for Rc in the future.

[eddyb]

Ah, right, I got it mixed up. Still, we don't pick the default, we just don't normalize, so doing the same for impl Trait would be just conservative, not wrong.

[nikomatsakis]

doing the same for impl Trait would be just conservative, not wrong.

I think what you mean is: it won't be unsound? As in, we won't normalize to the wrong thing? If so, I agree.

[bors]

☔️ The latest upstream changes (presumably #35267) made this pull request unmergeable. Please resolve the merge conflicts.

[arielb1]

I don't like the "capture all lifetimes in the impl trait" approach and would prefer something more like lifetime elision.

[bors]

☔️ The latest upstream changes (presumably #35166) made this pull request unmergeable. Please resolve the merge conflicts.

[nikomatsakis]

Nit: I can't make heads or tails of this comment :) An example would be great

[eddyb]

Are you talking about allowing impl Trait, or the "generics in scope"? The latter would be easy to provide an example from (<'a, T> for fn foo<'a, T>(...) -> impl Trait), but controlling whether impl Trait is legal in a certain position is harder to explain.
For it to make more sense, rscope would have to become one of several properties that get propagated recursively.

At the same time, the ItemContext in typeck::collect may be a better choice now for holding the "generics in scope", as it's the only AstConv implementation which allows impl Trait, but it's still not that uniform (only the return type of a function or inherent impl method supports it, not just any type in the signature of those items).

[nikomatsakis]

@eddyb after I read below I figured out what the comment meant. I think something like this would have helped me:

---

If this scope allows anonymized types, return the generics in scope that anonymized types will close over. For example, if you have a function like:

fn foo<'a, T>() -> impl Trait { ... }

then, for the scope that is used when handling the return type, this function would return 'a and T. This property is controlled by the region scope because it's fine-grained enough to allow restriction of anonymized types to the syntactical extentof a function's return type.

[nikomatsakis]

I went through the RFC and made a minimal list of test cases I think would like to see:

• positive tests for auto trait inference (something where we need to know that impl Trait: Send for the code to work)
• negative tests for auto trait inference (something where the code requires that impl Trait: Send but in fact it does not)
• test for interaction of auto trait inference and specialization (as mentioned in the comments)
• negative test where choice of regions is ambiguous, e.g., as described in this comment in the final example was one where the code ought to fail
• test where there IS a correct choice of regions (e.g. from that same comment); this could be positive or negative, depending on how the code currently behaves
• negative tests for attempts to use impl Trait syntax in places where it is not permitted:
  • arguments
  • traits and impls of traits
  • impl header
  • fn() -> impl Foo or Fn() -> impl Foo types
  • where clauses
• negative tests where we try to return values of two distinct types
• negative test for recursive fn that uses impl trait as in this example
• negative test that shows "true type" is not revealed to the caller (let x: u32 = foo())
• positive test that shows that using specialization (or sizeof) you can "reveal" the true type -- but only at trans time
• negative test shows an attempt to "leak type" with specialization fails
  • e.g., trait Foo { type T; } impl<T> Foo for T { default type T = (); } impl Foo for i32 { default type T = i32; } and then some code that would require <X as Foo>::T = i32 where X is an impl trait type
• positive test for return type identity (two calls w/ same arguments yield "equal" types)
• negative test for return type identity (two calls w/ a different type argument yield "unequal" types)
• optional: test for region return type identity and region inference (because the two types must be the same, a loan is extended?)
• feature-gate testing (uses of impl trait where feature gate is NOT enabled)

[nikomatsakis]

Er, I posted this comment earlier, but in the wrong place:

@eddyb ok so as discussed on IRC, I am in favor of landing this PR, on the assumption that we will address some of the thornier issues later.

One thing I think is missing is negative tests. I'd like to see at least some tests that if you try to return incompatible types that will result in an error. Probably some (more?) tests targeting "Auto trait" inference -- and the limits, e.g. around specialization -- are warranted as well.

One thing I have done in the past -- but didn't do yet for this PR -- is to make a list, based on the RFC, not the code, of things that might need to be tested, and then compare the code against that list. I'm happy to go through and do it, or you can and post it somewhere. :)

(In case it wasn't clear, the code itself seems good to me! Nice work as usual.)

[nikomatsakis]

@bors r+

[bors]

📌 Commit c0b7f97 has been approved by nikomatsakis

[bors]

☔️ The latest upstream changes (presumably #35592) made this pull request unmergeable. Please resolve the merge conflicts.

[eddyb]

@bors r=nikomatsakis

[bors]

📌 Commit 23f0494 has been approved by nikomatsakis

[bors]

⌛️ Testing commit 23f0494 with merge f55ac69...

[bors]

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-cargotest, auto-linux-64-cross-freebsd, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-opt-no-mir, auto-linux-64-opt-rustbuild, auto-linux-64-x-android-t, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-opt, auto-mac-64-opt-rustbuild, auto-mac-cross-ios-opt, auto-win-gnu-32-opt, auto-win-gnu-32-opt-rustbuild, auto-win-gnu-64-opt, auto-win-msvc-32-opt, auto-win-msvc-64-cargotest, auto-win-msvc-64-opt, auto-win-msvc-64-opt-no-mir, auto-win-msvc-64-opt-rustbuild
Approved by: nikomatsakis
Pushing f55ac69 to master...