Support variance for type parameters #738

Merged
merged 2 commits into from Feb 12, 2015

Projects

None yet
@nikomatsakis
Contributor

A revised proposal for properly supporting variance in Rust's type system.

Rendered view.

@nikomatsakis nikomatsakis self-assigned this Jan 26, 2015
@dgrunwald
Contributor

👍 I like PhantomData, this will make writing FFI code significantly easier.

Variance for type parameters: This is one of the things that users don't notice if it works properly, and is utterly confusing when types are unnecessarily invariant. This alone is a good reason to infer variance instead of requiring explicit variance annotations.

@Kimundi Kimundi commented on the diff Jan 27, 2015
text/0000-variance.md
+}
+```
+
+Here the `Context` struct has one lifetime parameter, `data`, that
+represents the lifetime of some data that it references. Now let's
+imagine that the lifetime of the data is some lifetime we call
+`'x`. If we have a context `cx` of type `Context<'x>`, it is ok to
+(for example) pass `cx` as an argment where a value of type
+`Context<'y>` is required, so long as `'x : 'y` ("`'x` outlives
+`'y`"). That is, it is ok to approximate `'x` as a shorter lifetime
+like `'y`. This makes sense because by changing `'x` to `'y`, we're
+just pretending the data has a shorter lifetime than it actually has,
+which can't do any harm. Here is an example:
+
+```rust
+fn approx_context<'long,'short>(t: &Context<'long>, data: &'short Data)
@Kimundi
Kimundi Jan 27, 2015 Member

Is this meant to take a reference for the first parameter? As it is, the example is a compile error because do_something expects a Context, but gets passed a &Context in approx_context.

@nikomatsakis
nikomatsakis Jan 27, 2015 Contributor

Fixed.

@bill-myers

Another alternative could be to have inference as this RFC suggests, but then emit an error if a variance annotation does not match the inferred variance, telling the user exactly what change needs to be made:

type parameter `T` is covariant, so you should declare `Foo` as `struct Foo<+T>`.
If you didn't intend that, add a `PhantomData<fn() -> T>` marker or equivalent to make `T` invariant.

Only inference makes it easier to write code, but it does violate the Rust philosophy that the types of public APIs should be explicitly written out.

Regarding annotations, the obvious "lightweight" approach is to add a single punctuation character for each variance, for instance "=" or nothing for invariance, "+" for covariance, "-" for contravariance.

Not sure whether this is better or not than just inference, but being the only violation of having explicit types for APIs is a bit disturbing.

@kennytm kennytm commented on the diff Jan 27, 2015
text/0000-variance.md
+marker traits are orthogonal to lifetimes, it actually rarely makes a
+difference what choice you make here. But imagine that we have a
+marker trait that requires `'static` (such as `Send` today, though
+this may change). If we made marker traits covariant with respect to
+`Self`, then `&'static Foo : Send` could be used as evidence that `&'x
+Foo : Send` for any `'x`, because `&'static Foo <: &'x Foo`:
+
+ (&'static Foo : Send) <: (&'x Foo : Send) // if things were covariant...
+ &'static Foo <: &'x Foo // ...we'd have the wrong relation here
+
+*Interesting side story: the author thought that covariance would be
+correct for some time. It was only when attempting to phrase the
+desired behavior as a fn that I realized I had it backward, and
+quickly found the counterexample I give above. This gives me
+confidence that expressing variance in terms of data and fns is more
+reliable than trying to divine the correct results directly.*
@kennytm
kennytm Jan 27, 2015

divine → derive?

@kennytm
kennytm commented Jan 27, 2015

Instead of PhantomData<*mut T>, could PhantomData<&'static mut T> work?

@nikomatsakis
Contributor

@kennytm using PhantomData<&'static mut T> would require that T:'static, so it's probably not what you want.

@dgrunwald
Contributor

Regarding annotations, the obvious "lightweight" approach is to add a single punctuation character for each variance, for instance "=" or nothing for invariance, "+" for covariance, "-" for contravariance.

"+" or "-" will be hard to understand for users not familiar with variance.
If we have to use explicit annotations, I'd prefer the C# approach using the keywords "in" and "out":
nothing = invariant, "out" = covariant, "in" = contravariant

@nikomatsakis
Contributor

@bill-myers yes, I agree but also disagree.

The reasons I would prefer not to have explicit variance annotations are listed in the RFC, but I'll repeat them:

  1. What do we call these annotations? Symbols like -, +, etc are notoriously confusing. Keywords like "covariant" are as well. Keywords like "in" and "out", as used in C#, don't fit so well in Rust, where the "interface" and "data" declarations are separated (for example, Vec<out T> just doesn't make sense to me).
  2. Having both annotations and phantom data seems very non-DRY, and I've found that the phantom data approach is both more broadly applicable and more reliable in terms of getting the right results.

That said, it is nice to be able to declare the expected variance (and I've found myself using internal Rust compiler debugging magic to force it to print out the results of inference on occasion). I imagine it might be nice to have a standard way of declaring the results you expect -- one option remains the writing of unit tests asserting the expected behavior.

Note that we've been using inference for quite some time now actually, it's just that it only applies to lifetime parameters (but it is quite crucial there; Rust would not work at all without it). Experience has been fairly positive thus far I would say -- the only real issues I see are when the expected results don't materialize (particularly around Option, where lifetime checking is much stricter, as the RFC shows).

@nikomatsakis
Contributor

Ah, and note that having some variance declaration also requires us to settle on a terminology around lifetime parameters. Overall it just feels like a lot more complexity to present the user with.

@kennytm
kennytm commented Jan 27, 2015

@nikomatsakis Ah I see, thanks.

@glaebhoerl
Contributor

I'm also in the camp of preferring interfaces to be made explicit, but respect the objection that there's no (obvious) ergonomic and easily comprehensible way to do so. PhantomData is a cool idea should we choose to go with inference.

My one small contribution to the question of potential explicit syntax is that I believe variances should be thought of and presented as additional capabilities granted, relative to the baseline of "invariant", to the user of the API - just like everything else that you publicly expose in an API is an additional capability relative to not having exposed it. So one capability is "can be up-casted", another is "can be down-casted" ("cast" is maybe not the right word), if you have only one or the other it's what we call "covariant" or "contravariant", while if you have both it's "bivariant". The crucial point is that this is the reverse of how they're usually framed, which is in terms of how the type parameter is used by/within the type being defined, which induces restrictions: where in means it's used as a (function) input, meaning you can't cast it in one direction, out means it's present as a data member or other "positive" position, meaning you can't cast it in the other, and the combination of the two results in invariance. What we want is for the two keywords (whatever we're able to come up with) to correspond to how the type parameter is not used by the type being defined, which allows for greater freedom for the user of the type, with the combination of both keywords (meaning the parameter is not used at all - neither in positive nor negative positions) expressing bivariance.

(Incidentally, this also happens to be how the Functor and Contrafunctor type classes are used by Glasgow Haskell's lens library.)

@nrc
Contributor
nrc commented Jan 27, 2015

Nice, I'm in favour of the inference approach and using 'annotations' for assistance (explicit variance annotations are nearly always awful). I was thinking that this is kind of hacking around writing exactly what the author means, but I think what I was thinking of is the phantom keyword stuff in the future directions section - I'd be keen to see that implemented asap.

As another reason to avoid explicit annotations, I imagine that the fact that Rust subtyping and thus variance only affects lifetimes will further confuse users - variance in other languages is almost always discussed in terms of containers and the type of elements. That we use implicit coercions for many subtyping-ish things, but that these are not covered by the variance annotations might be odd.

Question: is there any effect in using the traits here as bounds on type parameters? e.g., fn foo<X: PhantomFn() -> i32>(x: X) { ... }

@nrc nrc commented on the diff Jan 27, 2015
text/0000-variance.md
+"unsafe box". `Unique` should ensure that it is covariant with respect
+to its argument.
+
+However, this raises the question of how to implement `Unique` under
+the hood, and what to do with `*mut T` in general. There are various
+options:
+
+1. Change `*mut` so that it behaves like `*const`. This unfortunately
+ means that abstractions that introduce shared mutability have
+ a responsibility for add phantom data to that affect, something
+ like `PhantomData<*const Cell<T>>`. This seems non-obvious and
+ unnatural.
+
+2. Rewrite safe abstractions to use `*const` (or even `usize`) instead
+ of `*mut`, casting to `*mut` only they have a `&mut self`
+ method. This is probably the most conservative option.
@nrc
nrc Jan 27, 2015 Contributor

I vote for 2 - this is what I do in my personal projects, I didn't realise the std libs used *mut this way.

@nikomatsakis
nikomatsakis Feb 2, 2015 Contributor

Yes, I have currently implemented option 2: that is, *mut is invariant. I had been leaning towards what I think was 3 (ignore *mut references entirely and reauire explicit markers), but I fear that that is more error-prone. Basically if people aren't thinking carefully it's easy to make a covariant type that ought to have been invariant. Option 2 is correct by default, and requires you to opt-in (by using distinct types) to covariance. On the variance branch I've just been rewriting standard library stuff not to use raw *mut pointers but rather safe wrappers like Unique, which I think improves the code overall, since it's more declarative. It also handles the OIBIT requirements and so forth. Those are currently unstable though, we'll probably want some iteraiton on their design -- right now they are just kind of shallow wrappers around a raw pointer, we might be able to do better. (Note also that Unique on the branch is quite different from Unique on master).

@Gankro
Gankro Feb 11, 2015 Contributor

Hashmap (and my working branch of btree) don't use Unique because they contain what is essentially a void pointer to "bytes".

@aturon aturon referenced this pull request in rust-lang/rust Jan 28, 2015
Closed

Stabilization for 1.0-alpha2 #20761

29 of 38 tasks complete
@llogiq
Contributor
llogiq commented Jan 29, 2015

One possibly less confusing syntax for denoting the variance type would be

  • T (or =T) for invariant type (_ equals T)
  • T/ for covariant type T (read like T over _)
  • /T for contravariant type T (read like _ over T)

The idea is that the / denotes the position of our specified type related to T in an assumed hierarchy.

I don't know how much this would complicate the parser, though.

In any event, inference has my vote.

@alexcrichton alexcrichton referenced this pull request in rust-lang/rust Jan 30, 2015
Closed

HashMap<K,V> is always Send/Sync #21763

@Gankro
Contributor
Gankro commented Jan 30, 2015
@gereeter gereeter commented on the diff Jan 30, 2015
text/0000-variance.md
+
+ // Act as if we could reach a slice `[T]` with lifetime `'a`.
+ // Induces covariance on `T` and suitable variance on `'a`
+ // (covariance using the definition from rfcs#391).
+ marker: marker::PhantomData<&'a [T]>,
+}
+```
+
+Note that `PhantomData` can be used to induce covariance, invariance, or contravariance
+as desired:
+
+```rust
+PhantomData<T> // covariance
+PhantomData<*mut T> // invariance, but see "unresolved question"
+PhantomData<Cell<T>> // invariance
+PhantomData<fn() -> T> // contravariant
@gereeter
gereeter Jan 30, 2015

Shouldn't this be covariant? fn() -> T is roughly isomorphic to T, so the result should be the same as if PhantomData<T> were used. Alternatively, this could be changed to fn(T).

@nikomatsakis
nikomatsakis Feb 2, 2015 Contributor

yes, sorry, I meant fn(T)

@nikomatsakis
Contributor

@nick29581

I was thinking that this is kind of hacking around writing exactly what the author means, but I think what I was thinking of is the phantom keyword stuff in the future directions section - I'd be keen to see that implemented asap.

Me too, I'd be happy to revise the RFC to include the phantom syntax as a first-class thing, even if I'd want to land a branch that uses the markers for now. The usability improvements of having concrete syntax would be quite high, I think, and given how central all of this is to the type system, it seems justified to me.

Question: is there any effect in using the traits here as bounds on type parameters? e.g., fn foo<X: PhantomFn() -> i32>(x: X) { ... }

You can do it but it's meaningless -- since PhantomFn defines no methods, and is implemented for all types, adding such a bound buys you nothing. The only place that PhantomFn is useful is in the supertraits listing.

@bgamari bgamari commented on the diff Feb 3, 2015
text/0000-variance.md
+data with a shorter lifetime than `t` originally had. This is bad
+because the caller still has the old type (`Table<'long>`) and doesn't
+know that data with a shorter lifetime has been inserted. (This is
+traditionally called "invariant".)
+
+Finally, there can be cases where it is ok to make a lifetime
+*longer*, but not shorter. This comes up when the lifetime is used in
+a function return type (and only a fn return type). This is very
+unusual in Rust but it can happen.
+
+[v]: http://en.wikipedia.org/wiki/Covariance_and_contravariance_%28computer_science%29
+
+## Why variance should be inferred
+
+Actually, lifetime parameters already have a notion of variance, and
+this varinace is fully inferred. In fact, the proper variance for type
@bgamari
bgamari Feb 3, 2015

This should read "variance".

@theemathas

-1 for variance inference. I believe that while having implicit variance would help user if it works properly, it would cause utter confusion when it doesn't. The fact that rustdoc hides private fields doesn't help at all when someone hits a variance error in an external library. Cases that require anything other than contravariant lifetimes require (relatively) advanced concepts any way, and the confusion most likely stems from the fact that the names are scary.

I think it is simpler to use contravariance by default, use struct Foo<no_shrink_lifetime 'a> {...} or #[no_shrink_lifetime('a)] for invariant lifetimes, and use struct Foo<allow_extend_lifetime + no_shrink_lifetime 'a> {...} or #[allow_extend_lifetime('a)] #[no_shrink_lifetime('a)] for covariance

@tomjakubowski
Contributor

The fact that rustdoc hides private fields doesn't help at all when someone hits a variance error in an external library.

It shouldn't be too hard (knock on wood) for rustdoc to show variance on type/lifetime parameters. There's an open issue for that here rust-lang/rust#22108

@llogiq
Contributor
llogiq commented Feb 11, 2015

@theemathas, the cardinality of confusion on inference failure is directly proportional to the quality of error messages and documentation. As long as: a) the rules aren't too arcane to be explained to Joe coder and b) it is possible for the compiler to show the location in the code where to add markers/annotations/whatever, confusion should be quite limited.

@aturon aturon merged commit 77c090b into rust-lang:master Feb 12, 2015
@aturon
Contributor
aturon commented Feb 12, 2015

While there has not been a great deal of feedback, there is broad agreement that the new Phantom markers are clearer than today's markers (though could perhaps be more clear with special syntax), and that some variance for type parameters is needed. While some have expressed reservations about inference here, it's worth remembering that this inference has long been performed (though not largely used) in the past. Given that the variance is strictly related to lifetimes (the only form of subtyping), being implicit here seems worth at least attempting, and the implementation has been sitting on the side being rebased for quite a long time now.

We've decided to merge this RFC and land the implementation to gain some experience with this system. Please try it out and give feedback as to whether inference ever leads to unclear results in practice.

Tracking issue

@tomaka
tomaka commented Feb 16, 2015

In my library I currently have this code:

pub struct TextureType1 { handle: u32 }
pub struct TextureType2 { handle: u32 }
// ... around 50 different texture types

pub struct FramebufferObject<'a> {
    attachments: Vec<u32>,
    marker: ContravariantLifetime<'a>,
} 

impl<'a> FramebufferObject<'a> {
    pub fn attach1(&mut self, tex: &'a mut TextureType1) {
        self.attachments.push(tex.handle);
    }

    pub fn attach2(&mut self, tex: &'a mut TextureType2) {
        self.attachments.push(tex.handle);
    }
}

This ensures that the FramebufferObject doesn't live longer that the attached Textures.

How would I do that with PhantomData? Do I need to create some kind of enum containing all the possible PhantomData types or something?

@pnkfelix
Member

@tomaka if I properly understand what you are doing, I think you would write something like: marker: PhantomData<&'a ()>,

In other words, the phantom data represents the hidden references to data of lifetime 'a.

@alexchandel

Congrats guys, my code got a little uglier.

use std::marker::MarkerTrait;
pub trait LoadCommandBody: MarkerTrait {}

Thanks for that.

@steveklabnik
Contributor

@alexchandel please keep it constructive.

@petrochenkov
Contributor

Constructive: at least put MarkerTrait in the prelude.

@tomaka
tomaka commented Feb 23, 2015

MarkerTrait definitely looks hacky.

The syntax trait A : B {} is supposed to mean that all types that implement A must also implement B.
When you write trait A : MarkerTrait {}, it looks like MarkerTrait applies of the trait A itself, which is contrary to this definition. It makes no sense for a regular type like i32 to implement MarkerTrait.

@petrochenkov
Contributor

@tomaka
MarkerTrait is fun, every sized type implicitly implement it.

fn main() {
    fn f<T: std::marker::MarkerTrait>(_: T) {}
    struct S;
    f(S);
}
@tomaka
tomaka commented Feb 23, 2015

I'm complaining about the name MarkerTrait. Why would i32 implement a trait whose name is MarkerTrait? There has to be a better way to name this, like NoSelfVariance or something.

@NXTangl
NXTangl commented Dec 10, 2015

@tomaka Perhaps Rust should differentiate inheritance from type application. As it is, traits implementing traits is more like subtyping than trait implementation--if a type implements a trait, it implements all 'supertraits' of the trait, just as if a value is an instance of a type, it is an instance of all supertypes of the type. (I think this is why Haskell chose to use the class/instance terminology for its typeclasses.) In that case, traits describe sets of types (or, more generally, relations on types [in the database sense] which Rust represents using associated types) and types describe sets of values, and values don't need to be sets.

@burdges
burdges commented Dec 11, 2015

Yes, MarkerTrait needs a new name. Is it clearer if you explain it in terms of marking something? I don't think so.

If you dislike NoSelfVariance, which at least warns folks about the strangeness, then maybe SetOfTypes, PhantomMethod, StaticNeeded, etc. And one should consider prefixing the term with a term like Common.. or Typical.. like TypicalSelfCovariance or CommonPhantomSelf. Just whatever makes the most sense when you think about writing the documentation that explains it.

As an aside, I initially wrote the above with PhantomConstructor, but appears that's actually wrong. It's a phantom getter method really.

@Gankro
Contributor
Gankro commented Dec 11, 2015

Neither PhantomFn nor MarkerTrait exist in today's Rust. This RFC's text is fairly outdated wrt to the current state of the type system.

PhantomData is the only mechanism one uses to specify extra semantics.

@nikomatsakis
Contributor

Seems like it'd probably be worth revising the RFC. I'm surprised I didn't do that. I guess I just wrote an internals thread on the topic? This one, specifically: https://internals.rust-lang.org/t/refining-variance-and-traits/1787

@aidanhs aidanhs referenced this pull request Aug 15, 2016
Merged

Fix typo in fn variance #1716

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment