Remove markers and replace with a default rule for unused type/lifetime parameters

[nikomatsakis]

Inspired by @edwardw's PR rust-lang/rust#12828, this RFC proposes a small change to the behavior of variance inference if a type or lifetime parameter is unused. The goal is to more accurately capture user's intended behavior.

[bill-myers]

Could also make unused types and lifetimes a compiler error, and suggest to the user to apply a variance marker in the message.

[edwardw]

Then the marker types themselves become illegal.

[bill-myers]

The marker types have lang items, and those lang items would count as usage of the parameter.

[nrc]

Maybe it is just me, but from an engineering point of view, this seems like a really horrible idiom and we should not encourage it. If there is a need for this we should have some sort of explicit visibility thing in the language. I don't think we should change our type system to help its use.

[cmr]

It's not obvious to me why this is horrible or what the alternatives would be.

[nrc]

Sorry, that was strong language to use without explanation.

I meant it is horrible because it feels like a hack - you add an unused parameter to enforce using the struct in a certain scope - there is no indication of intent there. Someone reading the code would not know that that was what the lifetime parameter was for if they didn't know about the idiom. I would prefer something explicit to limit the usage, rather than implicitly relying on lifetime checking where there is not really a lifetime (from the perspective of inside the strucure). I also have no idea what such a thing would look like though.

[nikomatsakis]

Well, we have an existing design that gives a hint at what it looks like -- marker::ContravariantLifetime<'a>

[nrc]

I would encourage invariant as the default for types. I think programmers should assume invariant subtyping, even though covariant is the intuitive thing. We should not encourage this wrong-thinking.

[nrc]

I would expect this to make Foo invariant wrt T since field can be both read and written. If we are talking about coercion rather than subtyping this makes sense and so does the covariant default above.

[nikomatsakis]

This is not how Rust works, though. Mutability is imposed from the outside: field is not mutable, in this declaration.

[nikomatsakis]

(Note that if you had field: RefCell<T>, that'd be different (and the algorithm would do the right thing))

[nrc]

So you will infer different variances for Foo<T> and mut Foo<T> (covariant and invariant, respectively)? (My mistake was assuming you propose inferring variance per poly-type, vs per type).

[cmr]

No, because mut Foo<T> does not exist. There is no such thing as a mutable field. It would do the right thing with RefCell<T> because RefCell would need to be invariant with respect to T since it contains Unsafe<T>

[cmr]

'a is used here?

[nikomatsakis]

Right, I'll patch that.

[nikomatsakis]

@bill-myers yes, you're right, I should at least list "error" as an alternative. It's also appealing to force explicitness. I think my hope was that we could remove -- or almost remove -- markers entirely. In fact, given the rules here, I think we could remove them entirely, since they could be defined "in language".

For example, ContravariantType could be defined like:

struct CovariantType<T>; // just the default behavior
struct ContravariantType<T> {
    marker: CovariantType<fn(T)>
}
struct InvariantType<T> {
    marker: CovariantType<fn(T) -> T>
}

Similar tricks should work for lifetimes.

[pnkfelix]

Wouldn't a lint (warning you about bivariant type parameters or lifetime parameters) deal with this? The lint could then point the programmer at the std::kinds::marker module, and then the programmer who had some particular intuition about what that phantom type was for could actually explicitly write out their intent.

(You have of course mentioned std::kinds::marker below. I just want to understand if the purpose of this RFC is to reduce risk, which I think a lint will be better at, or for programmer convenience, which is where choosing the defaults to match expectations (or perhaps to match actual frequency of occurrence) would be relevant.)

((ugh, this is exactly what bill-myers said above. Sorry for the noise.. :( ))

[pnkfelix]

(Ah, niko's response to bill-myers implies to me that there are indeed goals here beyond just reducing risk. That is fine, though I think it may make the RFC stronger to list those other goals explicitly, if possible.)

[nikomatsakis]

On Tue, Mar 18, 2014 at 09:07:33AM -0700, Felix S Klock II wrote:

(Ah, niko's response to bill-myers implies to me that there are goals here beyond just reducing risk. That is fine, though I think it may make the RFC stronger to list those other goals explicitly, if possible.)

I released the RFC prematurely, I think. I updated it. I'm also
amenable to making unused parameters an error, though I am tempted by
the ability to remove markers altogether.

[nikomatsakis]

I just pushed a revised version of the RFC.

[cmr]

I agree with @bill-myers here. Unused type/lifetime parameters should be an error, or at the very least a deny-by-default lint. See https://botbot.me/mozilla/rust/msg/12337250/ for confusion in the wild. Especially since this is such a sublte issue (at least IMO) it's better to have compiler help than just documentation.

[cmr]

Actually I take that back. I re-read the RFC and am in favor. +1

[nikomatsakis]

There is one complication -- I hadn't considered cycles among types,
such that a lifetime parameter is unused even though it appears
textually. The ideas of the RFC are still applicable, but I have to
word it more carefully to account for this possibility. Basically I
imagine we have to run the current inference algorithm and then go
back, patch up any bivariant things to covariant/contravariant as
appropriate, and re-run the algorithm till we again reach a fixed
point. This is kind of a technicality of the implementation, to some
extent.

(I'm a bit torn about this myself; I definitely prefer error to the
current situation, but the idea of removing markers and just doing
"the right thing" is appealing. I guess the question hinges on how
often the default behavior I proposed is in fact the right thing.)

[lifthrasiir]

@cmr Unused lifetime parameters are sometimes useful, as in https://github.com/mozilla/rust/blob/ce62032/src/libstd/fmt/mod.rs#L1167-L1179 which allows for macros with optionally used lifetime parameters.

[huonw]

cc rust-lang/rust#5922 (@lifthrasiir, that macro is actually incorrect if we ever get hygienic lifetimes; and if we don't then one should be able to disable the warning lint (assuming it's a lint...).)

[gasche]

I have bad feelings about the proposed behavior in this RFC. My impression is that ML languages have under-estimated the importance of bivariance for too long (e.g. there is no syntax for that in the OCaml surface syntax), and that the discussed change makes Rust take the same road.

I think it is important, for the language to be "complete" and robust to unplanned scenarios, that bivariance of parameters (type or lifetime) be supported. Given that std::kind::marker does not have a BivariantType marker (which is also a sign of Rust's design neglecting bivariance), how would one specify that a type is bivariant if the variance inference is tweaked not to detect it? It is very likely that I misunderstood the proposed change and that there is a way to opt-in to bivariance -- in which case I think the RFC could be debated for taste reasons, but is acceptable.

An alternative proposal that I think would solve the problem of user surprise at bivariant parameters is the following: warn on "unused" type and lifetime parameters, with a warning message that mentions the problem and prompt people to explicitly add a variance annotation (or marker type or, better in most cases, Unsafe with its proposed invariant semantics) and enable '_a or _T to avoid that warning, just like with unused mutable variables.

In ML, bivariant type parameters are very useful to build "phantom types" (which rely on module abstraction). The idea is to have a type definition foo<T> that does not use T, but pretend in the module/crate's interface that it does (by adding an explicit variant annotation there). Internally you can and do convert between foo's of various types, but externally you enforce a stricter typing discipline. This allows the library designer to rely on the type system to impose usage constraints to the user, in a fairly flexible way -- phantom types have been largely superseded by GADTs, by those also add a layer of complexity. Even though current's Rust design does not encourage (or allow? I'm not sure) phantom types, it would be a loss to block that possibility for the future as well.

One could also consider scenarios where users may want to remove fields of a struct declaration, without changing its type parameters (to avoid having to update too much type annotations throughout the program). If the field was the only use of one of the type parameters, bivariance may be the variance you want.

The last argument would be that Niko's proposed change to the current inference algorithms makes the logic sensibly more complex; simple is always better, especially in a type system.

(Another experience I had in ML-land is that variance have very different meanings for the author and the users of a library, that people tend to conflate. There are two interpretations for F<T> being covariant, one being "if A is a subtype of B then F<A> is a subtype of F<B>, the other being "if F<A> is a subtype of F<B>" then by inversion A is a subtype of B", and we tend to conflate the two, while the latter can be fairly dangerous once you have abstract types and variance annotations at interface boundaries. I suspect the reason why people see no problem in not having a Bivariant or defaulting to Covariant also comes from this kind of blurred views.)

[nikomatsakis]

I have bad feelings about the proposed behavior in this RFC. My
impression is that ML languages have under-estimated the importance
of bivariance for too long (e.g. there is no syntax for that in the
OCaml surface syntax), and that the discussed change makes Rust take
the same road.

This is helpful feedback. I have indeed intentionally neglected
bivariance because I do not believe it to be of use. I'm also not sure
that your example convinces me otherwise, but I'm amenable to
persuasion.

In ML, bivariant type parameters are very useful to build "phantom
types" (which rely on module abstraction). The idea is to have a
type definition foo<T> that does not use T, but pretend in the
module/crate's interface that it does (by adding an explicit variant
annotation there).

I don't think Rust has an equivalent to this ML separation between
interface and implementation. That is, if foo were bivariant with
respect to T, that would be true both within the module itself and
externally, making your phantom type useless.

I think the Rust-y way to do what you are describing would be to have
your type be covariant (or invariant, it doesn't really matter if
there aren't subtyping relationships between your phantom types) and
provide a (private) means to interconvert freely between distinct
phantom types (perhaps via transmute).

Right now, at least as much as I understand how bivariance plays out,
it essentially means that a particular type/lifetime parameter is just
irrelevant with respect to subtyping. The only way I can see this
being relevant is if, for some reason, you were forced to add a type
parameter to your type to conform to some interface, but it really
isn't used or relevant. I've never encountered this scenario, though;
one can certainly implement traits that require generic parameters
even if your type does not require them. Maybe with HKT such a
scenario might arise.

[nikomatsakis]

On Tue, Mar 25, 2014 at 09:19:10AM -0700, gasche wrote:

(Another experience I had in ML-land is that variance have very
different meanings for the author and the users of a library, that
people tend to conflate. There are two interpretations for F<T>
being covariant, one being "if A is a subtype of B then F<A>
is a subtype of F<B>, the other being "if F<A> is a subtype of
F<B>" then by inversion A is a subtype of B", and we tend to
conflate the two, while the latter can be fairly dangerous once you
have abstract types and variance annotations at interface
boundaries. I suspect the reason why people see no problem in not
having a Bivariant or defaulting to Covariant also comes from
this kind of blurred views.)

Maybe my view is blurred, but I don't really understand this point.

[gasche]

Sorry for being unclear. I guess my general point is that not being able to express bivariance could become, not a major blocker, but at least an occasional pain, if combined with other type-system features that are not present in Rust today. Formulated as is, you see that it is a rather weak objection from a pragmatic point of view. It would not be worth a design change if the present design naturally omitted bivariance; but it does suggest that willingly making your system slightly less regular and more complex to remove bivariance (as proposed in this RFC) may be a cause of regret in a (far) future.

The two features I considered were:

1. Interface/Implementation separation, abstract types, or functors (code parametrized on modules). If you use them, you will encounter situations where the user of a library (or, in the functor case, module parameter) programmed against a parametrized datatype, but the implementor of the library for some reason does not need the parameter. In most cases it is not important that the declaration be accepted as bivariant, but it may need to be contravariant rather than covariant. Only for the "phantom types" use-case is it important that the implementor can rely on bivariance (of course you can use an unsafe cast, but you want to avoid that as much as possible as it increases the cost of maintenance, code audits, debugging etc.).
2. Subtyping or equality constraints (possibly through GADTs, bounded polymorphism, etc.). If the typing environment tells you that Foo<A> is equal to Foo<B>, can you deduce that A is equal to B? Today (before your RFC), the answer is "yes" precisely when Foo is inferred to be covariant, contravariant or invariant, and "no" if it is inferred bivariant. Tomorrow (if your RFC is applied as is), you will be in the tricky situation where for some covariant types, the answer is "no". This is not immediately a cause of concern (I don't see any part of today's Rust that would need to make such deduction, and where soundness could be at stake), but this very situation caused unsoundness in the OCaml type checker for example (found and fixed last year), and it would remain a potential pain point in any type system with subtyping. Keeping bivariance an explicit option of the system can help to keep ourselves honest with respect to this issue.

[nikomatsakis]

On Wed, Mar 26, 2014 at 05:57:45AM -0700, gasche wrote:

It would not be worth a design change if the present design
naturally omitted bivariance; but it does suggest that willingly
making your system slightly less regular and more complex to remove
bivariance (as proposed in this RFC) may be a cause of regret in a
(far) future.

It is also plausible to imagine a future way to explicitly "opt in" to
bivariance. I'll take a look at your paper on GADTs and subtyping and
try to get a better understanding of what you're getting at.

Tomorrow (if your RFC is applied as is), you will be in the tricky situation where for some covariant types, the answer is "no".

I don't quite follow this part. While I don't know why we would want
to deduce that A <: B from the facts that T<A> <: T<B> and T is
covariant with respect to its parameter, it seems to me that it would
be reasonable to do so. I don't know why it matters whether we decided
that T is covariant because of a defaulting scheme or because of a
marker type or other explicit notation.

[flaper87]

I'm not an expert on this field but based on my understanding, I like this proposal. I like the fact that it clears out the inference algorithm by removing possible weird cases that users may actually ignore (which I think makes the type-system safer) and it removes the variance markers.

That said, I think having a way to opt-in to bivariance is also important. The whole point of this RFC should be to make the type inference safer without forbidding other things - which follows pretty much what is being done for kinds.

As a final note, I think the lint that has been mentioned could be implemented now and warn the user about unused types and lifetimes.

[nikomatsakis]

Just a brief update: I am not sure if I still like this plan. In weekly meeting we decided to do some measurements (which I still haven't done) to try and understand how often this comes up.

[eddyb]

I was really worried about this, but now I like the idea, seeing how ContravariantType and CovariantLifetime can be emulated.