Tracking issue for RFC 2115: In-band lifetime bindings

[aturon]

This is a tracking issue for the RFC "In-band lifetime bindings" (rust-lang/rfcs#2115).

Steps:

• Stabilize '_ (this has its own tracking issue, #48469)
• #15872: Enable elision in impl headers (mentoring instructions here)
  • Now tracked by impl_header_lifetime_elision in #15872
• #45667: Support '_ in where-clauses on functions. -- blocked on #15872
• #46042: Enable implicit binders.
• Lint against single-letter lifetimes not confined to functions (?)
• Lint against single-use lifetimes: #44752
  • note: @pnkfelix added this as a "prereq", but its possible we do not have consensus that it is required (though @pnkfelix believes it was part of the RFC...)
• Apply to rustc, experimenting with naming conventions (e.g., working through the affect on rustc).
• Adjust documentation (see instructions on forge)
• Stabilization PR (see instructions on forge)

Unresolved questions:

• What conventions should we use for lifetime names, and should they be enforced via lint?
  • Considerations:
    • To help avoiding accidental shadowing, make it easy to know where a lifetime name is bound (and in particular if it is bound in some enclosing, possibly out-of-sight, scope).
    • Permitting "well-known" names like 'tcx and 'gcx to be used consistently
  • Some prior proposals:
    • Names bound in impls are upper-case (but: fails to permit well-known names)
    • Single-letter names cannot be bound across scopes (but: can't easily tell if longer names are bound across scopes or not)
• Can we find an unequivocally-better syntax than '_ for elided lifetimes?
  • These are now stable, so we didn't

Known bugs (possibly incomplete list):

• #52532

[glaebhoerl]

@aturon Could we word the unresolved question as something like "what is the appropriate convention, if any" rather than "is this particular convention worthwhile"? Thanks!

[vitiral]

also, that wasn't the only suggested alternative. There is also completely explicit lifetimes using 'self::lifetime_name

[durka]

$ rg -F "impl<'a>" /rust/src | wc -l
     730

This supposedly ergonomic change is going to require updating a lot of code, including all of our docs and examples. I guess you could just do s/'a/'aa but that seems to miss the point. Could we make some kind of interactive rustfix tool to let you choose new lifetime names and have them automatically updated?

[aturon]

@glaebhoerl Done, thanks!

[nikomatsakis]

I expanded the unresolved question a bit.

[nikomatsakis]

Mentoring instructions

I'm not sure 100% the best way to implement this yet, but let me start by giving some pointers. Perhaps @eddyb can elaborate, as he is the last one to touch the relevant code in a serious way, as far as I know.

First thing is a kind of list of what the tasks are in the RFC. This is roughly the order in which I think we should approach them. It may even be worth breaking this out into separate issues.

XXX task list moved into main header

I don't have time for detailed mentoring notes, but here are a few things to start reading into. At present, region resolution etc kind of works like this:

• During HIR lowering, we insert placeholders for elided lifetimes.
• After HIR lowering, we run the code in resolve_lifetime.rs.
• This creates the NamedRegionMap that, for each hir::Lifetime, contains a Region struct indicating what region is being named.
  • This struct is a bit complicated. =)

[eddyb]

To support '_ we have to do very little:

• treat it like the existing placeholder lifetimes in hir::Lifetime::is_elided
• remove the error for literally using '_ as a lifetime

[SimonSapin]

Lint against single-letter lifetimes not confined to functions

Sorry that I come “after the battle” (this RFC was accepted during the pre-impl-period rush), but as @durka pointed out applying this lint to existing code bases will likely generate a lot of busy work for questionable benefits. I looked for the rationale for this lint but couldn’t find it. https://rust-lang.github.io/rfcs/2115-argument-lifetimes.html#lifetimes-in-impl-headers mentions a “convention”, but doesn’t say why enforcing it is important enough to warn by default. We don’t have such a lint for local variable names or field names, for example.

I assume that proposed lints are intended to warn by default. If they’re silent by default I have no concern.

[rpjohnst]

The idea is that there will no longer be <'a> sections in functions, so it will be harder to tell apart lifetime names from impl scope and those from function scope. This makes it tricky to know whether a name used in a function is intended to refer to an impl-scope lifetime, or to define a new one (and should thus be renamed).

[SimonSapin]

@rpjohnst Sorry, I have a hard time figuring out if your comment is arguing for or against something, and whether that thing is the lint I mentioned or some other change proposed by this tracking issue.

[rpjohnst]

@SimonSapin It's just trying to describe the reasoning behind the lint's existence, not necessarily argue for or against it.

[nikomatsakis]

I want to speak up in favor of having some signal that there are "hidden lifetimes" in types. The current notation of '_ is not my favorite, but I don't yet have a compelling alternative.

What I find is that I am constantly wanting to know whether a type contains references or not. It tells me a lot about how the type can be used: types without lifetimes can be cloned or copied and stored for arbitrary amounts of type. Types with lifetimes are always temporary in scope and confined to some caller (modulo longer lived arenas like 'tcx in the compiler, which are the exception, and also readily identifiable). In the specific case of a type that appears in the return value, I find it invaluable to know if there is an elided lifetime within, since it tells me how long the borrow on self will last.

I suspect that I derive outsized value from this notation because my intuitions for this stuff are stronger than most (can't imagine why that might be). But I feel like having a consistent syntax that highlights where lifetimes are and where they are not might wind up being very important for helping other users to develop these annotations.

OK, let me give a few examples that I've run across recently. Just yesterday I was reading into the futures code and I saw this:

fn poll(self: PinMut<Self>, cx: &mut task::Context) -> Poll<Self::Output>

Now, I know that a task::Context is an accumulation of bits of state. But I inferred from this signature that this state must not have any references associated with it and hence must be owned by task::Context. This seemed surprising to me. But when I clicked through to the definition, I saw that I was mistaken, and that indeed the Context type carries references:

rust/src/libcore/task/context.rs

Lines 18 to 25 in 79fcc58

	/// Information about the currently-running task.
	///
	/// Contexts are always tied to the stack, since they are set up specifically
	/// when performing a single `poll` step on a task.
	pub struct Context<'a> {
	local_waker: &'a LocalWaker,
	spawner: &'a mut dyn Spawn,
	}

Note: to make matters worse, rustdoc actually hides these references, so it's actually quite hard to figure out what is going on. Anyway, then I kept reading, and I encountered this signature:

pub fn will_wake(&self, other: &Waker) -> bool

Here, I was confused for quite a while, because I misremembered that Waker carried a "hidden" lifetime (in fact, it was Context). I knew that somehow poll was supposed to "store away" a copy of the Waker, so that it could notify the executor when it should be rescheduled, but I couldn't figure out how that was possible, since Waker carried references. It took some time for me to realize that I has misremembered and that Waker does not have references, so you can clone it and not be tied to any active borrow.

Another example comes from the rustc source, which for a long time contained a function like this:

pub fn predecessors_for(&self, bb: BasicBlock) -> MappedReadGuard<Vec<BasicBlock>>

It happens that MappedReadGuard holds a reference into self, so this borrow will last as long as the return value is in use. But could you guess that?

This same principle applies to compiler error messages. There, it's quite nice that we have something we can highlight to show you which reference we are talking about. e.g., we can now give errors like this:

fn foo(x: Blah<'_, T>) { .. }
               ^^ let's call this lifetime `'1`

Here, I imagine that we can find other solutions (and in general I'd like to work on improving these messages in lots of ways), but in general having a visual signal for where references are will always be useful to us, I think.

As far as consistency goes, I think there is value in trying to close the gap between the various ways that you write types in Rust. Right now, types that appear in function signatures and those that appear in struct declarations sometimes look really different (e.g., &Ref<T> vs &'a Ref<'a, T>). If we have '_, then there is at least a signal telling you "when you put this into a struct, it's going to need lifetime parameters". (I would still like to have some way to have a single anonymous lifetime parameter of a struct, so that one could write &Ref<T> as well, but that's neither here nor there.)

[nikomatsakis]

@stjepang

We tried using <'_> everywhere in librustc (see #53816), but I'm not sure the end result is great. Signatures like these are not pretty:

I agree that this is not pretty, but I'm not sure that just a plan TyCtxt is really better. I personally find it a bit "surprising" to see things like TyCtxt without any parameters, since it is relatively unusual, and it makes my mind feel like something is missing. I suspect that for newcomers to rustc it might also be hard to remember what things have lifetimes and which do not, though it's hard for me to judge.

There are also types for which writing lifetimes in parameters is not that helpful:

• std::pin::PinMut<'_, T>
• std::cell::Ref<'_, T>
• std::sync::MutexGuard<'_, T>

Speaking personally, I find the '_ there ugly but I do find it serves a purpose. Without that, when I see Ref<T>, my mind pattern matches it with Rc<T> or other such pointer types, and those two things are quite different.

I think ultimately I agree with @aturon's comment here:

I personally do find a visual signal of some kind to be quite helpful, but am sad that we weren't able to find something more appealing and lightweight.

[nikomatsakis]

I should also add that I appreciate that typing Ref<'_, T> is kind of... challenging. It feels like the sort of thing where I want an IDE that, once I type Ref, just fills in the <'_, _> or something. :)

[eddyb]

@nikomatsakis Did I miss a part of your comments, or did you not address the specific suggestion of only requiring explicit '_ when there is an implicit relationship (i.e. when elision is involved)?

It happens that MappedReadGuard holds a reference into self, so this borrow will last as long as the return value is in use. But could you guess that?

The elision aspect would cover that. And IMO I find that far more interesting than being able to tell whether a type is parametrized when looking at an universal function.

I'd rather have IDEs and rustdoc provide an expanded version on hover, than have to write in all the lifetimes even in signatures where they're inconsequential.

[nikomatsakis]

@eddyb I did not specifically address that. I agree that is better than the status quo. It might be a good place to start, given the controversy of the more encompassing proposal.

However, I don't find this to be true:

And IMO I find that far more interesting than being able to tell whether a type is parametrized when looking at an universal function.

That is to say, I personally get a lot of value from knowing whether types are parameterized when looking at universal functions. Often, when somebody pings me with some lifetime error and associated source, the first thing I do is to spend a while figuring out "where the lifetimes are". Currently, this requires jumping through to the source of all the various types involved.

I agree though that there is a cost of heavier notation.

[stjepang]

Is there a lint that warns against elided lifetimes in types (I mean everywhere, not just in return position)? I'd like to see how much of an effect that has on a few projects I'm familiar with -- and if it's not too bad, it might change my opinion.

[Nemo157]

@stjepang try cargo rustc -- -W elided-lifetimes-in-paths, I think that still has the full behaviour

[eddyb]

Or even RUSTFLAGS=-Welided-lifetimes-in-paths cargo build (although I don't know if that can affect dependencies, given cargo's "cap lints").

[stjepang]

Ok, I've just run the lint on all the crossbeam crates, rayon, ripgrep, and hyper. Here's what I learned...

The vast majority of the reported warnings are either missing lifetimes in return position or in fmt::Formatter. There are only a few remaining cases where we need to add lifetimes but it's no big deal, honestly. I'm pleasantly surprised by the low impact of the lint.

Still, the large PR (around 550 lines were changed) submitted on librustc was making me feel uneasy. I thought maybe the code in librustc is a particularly bad example that uses lifetimes heavily.

Then I went on to calculate the ratio of warnings per total lines of code in those projects. Perhaps librustc has a higher ratio of warnings per total lines of code? It turns out it doesn't - the ratio is not significantly higher. It's just that librustc is really big.

I think my opinion has changed so allow me to make a 180 turn and give a 👍 for requiring lifetimes in all types. If anyone else is feeling unconvinced, I encourage you to do the same thing and try to assess the real impact of the lint.

[Nemo157]

If anyone else is feeling unconvinced, I encourage you to do the same thing and try to assess the real impact of the lint.

In https://github.com/rust-lang-nursery/futures-rs this results in 511 warnings (from 15028 lines of code according to tokei (including tests which I did not try building with the warning)). Scanning the output I'm relatively confident that every single warning is for an input parameter that is not connected to the output in any way, so would result in no warning for the "explicit lifetimes when elision is used" variant.

[stjepang]

In rust-lang-nursery/futures-rs this results in 511 warnings (from 15028 lines of code according to tokei).

To be fair, futures is a bit of a special case because almost all warnings come from task::Context<'_> and PinMut<'_, Self>. And there's a whole lot of Future impls in there, which you're not supposed to implement by yourself anyway.

Besides, PinMut<'_, Self> will soon become Pin<&mut Self>, which cuts the number of warnings down from 511 to 216.

[cramertj]

Besides, PinMut<'_, Self> will soon become Pin<&mut Self>, which cuts the number of warnings down from 511 to 216.

And task::Context<'_> will become &LocalWaker, fixing the remaining half (see #54339).

[scottmcm]

Signatures like these are not pretty:

pub fn to_dep_node(self, tcx: TyCtxt<'_, '_, '_>, kind: DepKind) -> DepNode;

Undeveloped thought:

pub fn to_dep_node(self, tcx: TyCtxt<'..>, kind: DepKind) -> DepNode;

(By analogy to Foo(_, _, _) => Foo(..) patterns.)

[emilio]

Hmm, this lint produces 1514 warnings on Servo's style component alone, and the fix looks a lot more verbose and ugly :(

[emilio]

Most of them are actually unused lifetimes, as in, there's no relation with them and the output of the function... I think @eddyb's approach of only warning when elision actually comes at play it's way better than the current warning, getting the usefulness without the annoyance.

[eddyb]

@scottmcm I frankly disagree we need explicit lifetimes in that case.

[stjepang]

Here's a related pet peeve of mine. Currently, this compiles:

struct Foo<'a>(&'a ());
impl Foo<'_> {} // anonymous lifetime

But this doesn't:

struct Foo<'a>(&'a ());
impl Foo {} // elided lifetime

Sometimes requiring <'_> and sometimes not requiring it feels inconsistent. We should pick a side: either require it or don't.

[mark-i-m]

I wonder, would it be reasonable to stabilize only in-band lifetimes on functions (not methods)? AFAICT, there is no ambiguity about where those lifetimes come from, right?

fn foo(x: &'a usize) -> &'a usize { ... }

[mark-i-m]

To be clear, I mean that we would stabilize a useful subset now as long as it is future-compatible with the rest of in-band lifetimes.

[durka]

I question whether partially adding such a feature simplifies anything, since even if one believes leaving out lifetime declarations is a good thing, we would actually be adding *more* rules about where that's allowed. And it still makes situations like the following harder to read. ``` fn foo<'a>(_: &'a i32) { fn bar<'a>(_: &'a i32) { } fn baz(_: &'a i32) { } } ```

…

On Thu, Nov 15, 2018, 00:03 Who? Me?! ***@***.*** wrote: To be clear, I mean that we would stabilize a useful subset now as long as it is future-compatible with the rest of in-band lifetimes. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#44524 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC3n4pn0YWVwjUEQRnvRBmsv9CrZenHks5uvPWjgaJpZM4PVFyw> .

[mark-i-m]

Hmm... I hadn't thought about nested functions...

[briansmith]

I came across this issue when converting my code to Rust 2018 edition. I have lots of code that elides lifetimes in parameters and the rustfix tool suggested I make make changes to add explicit lifetime annotations. Consider the following:

Initial code:

pub struct S<'a>(&'a [u8]);

pub fn f(a: S) -> S { a }

My understanding is that omitting the lifetime parameters like that is deprecated and cargo fix --edition-idioms rewrites f as:

pub fn f(a: S<'_>) -> S<'_> { a }

However, then I generalized the initial code (&[u8] implements AsRef<[u8]> so the generalized code really is strictly more general than the previous code):

pub struct S<A: AsRef<[u8]>>(A);

pub fn f<A: AsRef<[u8]>>(a: S<A>) -> S<A> { a }

When I run cargo fix --edition-idioms, it doesn't make any lifetime annotations explicit in the generalized version; it leaves the code as-is. It seems really arbitrary to me to encourage the more generic code to elide the lifetime parameters but discourage the less generic code from doing so.

In general, Rust programmers need to be able to understand generic code and that means that they'll need to understand that there are potentially unseen lifetimes arguments implicitly in play when a reference type is used as the concrete type in a generic context. I think encouraging explicit lifetimes in non-generic contexts is only going to make it harder for people to understand this.

Also, over the course of maintaining my "idiomatic" code, I found that the unnecessary-but-idiomatic explicit lifetimes actually resulted in bugs. See briansmith/ring#767 in particular. That bug wouldn't have occurred if I had used the old style of eliding the lifetime.

Therefore, I think we shouldn't call the use of unnecessary (according to the compiler) explicit lifetimes "idiomatic".

I hope this can be reconsidered before JetBrains CLion replaces its "elide lifetimes" quick-fix with what cargo fix --edition-idioms does.

[cramertj]

@briansmith The difference I see between those two is that S<A> is already visibly generic and dependent upon a type argument that may have a non-'static lifetime. S, on the other hand, has no visible parameters and so it may not be obvious that it could be non-'static.

[Nemo157]

Re-briansmith/ring#767 the new "idiomatic" way to write that signature is

pub fn open_in_place<'a>(
    key: &OpeningKey, 
    nonce: Nonce, 
    aad: Aad<'_>, 
    in_prefix_len: usize, 
    ciphertext_and_tag_modified_in_place: &'a mut [u8]
) -> Result<&'a mut [u8], Unspecified>

which makes it explicit that the passed aad parameter has a lifetime, but that it is not involved in the output (same as key does semi-implicitly with the &). If cargo fix suggested to turn Aad into Aad<'a> there instead then that is a bug in the lint.