Make closure capturing have consistent and correct behaviour around patterns

[meithecatte]

This PR has two goals:

• firstly, it fixes internal compiler error: two identical projections #137467. In order to do so, it needs to introduce a small breaking change surrounding the interaction of closure captures with matching against enums with uninhabited variants. Yes – to fix an ICE!
  • this also fixes ICE: called Option::unwrap() on a None value with refutable patterns #138973, a slightly different case with the same root cause.
• secondly, it fixes Closure captures are inconsistent between x and x @ _ irrefutable patterns #137553, making the closure capturing rules consistent between let patterns and match patterns. This is new insta-stable behavior.

Background

This change concerns how precise closure captures interact with patterns. As a little known feature, patterns that require inspecting only part of a value will only cause that part of the value to get captured:

fn main() {
    let mut a = (21, 37);
    // only captures a.0, writing to a.1 does not invalidate the closure
    let mut f = || {
        let (ref mut x, _) = a;
        *x = 42;
    };
    a.1 = 69;
    f();
}

I was not able to find any discussion of this behavior being introduced, or discussion of its edge-cases, but it is documented in the Rust reference.

The currently stable behavior is as follows:

• if any pattern contains a binding, the place it binds gets captured (implemented in current walk_pat)
• patterns in refutable positions (match, if let, let ... else, but not destructuring let or destructuring function parameters) get processed as follows (maybe_read_scrutinee):
  • if matching against the pattern will at any point require inspecting a discriminant, or it includes a variable binding not followed by an @-pattern, capture the entire scrutinee by reference

You will note that this behavior is quite weird and it's hard to imagine a sensible rationale for at least some of its aspects. It has the following issues:

• firstly, it assumes that matching against an irrefutable pattern cannot possibly require inspecting any discriminants. With or-patterns, this isn't true, and it is the cause of the internal compiler error: two identical projections #137467 ICE.
• secondly, the presence of an @-pattern doesn't really have any semantics by itself. This is the weird behavior tracked as Closure captures are inconsistent between x and x @ _ irrefutable patterns #137553.
• thirdly, the behavior is different between pattern-matching done through let and pattern-matching done through match – which is a superficial syntactic difference

This PR aims to address all of the above issues. The new behavior is as follows:

• like before, if a pattern contains a binding, the place it binds gets captured as required by the binding mode
• if matching against the pattern requires inspecting a disciminant, the place whose discriminant needs to be inspected gets captured by reference

"requires inspecting a discriminant" is also used here to mean "compare something with a constant" and other such decisions. For types other than ADTs, the details are not interesting and aren't changing.

The breaking change

During closure capture analysis, matching an enum against a constructor is considered to require inspecting a discriminant if the enum has more than one variant. Notably, this is the case even if all the other variants happen to be uninhabited. This is motivated by implementation difficulties involved in querying whether types are inhabited before we're done with type inference – without moving mountains to make it happen, you hit this assert:

rust/compiler/rustc_middle/src/ty/inhabitedness/mod.rs

Line 121 in 43f0014

debug_assert!(!self.has_infer());

Now, because the previous implementation did not concern itself with capturing the discriminants for irrefutable patterns at all, this is a breaking change – the following example, adapted from the testsuite, compiles on current stable, but will not compile with this PR:

#[derive(Clone, Copy, PartialEq, Eq, Debug)]
enum Void {}

pub fn main() {
    let mut r = Result::<Void, (u32, u32)>::Err((0, 0));
    let mut f = || {
        let Err((ref mut a, _)) = r;
        *a = 1;
    };
    let mut g = || {
    //~^ ERROR: cannot borrow `r` as mutable more than once at a time
        let Err((_, ref mut b)) = r;
        *b = 2;
    };
    f();
    g();
    assert_eq!(r, Err((1, 2)));
}

Is the breaking change necessary?

One other option would be to double down, and introduce a set of syntactic rules for determining whether a sub-pattern is in an irrefutable position, instead of querying the types and checking how many variants there are.

This would not eliminate the breaking change, but it would limit it to more contrived examples, such as

let ((true, Err((ref mut a, _, _))) | (false, Err((_, ref mut a, _)))) = x;

In this example, the Errs would not be considered in an irrefutable position, because they are part of an or-pattern. However, current stable would treat this just like a tuple (bool, (T, U, _)).

While introducing such a distinction would limit the impact, I would say that the added complexity would not be commensurate with the benefit it introduces.

The new insta-stable behavior

If a pattern in a match expression or similar has parts it will never read, this part will not be captured anymore:

fn main() {
    let mut a = (21, 37);
    // now only captures a.0, instead of the whole a
    let mut f = || {
        match a {
            (ref mut x, _) => *x = 42,
        }
    };
    a.1 = 69;
    f();
}

Note that this behavior was pretty much already present, but only accessible with this One Weird Trick™:

fn main() {
    let mut a = (21, 37);
    // both stable and this PR only capture a.0, because of the no-op @-pattern
    let mut f = || {
        match a {
            (ref mut x @ _, _) => *x = 42,
        }
    };
    a.1 = 69;
    f();
}

Implementation notes

The first commits of the PR perform various cleanups. The action happens in two parts:

• "ExprUseVisitor: properly report discriminant reads" makes walk_pat perform all necessary capturing. This is the part that fixes internal compiler error: two identical projections #137467.
• "ExprUseVisitor: remove maybe_read_scrutinee" removes the unnecessary "capture the entire scrutinee" behavior, fixing Closure captures are inconsistent between x and x @ _ irrefutable patterns #137553.

The new logic stops making the distinction between one particular example that used to work, and another ICE, tracked as #119786. As this requires an unstable feature, I am leaving this as future work.

[rustbot]

This PR changes a file inside tests/crashes. If a crash was fixed, please move into the corresponding ui subdir and add 'Fixes #' to the PR description to autoclose the issue upon merge.

[meithecatte]

Nadrieril suggested that this should be resolved through a breaking change – updated the PR description accordingly.

@rustbot label +needs-crater

r? @Nadrieril

[rustbot]

Error: Label needs-crater can only be set by Rust team members

Please file an issue on GitHub at triagebot if there's a problem with this bot, or reach out on #t-infra on Zulip.

[meithecatte]

@compiler-errors You've requested that the fix for #137553 land in a separate PR. However, ironically, the breaking changes are actually required by #137467 and not #137553. Do you think the removal of the now-obsolete maybe_read_scrutinee should happen in a separate PR, or should I do it here so that it also benefits from the crater run?

[compiler-errors]

We can crater both together if you think they're not worth separating. I was just trying to accelerate landing the parts that are obviously-not-breaking but it's up to you if you think that effort is worth it or if you're willing to be patient about waiting for the breaking parts (and FCP, etc).

@bors try

[bors]

⌛ Trying commit 8ed61e4 with merge 3b30da3...

[meithecatte]

We can crater both together if you think they're not worth separating. I was just trying to accelerate landing the parts that are obviously-not-breaking but it's up to you if you think that effort is worth it or if you're willing to be patient about waiting for the breaking parts (and FCP, etc).

That's the thing – one part is a breaking change, the other introduces insta-stable new behavior. There's no easily mergeable part to this.

[compiler-errors]

could we give this a less weird pr title pls 💀

@bors try

[Nadrieril]

Dear @rust-lang/lang, this PR proposes a breaking change to the language to fix some bizarre edge cases around precise closure captures. Crater found a single breakage, in a GitHub project. WDYT?

[traviscross]

If we were to OK this, we'd end up asking for a PR to be made to the affected project, particularly as it seems to be maintained. Probably you'll want to go ahead and do this now in any case, as it will make the story a bit simpler when we pick this up.

[traviscross]

Fortunately also, the regression is in the main.rs file of a bin crate, so there won't be any dependent crates affected.

[lino-levan]

PR merged on our end. Thanks for letting us know.

[traviscross]

Maybe?:

Suggested change

	#[derive(Clone, Copy, PartialEq, Eq, Debug)]
	enum Void {}
	pub fn main() {
	let mut r = Result::<Void, (u32, u32)>::Err((0, 0));
	let mut f = || {
	let Err((ref mut a, _)) = r;
	*a = 1;
	};
	let mut g = || {
	//~^ ERROR: cannot borrow `r` as mutable more than once at a time
	let Err((_, ref mut b)) = r;
	*b = 2;
	};
	f();
	g();
	assert_eq!(r, Err((1, 2)));
	}
	enum Void {}
	pub fn main() {
	let mut r = Result::<Void, (u32, u32)>::Err((0, 0));
	let mut f = || {
	let Err((ref mut a, _)) = r;
	*a = 1;
	};
	let mut g = || {
	//~^ ERROR: cannot borrow `r` as mutable more than once at a time
	let Err((_, ref mut b)) = r;
	*b = 2;
	};
	f();
	g();
	assert!(matches!(r, Err((1, 2))));
	}

I like doing away with those derives if they're not part of the demonstration.

[traviscross]

During closure capture analysis, matching an enum against a constructor is considered to require inspecting a discriminant if the enum has more than one variant. Notably, this is the case even if all the other variants happen to be uninhabited. This is motivated by implementation difficulties involved in querying whether types are inhabited before we're done with type inference...

I.e.:

enum Void {}

pub fn main() {
    let mut r = Result::<Void, (u32, u32)>::Err((0, 0));
    let mut f = || {
        let Err((ref mut a, _)) = r;
        *a = 1;
    };
    let mut g = || {
    //[stable]~^ OK
    //[post-pr]~^ ERROR: cannot borrow `r` as mutable more than once at a time
        let Err((_, ref mut b)) = r;
        *b = 2;
    };
    f();
    g();
    assert!(matches!(r, Err((1, 2))));
}

This is my only reservation about this, as this is a bit of a strange and unfortunate outcome as a language matter. The story we want to tell about uninhabited types is that Result<!, (u32, u32)> should basically work like (u32, u32), and it wouldn't in this case.

I believe you about the implementation difficulties. It is a bit surprising, though, given that we need to know that Void is uninhabited to allow those irrefutable pattern matches. Do we defer that check somehow to after type checking? (Is it possible to defer the check needed for this capture?) I'm curious if you could say more about that.

If it weren't for the fact that this kind of accidentally worked for other reasons, it'd be easy to say, "it's OK; we could always make this work later." But it feels less ideal to say that when we're incurring breakage.

Other than this reservation, this all seems right to me.

[meithecatte]

It is a bit surprising, though, given that we need to know that Void is uninhabited to allow those irrefutable pattern matches. Do we defer that check somehow to after type checking?

Well, as far as closure capture goes, we don't check this at all – we tacitly assume that matching against any pattern accepted for a let will not require matching the discriminant. And when this assumption gets invalidated, we ICE. Essentially, the pattern capture behavior for let was always broken – and the equivalent match never worked in this manner.

(Is it possible to defer the check needed for this capture?)

I believe this is the only instance where we would need to inspect whether a type is uninhabited before type inference is done. The only way we could properly delay it would be to do minimal capture analysis as the very last step of type inference – and make sure that we still properly handle cases where this leads to a cycle.

This is my only reservation about this, as this is a bit of a strange and unfortunate outcome as a language matter. The story we want to tell about uninhabited types is that Result<!, (u32, u32)> should basically work like (u32, u32), and it wouldn't in this case.

Well, there's already the difference that you can't use the x.0 syntax to access the fields of a Result<!, (u32, u32)>. I'd say that the never type here should affect layout and exhaustiveness checks, but otherwise behave like any other Result<T, E> – which is the direction this PR is going.

This is consistent with the interpretation that feature(never_patterns) is going for: that matching against a Result<!, (u32, u32)> like this is syntax sugar for a match with an Ok(!) pattern, which would semantically read the discriminant, and a field of type ! in one of the branches, only at which point does the uninhabited branch evaporate in a puff of smoke.

If it weren't for the fact that this kind of accidentally worked for other reasons, it'd be easy to say, "it's OK; we could always make this work later." But it feels less ideal to say that when we're incurring breakage.

Technically, this is true, but the fact that pattern-matching is included in precise captures isn't a very well-known feature in the first place, and Crater didn't find any concrete instance of this breakage. Currently, we let this through, at the cost of ICE-ing on perfectly reasonable code like in #137467.

[traviscross]

This is consistent with the interpretation that feature(never_patterns) is going for: that matching against a Result<!, (u32, u32)> like this is syntax sugar for a match with an Ok(!) pattern, which would semantically read the discriminant, and a field of type ! in one of the branches, only at which point does the uninhabited branch evaporate in a puff of smoke.

OK, I buy that. Looks good to me then. I propose we take the breakage and the new behavior as described.

@rfcbot fcp merge

[rfcbot]

Team member @traviscross has proposed to merge this. The next step is review by the rest of the tagged team members:

• @joshtriplett
• @nikomatsakis
• @scottmcm
• @tmandry
• @traviscross

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

cc @rust-lang/lang-advisors: FCP proposed for lang, please feel free to register concerns.
See this document for info about what commands tagged team members can give me.

[RalfJung]

This all makes sense to me at first sight, in particular the part about uninhabited variants in the last 2 comments. From a Miri perspective, we'll want to see a discriminant read at that point in the closure so that we can report UB as appropriate; this can't even work if the discriminant does not get captured in the first place. That said, I'd definitely like to hear from @Nadrieril before we land this. :)

Would it be possible to have a Miri test like this? I'm thinking of a repr(C) enum with 2 inhabited and an uninhabited variant, and then capturing &Enum that points to something invalid (i.e., the uninhabited variant), and then a let pattern like let (Var1 | Var2) = x; or so... IIUC, this so far would not have captured (since it's a let). Now with this PR, calling the closure should cause UB due to the invalid enum (but there should be no UB before since Miri does not enforce validity behind references).

[meithecatte]

Would it be possible to have a Miri test like this? I'm thinking of a repr(C) enum with 2 inhabited and an uninhabited variant, and then capturing &Enum that points to something invalid (i.e., the uninhabited variant), and then a let pattern like let (Var1 | Var2) = x; or so... IIUC, this so far would not have captured (since it's a let). Now with this PR, calling the closure should cause UB due to the invalid enum (but there should be no UB before since Miri does not enforce validity behind references).

This should pass, and I'll investigate creating a test like that. But do note that currently, the lowering actual MIR lowering code does check for inhabitedness, so a test with one inhabited variant, and one uninhabited one, would not pass, because the read of the discriminant would be skipped. To do this in an "honest" manner, we would want to also break compat outside of closures as well, and make the actual lowering perform a read in that case, bringing the semantics into a consistent state between the two.

[RalfJung]

Hm, I think we only skip the read in cases where it cannot fail since the place is by-val, or were there still some gaps in that? Though if let only does this for by-val cases we will not be able to do it in a closure either. I though there should be a test here that is UB with this PR but was accepted before. But maybe not.

[meithecatte]

Hm, I think we only skip the read in cases where it cannot fail since the place is by-val, or were there still some gaps in that? Though if let only does this for by-val cases we will not be able to do it in a closure either.

I think we're talking about a different check. See this part of the code, where if we're matching the only inhabited variant, we don't emit the TestCase that would read its discriminant into the decision tree.

rust/compiler/rustc_mir_build/src/builder/matches/match_pair.rs

Lines 269 to 282 in 7bfd952

	PatKind::Variant { adt_def, variant_index, args, ref subpatterns } => {
	let downcast_place = place_builder.downcast(adt_def, variant_index); // `(x as Variant)`
	cx.field_match_pairs(&mut subpairs, extra_data, downcast_place, subpatterns);
	let irrefutable = adt_def.variants().iter_enumerated().all(|(i, v)| {
	i == variant_index
	|| !v.inhabited_predicate(cx.tcx, adt_def).instantiate(cx.tcx, args).apply(
	cx.tcx,
	cx.infcx.typing_env(cx.param_env),
	cx.def_id.into(),
	)
	}) && !adt_def.variant_list_has_applicable_non_exhaustive();
	if irrefutable { None } else { Some(TestCase::Variant { adt_def, variant_index }) }
	}

[RalfJung]

That's deep in the guts of pattern elaboration, I cannot follow that code or relate it to the MIR we generate.^^ I am entirely talking about what it looks like in MIR. Though maybe I am entirely on the wrong train here. This PR does not change the MIR in the closure, does it? But it does mean more things get captured in some cases which should leave a trace in the MIR where the closure is constructed?

[meithecatte]

This PR does not change the MIR in the closure, does it?

Pretty much. It only changes what gets captured into the closure, and by extension the exact place expressions MIR later uses to access these upvars.

But it does mean more things get captured in some cases which should leave a trace in the MIR where the closure is constructed?

Yes, more things in some cases (#137467), less things in others (#137553).

Though if let only does this for by-val cases we will not be able to do it in a closure either.

To the best of my knowledge, this PR makes let and match equivalent in terms of closure capture behavior.

Hm, I think we only skip the read in cases where it cannot fail since the place is by-val, or were there still some gaps in that?

Okay, looks like I was not aware of some of the behavior here.

As it turns out, exhaustiveness only allows omitting the uninhabited branch when matching by value.

I tried this test case:

enum Void {}

enum Foo {
    A(u32),
    B(Void),
}

fn check_foo(x: &Foo) {
    match x {
        Foo::A(x) => {},
    }
}

and got this error, which I haven't seen before:

error[E0004]: non-exhaustive patterns: `&Foo::B(_)` not covered
 --> patlowering.rs:9:11
  |
9 |     match x {
  |           ^ pattern `&Foo::B(_)` not covered
  |
note: `Foo` defined here
 --> patlowering.rs:3:6
  |
3 | enum Foo {
  |      ^^^
4 |     A(u32),
5 |     B(Void),
  |     - not covered
  = note: the matched value is of type `&Foo`
  = note: `Void` is uninhabited but is not being matched by value, so a wildcard `_` is required
help: ensure that all possible cases are being handled by adding a match arm with a wildcard pattern or an explicit pattern as shown
  |
10~         Foo::A(x) => {},
11~         &Foo::B(_) => todo!(),
  |

This means that we only get to the code I linked when the uninhabitedness doesn't require reading through any references.

In this case, I will try to add a testcase like you described. Off the top of your head, are you aware of any similar tests that create enum values with invalid discriminants? I'm not sure how I'd go about doing that, I must admit unsafe code is not my strong suit.

[RalfJung]

To the best of my knowledge, this PR makes let and match equivalent in terms of closure capture behavior.

Right, but are they equivalent in terms of the MIR that is generated for the pattern?

In this case, I will try to add a testcase like you described. Off the top of your head, are you aware of any similar tests that create enum values with invalid discriminants? I'm not sure how I'd go about doing that, I must admit unsafe code is not my strong suit.

For this case I'd do something like:

#![feature(never_type)]

#[repr(C)]
enum E {
  V1, // discrminant: 0
  V2, // 1
  V3(!), // 2
}

fn main() {
    assert_eq!(std::mem::size_of::<E>(), 4);
    
    let val = 2u32;
    let ptr = (&raw const val).cast::<E>();
    // This is invalid:
    unsafe { ptr.read() };
}