Tracking issue for RFC 2008: Future-proofing enums/structs with #[non_exhaustive] attribute

[scottmcm]

This is a tracking issue for the RFC "Future-proofing enums/structs with #[non_exhaustive] attribute" (rust-lang/rfcs#2008).

Steps:

• Implement the RFC (mentoring instructions below) -- done in PR #45394
• Adjust documentation (see instructions on forge)
• Stabilization PR (see instructions on forge)

Issues to be resolved prior to stabilization

• #[non_exhaustive] is not yet implemented for enum variants.
• Current behavior on unions. The current implementation (#44109) sometimes ignores non_exhaustive on unions, sometimes reports an error.
• In addition, the attribute is not currently validated (neither syntactic form nor location), so it can be written in weird forms (#[non_exhaustive(foo)]) and in arbitrary locations, including suspicious ones like traits (#[non_exhaustive] trait Trait {}).
• Fix #53549

[eddyb]

I'd cc @rust-lang/compiler instead.

[tinaun]

im working on a pr for this. how should tests be structured?

[tinaun]

also: should there be a lint for #[non_exhaustive] on non public items, since it doesn't really do anything crate-locally?

[clarfon]

@tinaun shouldn't they all go under the useless_attribute lint? That includes both on non-pub items and structs with private fields.

[nikomatsakis]

@tinaun I would expect tests for this to be scattered about in the run-pass and compile-fail and ui directories:

• run-pass would be used to test things that should compile
• ui is usually used to test for things that don't compile (compile-fail can also be used, but I prefer ui personally)
  • if you want to test that a lint is issued, you can use #[deny] to convert the warning into an error

Just to keep things organized, I would make a subdirectory like src/test/run-pass/nonexhaustive-rfc2008 (and same for ui) to contain the tests.

So I would imagine using ui tests to check for:

• Errors when exhaustively matching (without _) something from another crate that is tagged #[nonexhaustive]

The run-pass might be used to test that an exhaustive match within the same crate compiles just fine.

[nikomatsakis]

@tinaun any progress? need any tips?

[tinaun]

I'm currently away for the weekend but I would love to have some help with understanding the implementation of privacy in the complier. sorry for forgetting about this!

[petrochenkov]

So, this feature shouldn't require too many changes to implement.

• First, we need to require .. in patterns, this needs to be done in fn check_struct_pat_fields in librustc_typeck/check/_match.rs
• Then we need to prohibit any struct expressions (with .. or not, doesn't matter), this needs to be done in fn check_expr_struct in librustc_typeck/check/mod.rs
• After that we need to tweak visibilities for struct constructors. This currently requires changes to three places - 1) fn build_reduced_graph_for_item/ItemKind::Struct + fn build_reduced_graph_for_variant in librustc_resolve 2) fn encode_struct_ctor and fn encode_enum_variant_info in librustc_metadata and 3) fn def_id_visibility in librustc_privacy.
  I all cases visibility of a struct/variant with non_exhaustive attribute should be lowered to pub(crate) if it was pub.
  EDIT: Separating visibilities for enum variants and their constructors may potentially require some larger infrastructural changes because they share DefId currently, so non_exhaustive on tuple/unit variants can be omitted from initial implementation.
• Test everything! In this case it's probably better to write failing tests before doing anything else and then make them pass gradually.
• EDIT: I completely forgot about non_exhaustive on enums and match exhaustiveness checks.

[nikomatsakis]

@tinaun just checking in -- did those instructions help? Still got questions? =)

UPDATE: @tinaun responded on Gitter.

[davidtwco]

Hoping to take a stab at this after speaking with @nikomatsakis in Gitter. @tinaun are you still working on this?

Update: @tinaun replied on Gitter.

[davidtwco]

@nikomatsakis @eddyb @arielb1 @pnkfelix (I'm not sure who I should be pinging about this, apologies)

So far I've implemented most of the mentoring instructions but there are a few things I've run into issues with.

In this section where I require .. (as per the first mentoring instruction) - this seems to work, except that it doesn't take into account whether the struct/enum is within the crate (and therefore should be able to match without a wildcard) - this causes issues with the run-pass/rfc-2008-non-exhaustive/struct_within_crate.rs test (and the similar enum test). I'm not sure what I should be checking to do that.

I've also not made changes to the visibilities in encode_enum_variant_info and build_reduced_graph_for_variant as when I did so, I found that I was no longer able to access the variants of a enum in order to match them from outside of a crate - similarly to the previous question, I wasn't sure how to have the change apply only outside of a crate.

I hope this is on the right track and I'd appreciate any feedback.

[arielb1]

@davidtwco

To check whether an ADT is within the same crate as the crate you are type-checking, find the def-id of the ADT and check whether it's local - e.g. if you have an adt: &AdtDef, you can check adt.did.is_local().

[arielb1]

For the visibility, you can ask @petrochenkov but I think you want to set the visibility to pub(crate) aka ty::Visibility::Restricted(DefId::local(CRATE_DEF_INDEX))

[arielb1]

If you can find me, feel free to ping me on IRC.

[davidtwco]

Thanks, that's super helpful.

With regards to the visibility, that's what I was setting the visibility to, but I'll take another stab at it and see what I can come up with.

[nikomatsakis]

@davidtwco glad to see that @arielb1 got you unblocked! Looking forward to seeing the PR. =) Let us know if you hit any other problems (are here or on gitter)

[davidtwco]

@nikomatsakis @arielb1

Apologies for how long this is taking and for a further query.

Since my previous comment here I think I've got it working with structs and after speaking with @nikomatsakis in Gitter earlier today, I've added a bunch more tests for cases that I hadn't considered. There are only five tests that still don't pass:

The first of which is related to instantiating a tuple struct, I think I've went wrong with the visibility somewhere for this so I'm going to look into that. The second of which is similar but for instantiating a unit struct.

The rest of which are all related to a lack of errors for things that should fail on enums with the attribute. @nikomatsakis mentioned that there are two cases with enums, handling new variants and new fields to existing variants - I do believe that the new fields case is handled by the existing structs code but I'm not 100% sure. However, I don't think the new variants case is handled. The function mentioned in the first bullet point of the mentoring instructions doesn't seem to apply for this case and I've not been able to figure out where the equivalent is. As the edited mentoring instructions now mention, there aren't any instructions for handling enums so I've been struggling to pinpoint where I need to make changes. I'd appreciate some pointers as to where I should be working for that.

All the help so far as been fantastic so I appreciate that.

[davidtwco]

@rust-lang/compiler @rust-lang/lang As far as I'm aware, with #51854 having landed, the only thing remaining for this RFC is implementation of the attribute for variants (they currently share a DefId with the enum which meant it was skipped on the initial implementation) - I'm happy to work on this if someone can provide some pointers. Is there anything else blocking stabilization?

[nikomatsakis]

@davidtwco I see a few things listed in the issue header:

• #[non_exhaustive] is not yet implemented for enum variants.
• Current behavior on unions. The current implementation (#44109) sometimes ignores non_exhaustive on unions, sometimes reports an error.
• In addition, the attribute is not currently validated (neither syntactic form nor location), so it can be written in weird forms (#[non_exhaustive(foo)]) and in arbitrary locations, including suspicious ones like traits (#[non_exhaustive] trait Trait {}).

Have the other two been resolved?

[davidtwco]

@nikomatsakis #49345 resolved the latter two issues. The first issue is still pending.

[nikomatsakis]

So, to implement #[non_exhaustive] for enum variants, we might want to extend the existing check that works for structs:

rust/src/librustc_typeck/check/mod.rs

Lines 3575 to 3582 in fefe816

	// Prohibit struct expressions when non exhaustive flag is set.
	if let ty::TyAdt(adt, _) = struct_ty.sty {
	if !adt.did.is_local() && adt.is_non_exhaustive() {
	span_err!(self.tcx.sess, expr.span, E0639,
	"cannot create non-exhaustive {} using struct expression",
	adt.variant_descr());
	}
	}

In that code, we have this variable variant which is a VariantDef.

Currently, we have the is_non_exhaustive flag being set on the AdtDef itself. I think that for structs, this is actually wrong. What I would do is to have two "non-exhaustive" flags (perhaps with distinct names):

• On the AdtDef, we would have can_extend_variant_list
• On each VariantDef, we would have can_extend_field_list.

When we build the adt-def for a struct, if it is tagged with #[non_exhaustive], we would set the can_extend_field_list flag. For an enum, we would set one or both flags, depending on where the #[non_exhaustive] attributes appear.

Once this is done, the existing code is almost correct, but it must be changed from saying adt.is_non_exhaustive to variant.is_non_exhaustive. This should be correct in either case.

The AdtDef for a struct is created by the adt_def provider:

rust/src/librustc_typeck/collect.rs

Lines 558 to 560 in fefe816

	fn adt_def<'a, 'tcx>(tcx: TyCtxt<'a, 'tcx, 'tcx>,
	def_id: DefId)
	-> &'tcx ty::AdtDef {

You can see that structs have one variant, built here:

rust/src/librustc_typeck/collect.rs

Lines 594 to 597 in fefe816

	(AdtKind::Struct, vec![
	convert_struct_variant(tcx, ctor_id.unwrap_or(def_id), item.name,
	ty::VariantDiscr::Relative(0), def)
	])

The existing flag is set in AdtDef::new, invoked from alloc_adt_def. This logic will have to change to take into account the kind of adt we have:

rust/src/librustc/ty/mod.rs

Lines 1912 to 1914 in fefe816

	if tcx.has_attr(did, "non_exhaustive") {
	flags = flags | AdtFlags::IS_NON_EXHAUSTIVE;
	}

Anyway, I'm not -- I confess -- 100% sure where to set the flag best, but that's the idea. =)

[davidtwco]

Submitted #52775 with an implementation of the attribute on variants.

[durka]

In the currently implementation, E0639 fires when the attribute is applied to enums, while it should only be relevant when the attribute is on a specific variant: #53549

[SimonSapin]

@rust-lang/lang Can we stabilize what’s implemented (attribute on enum types to allow adding variants later) without waiting on what’s not (yet)? (attribute on struct types or enum variants to allow adding fields later)

[nikomatsakis]

Seems reasonable to me -- @petrochenkov are you aware of any "gotchas" there?

[petrochenkov]

No, not aware.
My initial concerns were fixed in #49345 and I haven't looked at the feature after that.

[clarfon]

Was the non-exhaustiveness added to rustdoc or is that still a to-do? I'm fine with just stabilising enums and leaving the rest unstable.

[davidtwco]

@clarcharr It has been added to rustdoc.

[npmccallum]

I'm very much in favor of stabilizing this for enums and leaving the rest unstable.

[clarfon]

As far as I'm aware, the only thing that needs to be done is actually separating out the feature flag and an FCP. The FCP could probably be started before that's merged though, as it can be done as part of stabilisation.

[davidtwco]

With #59376 having landed, #[non_exhaustive] is now implemented for enum variants.