extern type cannot support size_of_val and align_of_val

[joshtriplett]

Based on discussion on #43467 and in @rust-lang/lang meetings, an opaque extern type type cannot support size_of_val or align_of_val. We believe that we should panic or abort in this case. We also believe that we should have a compile-time lint that detects this whenever possible, since at some level the compiler should know at compile time if code invokes size_of_val or align_of_val on an extern type.

I've opened this separate issue to document that decision, and will propose FCP on it to give time for final comments.

[joshtriplett]

@rfcbot fcp merge

[rfcbot]

Team member @joshtriplett has proposed to merge this. The next step is review by the rest of the tagged teams:

• @aturon
• @cramertj
• @eddyb
• @joshtriplett
• @nikomatsakis
• @nrc
• @pnkfelix
• @scottmcm
• @withoutboats

No concerns currently listed.

Once a majority of reviewers approve (and none object), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[kennytm]

we should have a compile-time lint that detects this whenever possible

will the lint be emitted before or after monomorphization?

P.S. cc rust-lang/rfcs#2310

[eddyb]

I'm specifically in favor of "before", only emitting anything after monomorphization under some general umbrella of "this code will always panic at runtime" warnings.

[joshtriplett]

It would be nice to handle generic functions that get monomorphized with an extern type, though. That will come up.

[hanna-kruppe]

The RFC text (and the discussion even more so, IIRC) mention the possibility of adding a new trait for encoding this constraint. I assume this has been dropped because that trait would have to be included in bounds by default like Sized?

[Ericson2314]

@rkruppe yes sadly.... rust-lang/rfcs#2310 was opened as backup plan. See the end of #43467.

[comex]

I'm a bit confused why this is in rust-lang/rust instead of rust-lang/rfcs. (I have rust-lang/rfcs set as watched, so I get email copies of all issues and PRs there, but not here.)

[joshtriplett]

Because it's a bug about extern type behavior.

[nikomatsakis]

@comex

I'm a bit confused why this is in rust-lang/rust instead of rust-lang/rfcs. (I have rust-lang/rfcs set as watched, so I get email copies of all issues and PRs there, but not here.)

Because this is not a new RFC, it's nailing down a specific unresolved question around extern types (though we should have linked from the tracking issue to here -- fixed). This is pretty standard practice.

[jethrogb]

Why lint when you can have type safety? Two convincing comments from the other discussion thread:

@mikeyhew #43467 (comment)

OK, I just want to say that I am very strongly of the opinion that Rust should use built-in traits like DynSized to express the difference in capabilities between extern types and the dynamically-sized types that currently exist in Rust (i.e. trait objects and slices). All of the alternatives that I have seen – panicking, returning Option from size_of_val, post-monomorphisation lints – are less powerful, and the issues with ?-traits that people keep bringing up need to be tested and not just speculated about. We need to at least try doing things the builtin-traits way and see what it's like, and see what the ergonomic impact is like, and see if we can reduce it, before settling for something inferior.

Maybe I'm overreacting, I just got the sense from reading some of the comments in this thread that something might be done in order to get extern types out the door, that might put us in a backward-compatibility trap later on. Now that I have more time to work on Rust, I'm planning on writing an eRFC to add DST-specific builtin traits like DynSized and SizeFromMeta, so we can start experimenting with them and Custom DST.

@Ericson2314 #43467 (comment)

1. I don't think "always use the smallest tool for the job" applies here. The decreased power of lints is directly worse for uses. C.f. non-null lints v.s. Option in other languages. Lints are easily lost amid other warnings, and the fact is only some users will care. This means while individual code bases might obey them, the ecosystem as a whole can not be trusted to uphold the invariants the lints try to maintain. This a real loss for fostering an ecosystem of DynSize abstractions, or whatever the niche feature is, as for such niche things, being able to sync up few and scattered programmers and form a community is all the more important.

2. Ecosystem-wide enforcement is also good for the "regular users don't need to care" goal. If some library happens to use truly unsized types, and the consumer is unaware, they could face some nasty unexpected packages. With ?DynSize they do get bothered with a compile time error they didn't expect, but that is much less nasty to deal with with than a run-time bug. If they don't want to learn ?DynSized, they can go use a different library; better to have the opportunity to do that up front than after you're tied to the library too deeply because it took a while to excise the {size,align}_of_val panic.

And my own perspective: there are various places where I'd like to be generic over things that the language doesn't allow. That goes hand in hand with the type system. For example, you can't be generic over lengths when using arrays or calling conventions when using function pointers. Let's not add more features in that could reasonably be described correctly in the type system but aren't.

[joshtriplett]

@jethrogb

Why lint when you can have type safety?

If the lint is deny-by-default (which I think it should be), it'd effectively act the same as a type error. The second comment you quote seems based on the idea that it'd be a warning and not an error.

So the only difference is whether to provide a more general trait-based mechanism or to handle the specific case at hand. I haven't seen any fundamental opposition to the idea of introducing those traits in the future if we have a more general case that needs them, but if we want those traits for some future features that we haven't yet introduced to the language, let's design those traits alongside those future language features.

[kennytm]

Would it be possible to make substituting an extern type in generics and using them in struct fields an error altogether (feature-gate it), until the situation of DynSized reached consensus? extern type could still be used for FFI as originally designed. This would side-step the entire size_of_val problem for stabilizing extern type.

// ok:

fn f1(a: &ExternType) {}
fn f2() -> *mut ExternType {}
fn f3<T: ?Sized>(a: &T, b: *const ExternType) {}
fn f4<A, R, F: Fn(A) -> R>(func: F, a: A) -> R {}
let w1 = size_of::<*const ExternType>();
let w2 = size_of_val(&f2());
let w3 = f4(|x| x, &*f2());
//^ substituting `&ExternType` is ok.

// errors (gated):

fn g1<T: ?Sized>(a: &T) {}
g1(&*f2()); //~ ERROR: Cannot substitute an extern type to `T`

fn g2() -> Box<ExternType> {} //~ ERROR: Cannot substitute an extern type to `T`

struct S1(ExternType);  //~ ERROR: Cannot use an extern type as struct field.

struct S2<R: ?Sized>(R);
let s2: S2<ExternType>; //~ ERROR: Cannot substitute an extern type to `R`

struct S3;
impl Deref for S3 {
    type Target = ExternType; //~ ERROR: Cannot use extern type as associated type.
    fn deref(&self) -> &ExternType {}
}

[Ericson2314]

If the lint is deny-by-default (which I think it should be), it'd effectively act the same as a type error. The second comment you quote seems based on the idea that it'd be a warning and not an error.

In terms of trust, only lints that cannot be turned off allow foreign code to be trusted to the same degree.

The generics thing that @jethrogb points out is good too. Let's assume we have a monomorphization-time lint as you proposed, @joshtriplett (I agree with you that that's better than only having a pre-monomorphization time lint). Then, even thought its just as safe (same things will be prevented in the end), the user experience is still worse. The library could be fine over it's testsuite, but not fine when instantiated downstream. And for upstream, even if the test do catch such bad instantiations, type errors pop up sooner than monomorization-time lints for a quicker debug cycle.

[Ericson2314]

@kennytm I mentioned before in a buried comment that we can stabilize extern type and change the types of {size,align}-of-val to have the ?DynSize bound without stabilizing DynSized. This only makes polymorphic code with a ?DynSize bound require unstable Rust, a very niche use-case but precisely the one hurt by using lints. Problem solved!

In many ways, that is effectively the same as what you propose for stable code, unstable DynSize is just the easiest way to implement it :).

[kennytm]

@Ericson2314 That's a much bigger change because you need to introduce a new concept DynSized, while feature-gating substitution + struct field is much easier as an incremental step.

Furthermore we are not even committed to whether ?Sized should opt-out of DynSized yet!

[Ericson2314]

@kennytm DynSized is not actually that big of a change despite all the controversy because the traits system so accurately models what's going on. It's already been implemented in a PR in the past, for example. Feature-gating substitution + struct field I'd wager would be harder because it's ad-hoc, unless we've happened to do exactly that lint before. [Closest thing I can think of is packed structs, and that's not as close.]

Furthermore we are not even committed to whether ?Sized should opt-out of DynSized yet!

Nothing I propose forces a final decision, since DynSized is still unstable. Here's a case analysis demonstrating.

• Case: DynSize is "positive" trait, {size,align}_of_val has DynSize bound. Not possible no matter what:
  1. breaks existing code instantiating {size,align}_of_val with type variable.
• Case: No DynSize, {size,align}_of_val has existing type. Possible by doing:
  1. Remove all mention of DynSize (OK since unstable).
  2. Add panicking behavior so {size,align}_of_val is defined on extern type.
• Case: DynSize is "positive" trait, {size,align}_of_val has existing type (but is deprecated). Possible by doing:
  1. Same steps as above
  2. Add back DynSize as normal "postive" trait.
  3. Make the {size,align}_of_val replacements using DynSized
  4. (Optional) Deprecate {size,align}_of_val

[glaebhoerl]

How does this relate to #48055? It seems like that is another case where we'd assume for arbitrary T: !Sized that we nonetheless know the size at runtime and can move it around by memcpy - potentially a much bigger 'hole' in the system than size_of_val and align_of_val?

[comex]

Well, not necessarily out of the box, without a way to get an owned T. But &move T could be problematic if ever implemented, and someone might decide that functions like ptr::read should have their Sized bounds removed…

Anyway, definitely seems like a good idea to restrict that feature to DynSized types (good thing it's not even implemented yet).

[briansmith]

If the lint is deny-by-default (which I think it should be), it'd effectively act the same as a type error. The second comment you quote seems based on the idea that it'd be a warning and not an error.

I propose that it be (1) deny-by-default, (2) not allowed to be changed from that default, and (3) formatted to look like a type error instead of a lint message. Then it would be indistinguishable, to the user, from a type error.

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[briansmith]

I still think that when we have the option of enforcing a typing rule in the type checker vs. the lint mechanism, we should enforce it in the type checker. I understand and don't necessarily disagree with the objections to DynSized but there is an alternative: special-case these intrinsics and extern type in the type system. It's not very elegant but the whole point of static type systems is to prevent static typing errors at compile time and not at runtime.

[SimonSapin]

There is precedent for such no-so-elegant special-casing: instanciating std::mem::transmute<T, U> errors if T and U do not have the same size.

[rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

[CAD97]

I'd like to bring up the idea of only allowing extern types behind raw pointers, and not behind references or by value again (no matter whether that's implemented via a permanently unstable bound or whatever special behavior).

I've proposed allowing getting the layout of a raw pointer's pointee type, and while size/align_of_val are stable and cannot have special behavior for extern type (beyond panicking or a dummy value), but the new for_ptr APIs could return an Option to encode the possibility of extern types.

The main issue with "extern type only behind pointer" (ignoring implementation concerns) is that the RFC explicitly mentions using extern types for FAM-like scenarios:

extern {
    type OpaqueTail;
}

#[repr(C)]
struct FfiStruct {
    data: u8,
    more_data: u32,
    tail: OpaqueTail,
}

While suboptimal, the same effect can be achieved by just making FfiStruct an extern type:

extern {
    type FfiStruct;
    type OpaqueTail;
}

impl FfiStruct {
    fn data(*mut self) -> *mut u8 { // though it's a lot less convenient without pointer receivers
        self.cast::<u8>().offset(0).cast::<u8>()
    }
    fn more_data(*mut self) -> *mut u32 {
        self.cast::<u8>().offset(4).cast::<u32>()
    }
}

Alternatively, you could still allow extern type as an unsized tail of a struct, but make that "poison" the struct with whatever poison makes extern type unusable behind a reference or by-value.

It seems to me that this is the most "theoretically pure" solution to the "align_of_val problem". Have some sort of reuqired-by-default gate that only core (and maybe other std subcrates) are allowed to turn off, whether that be DynSized or HasNoExternTrait.

[nikomatsakis]

I'd like to bring up the idea of only allowing extern types behind raw pointers, and not behind references or by value again (no matter whether that's implemented via a permanently unstable bound or whatever special behavior).

I'm not really sure how we could do that without some kind of trait bound, unless the intent is to not permit generics to be bound to extern types. In particular, you could have fn foo<T: ?Sized>(x: T) { let y = &x; ... } today, so we'd have to exclude that T is an extern type somehow.

[CAD97]

In particular, you could have fn foo<T: ?Sized>(x: T) { let y = &x; ... } today

Well, not that exact formulation, as error[E0277]: the size for values of type T cannot be known at compilation time for note: all local variables must have a statically known size/help: unsized locals are gated as an unstable feature.

What is allowed is unsafe fn foo<T: ?Sized>(x: *const T) { &*x; }. If we want to get super language lawyer-y about it, we could say that a reference to an extern type is valid but not safe. But that also seems like a poor kludge just to make mem::size_of_val::<T: ?Sized> not have to lie about extern type.

But yes, while I now agree that while this (no reference to extern type) would be theoretically pure, actually implementing it would be at least as dirty as just letting mem::[size|align]_of_val panic.

[oli-obk]

I know this has been brought up in random places but hasn't been discussed here yet. We could require methods on extern types for size and alignment. That way we leave the actual decision to users:

• extern { type Foo; } or types with that as its unsized field can't be used as a generic param
• extern { type Foo { fn size(self: *const Self) -> usize { panic!("sorry") } } } gives the panicking behaviour (a lint could detect this and report post monomorphization)
• extern { type Foo { fn size(self: *const Self) -> usize { strlen(self as *const u8) } } } could use the knowledge about which types are legal here to compute the size, no matter how complex or trivial that is.

[Ericson2314]

That is better than just "panics + usable in generics", but it feels like an ad-hoc bootleg version of ?DynSized. In what respects is it better than that?

[oli-obk]

In what respects is it better than that?

It doesn't require adding a new opt-out trait and adding bounds for it to various functions. ?DynSized requires the type system to do ~something beyond rejecting some extern types in substs (which is in fact an orthogonal feature, we could just start with the variant where they are always allowed and see if we want the generics rejection later and add it by only allowing the extern { type Foo; } syntax later).

[Ericson2314]

It doesn't require adding a new opt-out trait and adding bounds for it to various functions.

But it does require adding a new ad-hoc monomorphization passes and limmitting the instantiantiation of generics. That strikes me as exactly the same amount of work in rustc as using the existing proven mechanism (opt-out traits).

?DynSized requires the type system to do ~something beyond rejecting some extern types in substs

I don't understand this? What's the actual burden here?

Which is in fact an orthogonal feature, we could just start with the variant where they are always allowed and see if we want the generics rejection later and add it by only allowing the extern { type Foo; } syntax later

So you are saying that syntax is unstable, but the existence of extern types and instantiating them in your own generics is stable? Isn't that just what we have today?

Conversely, we could implement DynSized today, but not generalize any std lib code to use ?DynSized, and also special case it in error messages. What's wrong with that? Even if DynSized was never stable, and so users could never write ?DynSized, it strikes me as the right way to implement the restriction on generics without reinventing the wheel.

---

It really seems like the ideological opposition to opt-in traits is leading everyone down weird paths! I am not sure what to say than these alternatives seems like massive epicycles and code smells to me, and I am kinda shocked few other folks seem to have the same gut reaction --- didn't we all agree panicks are post-monomorphization lints and their ilk are bad?

[oli-obk]

So you are saying that syntax is unstable, but the existence of extern types and instantiating them in your own generics is stable? Isn't that just what we have today?

I'm trying to say that we should stabilize only

extern {
    type Foo {
        fn size(self: *const Self) -> usize {
            strlen(self as *const u8)
        }
    }
}

and figure out if we want the "rejected in generics" form of extern types later. After that change people can use the panicking variant, which isn't too great, but not too bad either. We could even figure out a separate opt out trait for rejecting types to be used in generics by default that is independent of extern type.

[oli-obk]

Conversely, we could implement DynSized today, but not generalize any std lib code to use ?DynSized, and also special case it in error messages. What's wrong with that? Even if DynSized was never stable, and so users could never write ?DynSized, it strikes me as the right way to implement the restriction on generics without reinventing the wheel.

that seems like a reasonable implementation of

We could even figure out a separate opt out trait for rejecting types to be used in generics by default that is independent of extern type.

If we allow people to write their own size computation, we also allow people to panic there. This is nothing new. Only now I am realizing that rejecting the use in generics and my fn size suggestion are entirely orthogonal in fact. My suggestion can also be implemented later. I somehow thought that we'd end up painting ourselves into a corner, not sure why.

So yea, either an opt out trait that users never see or compiler magic to reject them seems fine to me now. The opt out trait seems nicer than other compiler magic, tho due to it needing to be a lang-item I'm not sure it's actually cleaner, we can try both impls.

[RalfJung]

(FWIW the opposition to new opt-out traits is certainly not ideological, it is based on concrete concerns. We all want to make Rust the best language it can be, we just disagree on how to best achieve that. So let's stay constructive. :)

[Ericson2314]

Only now I am realizing that rejecting the use in generics and my fn size suggestion are entirely orthogonal in fact.

I am still trying to understand this. Are you saying with the generics prohibition, then size_of_val and align_of_val aren't possible to call anyways so we don't need panicking? sound good to me?

[Ericson2314]

@oli-obk

So yea, either an opt out trait that users never see or compiler magic to reject them seems fine to me now. The opt out trait seems nicer than other compiler magic, tho due to it needing to be a lang-item I'm not sure it's actually cleaner, we can try both impls.

Also, keep in mind we a very analogous situations to the status quo were if an empty type is the last field, then the outer type must also be prohibited from instantiations.

This and probably other DST logic I am forgetting tip the scales for me. I would like to reuse all that code!

[Ericson2314]

@oli-obk

(FWIW the opposition to new opt-out traits is certainly not ideological, it is based on concrete concerns. We all want to make Rust the best language it can be, we just disagree on how to best achieve that. So let's stay constructive. :)

OK well to lay my cards out on the table, I think this would be a good state of affairs:

1. Use an unstable DynSized implementation, with explicitly no plans to stabilize, as an implementation detail.
2. I can nonetheless make examples of using ?DynSized to do various FFI things. I can to pain a broader picture how thin types should be embraced widely to improve FFI concerns. As a measure of good faith I can write down in the docs "warning, this uses a unstable feature with no plans on stabilizing!!!!"
3. A popularity contest where this sort of stuff gains "mind share" as at least possible.

So I could see (2) being viewed as "backstabbing" the perma unstable decision; I'm certainly willing to admit that. On the other hand, going with ad-hoc implementation strategies rather than traits feels like:

We know this ways is a jankier implementation, but we are so dead-set against their being any more opt-out traits, we really want to avoid creating one that might become popular and force our hand.

I am not trying to stir the pot and allege you all are consciously thinking that, but that's just what it has felt like being on the other end.

The pitch to you all with the compromise path I outlined is basically:

If ?DynSize is such a clearly bad idea, and we document clearly it is unlikely to ever become stable, then what exactly is the risk of the allow the nightly-only ?DynSize popularity contest to go forward?

Allowing ?DynSized on nightly, even with all those disclaimers, would mean a huge deal to me, as it means I can e.g. write a blog post addressing the FFI and ABI concerns that have been brewing as of late, and any crazy ideas will be made far more tangible with people able to play with the stuff in the night rustc playpen and see what the hell I am talking about first-hand.

I would never normally say "please change master of Rustc so I can argue against you all better!" I mean, what sort of hubris is that! But I think this is a sort of special case in that DynSized seems to me like the easiest by far way to implement this feature, wholly separate from it enabling me to spout me stuff :).

[oli-obk]

I like experimenting with ?Dynsized on nightly to show either its use as a forever unstable thing (like Freeze) or potential for useful APIs. Without implementing it we can't really tell

[Ericson2314]

Woo cannot heart that enough!

like Freeze

Oh I didn't know (or long ago forgot) about that. Thrilled to hear this is such precedent for "Compiler-internal traits" too!

[scottmcm]

It's not obvious to me how much a forever-unstable ?DynSized would help, because it's completely reasonable that people want to do their own impl<T: ?????> Foo for *const T that would work with extern types.

[Ericson2314]

@scottmcm Well we are saying if people indeed want to do that, then there will be preassure to stabilize that. And perhaps that's not a bad thing, it coming from real users solving imminent problems, rather than speculation from me and others about how problems in the abstract ought to be addressed.

Still, it's good to think: what are the alternatives?

• If we do the extern type generics ban with out DynSized, then those users are shit out of luck. They can't even use nightly to ascertain whether ?DynSized would help there problem or not. They just see the generics cannot be instantiated and hit a brick wall.

• If we don't do the generics ban, then users on stable can write such code right away, but size_of_val used with generics will be an annoying foot-gun. (If extern type ends up used as much as I hope for better FFI, that footgun will only become more and more commonplace.)

Given those alternatives I think starting with ?DynSized is the best. We can always get rid of DynSized and go with the generics ban or size_of_val panicking later.

[nikomatsakis]

I'm not opposed to have ?DynSized as an unstable implementation trait. That said, I also think that if we see people using it in the wild, the question is not just are they using it but how likely is it that it is helping prevent and catch bugs.

In other words, one bit of feedback I would be interested in is--- how often did you find compilation errors related to this trait helping to clarify your thinking ("ah, of course I can't do it that way, we might not know the size!") etc.

[Ericson2314]

@nikomatsakis Glad to hear that!!

My heterodox hunch is that actually most ?Sized could becomes ?DynSized as moving (or cloning) "scary, ambiguously large" things is relatively uncommon. To me, that means

• On one hand, since people are really trying to do things which require size_of_val, perhaps the avoided bugs won't be too frequent.
• On the other hand, since people rarely do want the size anyways, size_of_val being opt-out is a sort of funny default to begin with.

I am thinking that if ?DynSized does become kosher as I (not so secretely) hope, it might make sense in some future edition to make DynSized opt-in so:

• ?Sized today becomes ?Sized + DynSized eventually.
• ?DynSized (or would it have to be ?Sized + ?DynSized?) tomorrow becomes ?Sized eventually.

:) This is all very roundabout way of saying if today's situation is indeed a bit of an "anti-huffman encoding" of people's desires as I allege, bug empiricism might be sort of a statistically wonky, and even ignoring size_of_val panicking, better means of avoiding accidental DST moves might be valuable in and of itself.

[nbdd0121]

How about just make usage of size_of_val on extern type a post-monomorphization error?

[Ericson2314]

@nbdd0121Post monomorphization errors are generally regarded a terrible developer experience akin to C++ template errors.

@nikomatsakis saying that we can try out ?DynSized on an unstable basis and then say how things go is great news. If there are issues then with more information to the inform the debate I think alternatives like post-monomorphization errors can better be discussed.