Tracking issue for RFC 1861: Extern types

[aturon]

This is a tracking issue for RFC 1861 "Extern types".

Steps:

• Implement the RFC (Implement RFC 1861: Extern types #44295)
• Adjust documentation (see instructions on forge)
• Stabilization PR (see instructions on forge)

Unresolved questions:

• Should we allow generic lifetime and type parameters on extern types?
  If so, how do they effect the type in terms of variance?

• In std's source, it is mentioned that LLVM expects i8* for C's void*.
  We'd need to continue to hack this for the two c_voids in std and libc.
  But perhaps this should be done across-the-board for all extern types?
  Somebody should check what Clang does. Also see "extern type" should use opaque type in LLVM #59095.

[jethrogb]

This is not explicitly mentioned in the RFC, but I'm assuming different instances of extern type are actually different types? Meaning this would be illegal:

extern {
    type A;
    type B;
}

fn convert_ref(r: &A) -> &B { r }

[canndrew]

@jethrogb That's certainly the intention, yes.

[glaebhoerl]

Relatedly, is deciding whether we want to call it extern type or extern struct something that can still be done as part of the stabilization process, or is the extern type syntax effectively final as of having accepted the RFC?

EDIT: rust-lang/rfcs#2071 is also relevant here w.r.t. the connotations of type "aliases". In stable Rust a type declaration is "effect-free" and just a transparent alias for some existing type. Both extern type and type Foo = impl Bar would change this by making it implicitly generate a new module-scoped existential type or type constructor (nominal type) for it to refer to.

[Ericson2314]

Can we get a bullet for the panic vs DynSized debate?

[plietar]

I've started working on this, and I have a working simple initial version (no generics, no DynSized).

I've however noticed a slight usability issue. In FFI code, it's frequent for raw pointers to be initialized to null using std::ptr::null/null_mut. However, the function only accepts sized type arguments, since it would not be able to pick a metadata for the fat pointer.

Despite being unsized, extern types are used through thin pointers, so it should be possible to use std::ptr::null.

It is still possible to cast an integer to an extern type pointer, but this is not as nice as just using the function designed for this. Also this can never be done in a generic context.

extern {
    type foo;
}
fn null_foo() -> *const foo {
    0usize as *const foo
}

Really we'd want is a new trait to distinguish types which use thin pointers. It would be implemented automatically for all sized types and extern types. Then the cast above would succeed whenever the type is bounded by this trait. Eg, the function std::ptr::null becomes :

fn null<T: ?Sized + Thin>() -> *const T {
    0usize as *const T
}

However there's a risk of more and more such traits creeping up, such as DynSized, making it confusing for users. There's also some overlap with the various custom RFCs proposals which allow arbitrary metadata. For instance, instead of Thin, Referent<Meta=()> could be used

[SimonSapin]

I think we can add extern types now and live with str::ptr::null not supporting them for a while until we figure out what to do about Thin/DynSized/Referent<Meta=…> etc.

[plietar]

@SimonSapin yeah, it's definitely a minor concern for now.

I do think this problem of not having a trait bound to express "this type may be unsized be must have a thin pointer" might crop up in other places though.

[SimonSapin]

Oh yeah, I agree we should solve that eventually too. I’m only saying we might not need to solve all of it before we ship any of it.

[plietar]

I've pushed an initial implementation in #44295

[kennytm]

@plietar In #44295 you wrote

Auto traits are not implemented by default, since the contents of extern types is unknown. This means extern types are !Sync, !Send and !Freeze. This seems like the correct behaviour to me. Manual unsafe impl Sync for Foo is still possible.

While it is possible for Sync, Send, UnwindSafe and RefUnwindSafe, doing impl Freeze for Foo is not possible as it is a private trait in libcore. This means it is impossible to convince the compiler that an extern type is cell-free.

Should Freeze be made public (even if #[doc(hidden)])? cc @eddyb #41349.

Or is it possible to declare an extern type is safe-by-default, which opt-out instead of opt-in?

extern {
    #[unsafe_impl_all_auto_traits_by_default]
    type Foo;
}
impl !Send for Foo {}

[eddyb]

@kennytm What's the usecase? The semantics of extern type are more or less that of a hack being used before the RFC, which is struct Opaque(UnsafeCell<()>);, so the lack of Freeze fits.
That prevents rustc from telling LLVM anything different from what C signatures in clang result in.

[kennytm]

@eddyb Use case: Trying to see if it's possible to make CStr a thin DST.

I don't see anything related to a cell in #44295? It is reported to LLVM as an i8 similar to str. And the places where librustc_trans involves the Freeze trait reads the real type, not the LLVM type, so LLVM treating all extern type as i8 should be irrelevant?

[eddyb]

@kennytm So with extern type CStr;, writes through &CStr would be legal, and you don't want that?
The Freeze trait is private because it's used to detect UnsafeCell and not meant to be overriden.

[scottmcm]

We discussed this in the lang team backlog bonanza today.

We'd like to make progress on this one, but were unsure exactly where things stood related to these around the big questions like whether pointers are thin and ptr::null works (did https://rust-lang.github.io/rfcs/2580-ptr-meta.html change anything here?), whether size_of_val would panic, and related.

Is anyone interested here in summarizing the state of things?

[SimonSapin]

I didn’t realize it was a question whether pointers to extern types should be thin. What metadata would there be?

RFC 2580 was accepted saying that null, null_mut, and NonNull dangling should be extended to T: ?Sized + Thin but that’s not implemented yet. The "natural" way to do it would requires a similar language change in integer-to-pointer casts with as, which I imagine would require some logic in the compiler to reason about the thin-ness of pointers to some generic type parameter based on the presence of a Pointee<Metadata = ()> bound, like it does today with (potentially implicit) Sized bounds.

That change in as semantics was not parts of the RFC though, so T-lang should probably discuss it specifically.

[Ericson2314]

I am not sure if this relates to the current iteration of the question, but discussions about custom DSTs in the past had extern types as a building block, with the user able to specify a custom Metatdata used to help define size_of_value and align_of_value.

...like it does today with (potentially implicit) Sized bounds.

Please, let's cross the implicit bound rubicon!

[SimonSapin]

I think custom metadata would be compatible with extending null and friends to T: ?Sized + Thin. (with Thin = Pointee<Metadata = ()>). Those custom DSTs would not be Thin.

As to implicit bounds I’m only talking about the existing ones: T: Sized is assumed in most (all?) generic contexts unless T: ?Sized is specified. I’m not proposing adding any more. As far as I understand this is how as is legal here:

fn null<T>() -> *const T {
    0 as *const T
}

[Ericson2314]

Ah, yours does work today, but

fn null<T: ?Sized>() -> *const T {
    0 as *const T
}

is does not. Great! That means the null, null_mut, and NonNull changes should be just fine, but also orthogonal to this issue, I think?

[SimonSapin]

This is the tracking issue for extern types in general, so whether null and friends work with them is relevant. But yes it’s a sub-topic that can be discussed separately to reduce load on this already long thread. I’ve opened #93959

[SimonSapin]

questions like whether pointers are thin and ptr::null

#94954 implements this part of the pointer metadata RFC

[Skepfyr]

I haven't been involved in this so far but I'm going to attempt to summarize where this is in the hope that it allows some forward progress.

The original RFC kind of relied on the custom DSTs RFC which has been postponed, this means some of the workarounds mentioned may need to be made more permanent for this to be stabilised. Extern types have been implemented in #44295, this also flagged some questions not covered by the RFC. The other relevant piece of work is the pointer metadata RFC (tracking issue).

Some of this is copied from @jethrogb's summary above: #43467 (comment)

Should we allow generic lifetime and type parameters on extern types?

This appears to be entirely resolved by wrapping the extern type in a #[repr(transparent)] wrapper and adding some PhantomDatas as in: #43467 (comment)

That does assume you can put extern types in transparent structs, see later, if not then:

1. Do nothing, I haven't seen any particularly compelling arguments they're necessary.
2. Allow them and just assume everything's invariant.
3. Come up with some way of defining them and their variance (these feels beyond the scope of this feature).

What LLVM type should be used?

Extern types currently use type {} which is LLVM's equivalent of (). However it's desirable for c_void to be an extern type (see rust-lang-nursery/portability-wg#13) but that must currently be an i8. This may all be made simpler by LLVM opaque pointers which will be required by LLVM 16 and is probably advisable for LLVM 15. Also see #59095.

std::ptr::null/null_mut

These functions did not work as T had to be sized, the pointer metadata RFC weakened this to T: ?Sized + Thin, so these now work correctly since #94954 was merged.

size_of_val (and partially align_of_val)

size_of_val / align_of_val can be called with extern types, and respectively return 0 and 1. This differs from the RFC, which specified that they should panic. This is essentially always wrong and not very Rust-y, a previous lang meeting suggested size_of_val should be changed to panic and a lint added to try and spot this at compile time. align_of_val should probably act similarly but has extra difficulties explained below.

Auto traits are not implemented by default.

Manual unsafe impl Sync for Foo is possible for all auto-traits except for Freeze. Types like CStrshould be extern types (or at least Thin DSTs) but don't have interior mutability. However, Freeze is private so it's not observable whether it's implemented and it's only used for optimisations so it's probably fine to ignore.

Trait objects cannot be built from extern types.

&Extern as &dyn Trait doesn't work because Extern is !Sized, however it is Thin so this could be allowed. Unfortunately this hits the unknown size & alignment problem below so may not be possible after all. Interestingly &Foo<Extern> as &Foo<dyn Trait> does work and gets the size & alignment wrong.

type class of the extern type

See #43467 (comment). The C ABI allows for different address spaces (in LLVM terminology). I don't know whether Rust supports this anywhere though.

Unknown alignment of extern types

This is the big one and is hard to summarise, the RFC allows types like:

extern {
    type OpaqueTail;
}

#[repr(C)]
struct FfiStruct {
    data: u8,
    more_data: u32,
    tail: OpaqueTail,
}

Note I'm not clear that the above should be allowed, C rejects it for the reason I'm about to describe.

However the following code causes everything to blow up:

extern {
    type OpaqueTail;
}

#[derive(Debug)]
#[repr(C)]
struct Foo<T: ?Sized> {
    data: u8,
    tail: T,
}

fn foo_tail(foo: &Foo<OpaqueTail>) -> &OpaqueTail {
    &foo.tail // What offset is this field at?
}

fn hidden<T: Debug>(foo: &Foo<T>) {
    println!("{:?}", foo as &Foo<dyn Debug>) // extern type field access hidden behind trait object coercion
}

Currently rustc assumes that all extern types have alignment 1 but this is clearly wrong. There have been a few suggestions for how to fix this:

1. Post-monomorphisation error/lint - rustc will know when it's trying to layout a struct like this so could just throw a compile error, however there are good reasons to dislike post-monomorphisation errors in general and this would prevent the OpaqueTail pattern.
2. Allow #[repr(align(N))] on extern types - would be useful but doesn't solve the problem because there are opaque types where the alignment is unknown.
3. Prevent extern types being supplied as generic parameters, this could prevent legitimate use cases, may prevent blanket trait implementations from being applied, and may prevent you from implementing traits like PartialEq.
4. A DynSized trait this does approximately the same as the above but could allow ?DynSized bounds to selectively relax the restrictions. This has been discussed extensively before Add DynSized trait (rebase of #44469) #46108 (comment), RFC: DynSized without ?DynSized — Lint against use of extern type in size_of_val, and more rfcs#2310, More implicit bounds (?Sized, ?DynSized, ?Move) rfcs#2255 (comment).

There is evidence to suggest something like OpaqueTail is wanted, see this rustc code but obtaining a reference the tail is only safe if you can #[repr(align(N))] the type (or field). Otherwise I feel like you actually want (a thing I just made up):

#[repr(C, unsized)]
struct Header {
    data: u8,
}

Where next?

The conclusion from a previous lang team meeting was that the correct answers to a bunch of the hard questions here is tied to custom DSTs and we should attempt to find the minimum possible thing to stabilise. However the alignment issues makes deciding what that minimum thing is difficult. Especially as custom DSTs may never land.

[simonbuchan]

Nice summary!

Interestingly, if we had #[repr(unsized)] it would seemingly be redundant with extern { type Foo; }, as you could use #[repr(unsized)] struct Foo. That implies that it should be spelled extern { struct Foo { head: u32, } }.

[Ericson2314]

It sounded to me like there was some interest from the lang team on trying out DynSize on a super unstable basis in the last few months.

[madsmtm]

I don't quite understand why giving extern types alignment 1/no alignment is wrong? It may be that the type Foo in your example doesn't report the "correct" alignment, but since the user can never create it manually, I assume that it would always be created by something that does know the required alignment.

Building upon your example, playground link (which doesn't compile).:

#[derive(Default, Debug)]
#[repr(C)]
struct Bar {
    data: u8,
    // 8 bits of padding here
    x: u16,
}

fn main() {
    let bar = Bar::default();
    let foo: &Foo<OpaqueTail> = unsafe { mem::transmute(&bar) };

    // Possible, but would just point to the padding, and effectively useless
    let tail = foo_tail(foo);

    // Not possible, since `OpaqueTail` doesn't implement `Debug`
    hidden(foo);
}

Note that &foo.tail is allowed, but you can't do anything useful with it without first converting it to another type. But maybe there's something I'm missing here?

Related, I opened a proposal over at the Unsafe Code Guidelines to change how extern types works wrt. references: rust-lang/unsafe-code-guidelines#356

[Skepfyr]

Hopefully this playground demonstrates the issue:
https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=b7350128f9f2869c7eda52d1574c78d9

Edit: Alternatively, here's a minor alteration that segfaults!
https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=5b180f64c9c39a555114ca01bcf60d18

[madsmtm]

Yeah, though I don't think this is an issue since this kind of API doesn't exist in the real world? Or am I wrong here? I should think use_foo would usually take *const CHeader, and then the problem would be moot.

My point is, I can't see the reason we should disallow Rust from making references to opaque types, it's just that actually using them, as in your example, is impossible without further knowledge (e.g. maybe it is documented that CFoo has pointer alignment, and then one could include the padding and raise the alignment of Header such that it would work fine).

[Ericson2314]

@madsmtm If the foreign type is the tail of the a struct, we want the field to be at a proper offset.

Also, it's just very bad luck to say "It's OK to lie here because it doesn't matter", thing change including the whatever underpins that rational today. I prefer not to make decision decision that will cause massive headaches for designs (including future me) tomorrow.

[RalfJung]

Yeah, it seems like for soundness we should disallow extern types to be used in non-transparent structs. But of course when generics are involved we can't really do that. So this is yet another way in which extern types cause problems when being used to instantiate generic (even ?Sized) types.

[madsmtm]

@madsmtm If the foreign type is the tail of the a struct, we want the field to be at a proper offset.

Also, it's just very bad luck to say "It's OK to lie here because it doesn't matter", thing change including the whatever underpins that rational today. I prefer not to make decision decision that will cause massive headaches for designs (including future me) tomorrow.

I guess my understanding of extern types were more "opaque blob of bytes of unknown length" rather than "corresponds to a specific C type, we just don't know the size of the type". Under the former, I think it is perfectly rational that the extern type would include any padding, but of course it isn't under the latter.

Note that I don't have any experience with language design, I'll believe you if you say the latter way of viewing extern types is correct!

[codeflo]

The conclusion from a previous lang team meeting was that the correct answers to a bunch of the hard questions here is tied to custom DSTs and we should attempt to find the minimum possible thing to stabilise. However the alignment issues makes deciding what that minimum thing is difficult.

As someone who needs opaque pointers from time to time to interface with C, and who finds the workaround suggested in the Rustonomicon painfully ugly and error-prone, thank you for the summary, @Skepfyr.

I wonder to what extent trying to solve the harder problem of also allowing opaque tails is blocking progress on stabilizing a much simpler first solution that would for now only enable opaque pointers, and which might hopefully already cover 95% of the actual uses.

In particular, I personally would be delighted by a concept of an "thin opaque type" that would try to closely mirror the semantics of an incomplete struct in C and C++ as much as possible. In particular, it would

• enable thin pointers and references
• allow generics
• give a compile-time error after monomorphization of size_of and align_of
  • (If that's really undesired, a panic with a linter hint is probably fine, but I don't see why that would be an improvement.)
• give a compile-time error after monomorphization when used as a member of a non-transparent struct

Such a solution would be perfectly open to future extensions that would enable more programs to compile if certain hints for the type are given.

Post-monomorphisation error/lint - rustc will know when it's trying to layout a struct like this so could just throw a compile error, however there are good reasons to dislike post-monomorphisation errors in general and this would prevent the OpaqueTail pattern.

I agree that post-monomorphization errors feel un-Rusty. Unfortunately, that seems almost unavoidable to be fully compatible with C here, which is what this feature is all about.

[Skepfyr]

I'm going to pretend that the following traits exist and that Sized isn't implied by default to make talking about all this easier:

trait DynAligned { fn align(&self) -> usize; }
trait DynSized: DynAligned { fn size(&self) -> usize; } // This is what ?Sized means.
trait Aligned: DynAligned { const ALIGN: usize; }
trait Sized: Aligned + DynSized { const SIZE: usize; } // Equivalent to the existing Sized trait.

Fundamentally some extern types need to be !DynAligned, this means that they cannot be placed in structs, but also that means that they don't satisfy any existing generic trait bounds (implicit or explicit) so can't be used in generics.

I'm not entirely convinced that what I said above is true, it's possible that all extern types have a alignment known at compile time, and #[repr(align(n))] solves half the above. Even then the types would be Aligned but not DynSized which means they don't quite satisfy ?Sized.

We could avoid the above by not having traits and just making any issues be post-monomorphisation errors. There are some of those already but I cannot see that being acceptable, what's the point of traits if we just ignore them sometimes? On the other hand, the compiler does understand size and alignment unlike most traits, and we could make everything "just work" with post-monomorphisation errors (it would mean a bunch of special casing in the compiler though).

I think that sorting all this out may require another RFC that supplants this one, given the kinds of questions that we're asking.

[Skepfyr]

I've opened rust-lang/rfcs#3396 in a bid to fix this properly.