Tracking issue for RFC 1861: Extern types

[aturon]

This is a tracking issue for RFC 1861 "Extern types".

Steps:

• Implement the RFC (#44295)
• Adjust documentation (see instructions on forge)
• Stabilization PR (see instructions on forge)

Unresolved questions:

• Should we allow generic lifetime and type parameters on extern types?
  If so, how do they effect the type in terms of variance?

• In std's source, it is mentioned that LLVM expects i8* for C's void*.
  We'd need to continue to hack this for the two c_voids in std and libc.
  But perhaps this should be done across-the-board for all extern types?
  Somebody should check what Clang does.

[jethrogb]

This is not explicitly mentioned in the RFC, but I'm assuming different instances of extern type are actually different types? Meaning this would be illegal:

extern {
    type A;
    type B;
}

fn convert_ref(r: &A) -> &B { r }

[canndrew]

@jethrogb That's certainly the intention, yes.

[glaebhoerl]

Relatedly, is deciding whether we want to call it extern type or extern struct something that can still be done as part of the stabilization process, or is the extern type syntax effectively final as of having accepted the RFC?

EDIT: rust-lang/rfcs#2071 is also relevant here w.r.t. the connotations of type "aliases". In stable Rust a type declaration is "effect-free" and just a transparent alias for some existing type. Both extern type and type Foo = impl Bar would change this by making it implicitly generate a new module-scoped existential type or type constructor (nominal type) for it to refer to.

[Ericson2314]

Can we get a bullet for the panic vs DynSized debate?

[plietar]

I've started working on this, and I have a working simple initial version (no generics, no DynSized).

I've however noticed a slight usability issue. In FFI code, it's frequent for raw pointers to be initialized to null using std::ptr::null/null_mut. However, the function only accepts sized type arguments, since it would not be able to pick a metadata for the fat pointer.

Despite being unsized, extern types are used through thin pointers, so it should be possible to use std::ptr::null.

It is still possible to cast an integer to an extern type pointer, but this is not as nice as just using the function designed for this. Also this can never be done in a generic context.

extern {
    type foo;
}
fn null_foo() -> *const foo {
    0usize as *const foo
}

Really we'd want is a new trait to distinguish types which use thin pointers. It would be implemented automatically for all sized types and extern types. Then the cast above would succeed whenever the type is bounded by this trait. Eg, the function std::ptr::null becomes :

fn null<T: ?Sized + Thin>() -> *const T {
    0usize as *const T
}

However there's a risk of more and more such traits creeping up, such as DynSized, making it confusing for users. There's also some overlap with the various custom RFCs proposals which allow arbitrary metadata. For instance, instead of Thin, Referent<Meta=()> could be used

[SimonSapin]

I think we can add extern types now and live with str::ptr::null not supporting them for a while until we figure out what to do about Thin/DynSized/Referent<Meta=…> etc.

[plietar]

@SimonSapin yeah, it's definitely a minor concern for now.

I do think this problem of not having a trait bound to express "this type may be unsized be must have a thin pointer" might crop up in other places though.

[SimonSapin]

Oh yeah, I agree we should solve that eventually too. I’m only saying we might not need to solve all of it before we ship any of it.

[plietar]

I've pushed an initial implementation in #44295

[kennytm]

@plietar In #44295 you wrote

Auto traits are not implemented by default, since the contents of extern types is unknown. This means extern types are !Sync, !Send and !Freeze. This seems like the correct behaviour to me. Manual unsafe impl Sync for Foo is still possible.

While it is possible for Sync, Send, UnwindSafe and RefUnwindSafe, doing impl Freeze for Foo is not possible as it is a private trait in libcore. This means it is impossible to convince the compiler that an extern type is cell-free.

Should Freeze be made public (even if #[doc(hidden)])? cc @eddyb #41349.

Or is it possible to declare an extern type is safe-by-default, which opt-out instead of opt-in?

extern {
    #[unsafe_impl_all_auto_traits_by_default]
    type Foo;
}
impl !Send for Foo {}

[eddyb]

@kennytm What's the usecase? The semantics of extern type are more or less that of a hack being used before the RFC, which is struct Opaque(UnsafeCell<()>);, so the lack of Freeze fits.
That prevents rustc from telling LLVM anything different from what C signatures in clang result in.

[kennytm]

@eddyb Use case: Trying to see if it's possible to make CStr a thin DST.

I don't see anything related to a cell in #44295? It is reported to LLVM as an i8 similar to str. And the places where librustc_trans involves the Freeze trait reads the real type, not the LLVM type, so LLVM treating all extern type as i8 should be irrelevant?

[eddyb]

@kennytm So with extern type CStr;, writes through &CStr would be legal, and you don't want that?
The Freeze trait is private because it's used to detect UnsafeCell and not meant to be overriden.

[kennytm]

@Ericson2314 but you could use a union for "different structs sharing a prefix"

[eddyb]

Note that in #43467 (comment) I linked to

rust/src/librustc/ty/mod.rs

Lines 594 to 609 in f686885

	extern {
	/// A dummy type used to force Slice to by unsized without requiring fat pointers
	type OpaqueSliceContents;
	}
	/// A wrapper for slices with the additional invariant
	/// that the slice is interned and no other slice with
	/// the same contents can exist in the same context.
	/// This means we can use pointer for both
	/// equality comparisons and hashing.
	#[repr(C)]
	pub struct Slice<T> {
	len: usize,
	data: [T; 0],
	opaque: OpaqueSliceContents,
	}

- the current version of that is

rust/src/librustc/ty/mod.rs

Lines 599 to 615 in a3f0f51

	extern {
	/// A dummy type used to force List to by unsized without requiring fat pointers
	type OpaqueListContents;
	}
	/// A wrapper for slices with the additional invariant
	/// that the slice is interned and no other slice with
	/// the same contents can exist in the same context.
	/// This means we can use pointer for both
	/// equality comparisons and hashing.
	/// Note: `Slice` was already taken by the `Ty`.
	#[repr(C)]
	pub struct List<T> {
	len: usize,
	data: [T; 0],
	opaque: OpaqueListContents,
	}

I really like this pattern, and I don't know what to replace it with, if we decide that extern type fields were not allowed at offsets greater than 0.

[rkruppe]

Long-term the Right Way to encode that is ThinSlice with custom DSTs. But who knows when we'll get an accepted RFC for those.

[RalfJung]

@eddyb so you are suggesting miri should just assum extern types to have size 0 (i.e., to use layout.size even though they are unsized)? Will whoever later changes this stuff remember to also to it for miri?^^

My biggest concern here is not how to implement something in miri, but how to keep it in sync later.

[eddyb]

@RalfJung The size doesn't matter for fields' offset, the alignment does.
IMO we could panic in size_of_val while having align_of_val return the alignment, which maybe we could also allow controlling with #[repr(align(N))] on the extern type.

There's also the question of defaulting to byte alignment or always requiring #[repr(align(N))].

[RalfJung]

You are right. So you are suggesting that miri assumes extern types to have the alignment given in their layout, but unknoweable size?

That does seem reasonable, but IMO it should be consistent with what LLVM codegen does so I'll hold off on implementing anything here.

[eddyb]

Note that for fixing #55541, you only need to assume an alignment, you can leave the size unknown.

[RalfJung]

Following up on that comment above, @eddyb made another remark at #55672 (comment):

You can't access that alignment without first having a reference, so having it too high would be a problem (the assumption might not be guaranteed by what you're FFI-ing to), but having it too low wouldn't break anything (other than field offsets).

IOW, the proposal is to make align_of_val work for extern type (and have it return the alignment annotated at the type, or 1 per default), but make size_of_val panic. That is enough to make field access never panic. It is also justified because without a size, we can never allocate such a type, and hence we only ever "consume" the alignment: When we get pointers to such types, we rely on them being sufficiently aligned, but we never create pointers to such type, so the fact that our alignment might be too low is not a problem.

Does that seems like a proposal worth considering?

[Ericson2314]

@RalfJung Yes but

But we never create pointers to such type

But to be pendantic, in computing the field offset we are creating a pointer. If we were given the layout of the struct up front I'd say we would indeed be "rely[ing] on them being sufficiently aligned", but we are computing it ourselves from the information about the types of the fields.

As such, I'd say we should ask for an explicit #[repr(align(N))], as @eddyb floated. I want to make as few presumptions as possible in the no-information case. But if we put in core a Void for C, or OpaqueData, or any other "off the shelf helper", we should give it #[repr(align(1))].

[RalfJung]

in computing the field offset we are creating a pointer

Fair enough. However, we do so only by computing an offset on an already existing pointer.

But yes, in a case like (i32, ExternType), we might indeed be computing the wrong offset for the 2nd field, I have not considered that. In Rust itself that cannot be a problem, because you cannot do much with that pointer, but if you then feed it back to the other side of the extern interface things will go wrong.

Seems like there is no good way to support your "opaque slice" type after all, @eddyb. :/ Though TBH it seems more like a hack anyway.

A mandatory alignment annotation would be an option, but I suspect many people will be rather surprised by that requirement.

[Ericson2314]

Seems like there is no good way to support your "opaque slice" type after all, @eddyb. :/ Though TBH it seems more like a hack anyway.

In @eddyb's case the opaque tail doesn't correspond to any type on the foreign side that can be pointed to directory, so the field offset wouldn't be done anyways, so it still works.

A mandatory alignment annotation would be an option, but I suspect many people will be rather surprised by that requirement.

Eh, this just means that size_of_val and align_of can panic. And if unstable DynSized ends up happening as a practical way to prevent the instantiation of generics with extern types, then it's nice and symmetrical and #[repr(align(N))] could be the stable way to write an impl of some unstable Align trait.

[briansmith]

#[repr(align(N))] on the extern type

I don't object to that idea. However, note that the lack of #[repr(align(N))] doesn't mean, necessarily, 1; it more likely means "I don't know" or "the alignment isn't part of the ABI" or more generally "the alignment requirement is a function of more than just the type of the object".

[RalfJung]

Eh, this just means that size_of_val and align_of can panic.

size_of_val sure, but align_of?

it more likely means "I don't know" or "the alignment isn't part of the ABI" or more generally "the alignment requirement is a function of more than just the type of the object".

So, as long as we make sure that struct fields have a manually specified alignment we are good? We could check that when structs are defined and on generic type parameters. That is, allow extern type without repr(align), but do not allow such "underspecified" extern type to ever occur in a struct. That means align_of_val can panic, but we know it will not when used for field access.

(We could also allow to write down the struct but then not allow to access the field, but that seems more confusing.)

I recall that there once was the proposal to not allow extern type as generic type parameters, but I do not remember why that was not pursued. Generally there have been tons of designs considered here, is there a place documenting why which desig was discarded? The RFC does not mention any of this.

[kennytm]

I recall that there once was the proposal to not allow extern type as generic type parameters, but I do not remember why that was not pursued.

@RalfJung #49708 (comment)

[Ericson2314]

That is, allow extern type without repr(align), but do not allow such "underspecified" extern type to ever occur in a struct.

Yes

size_of_val sure, but align_of?

Yes. Struct alignment is static so it uses align_of not align_of_value. Without the restrictions on generic instantiation with extern type, it is still possible to end up with a alignment-less extern type in a struct field, so the only sane thing to is panic (or post monomorphism error).

Panics and post-monormorphism errors suck, so using more traits like DynSized and Aligned to statically prevent these bad things by construction, while not stabilizing them to avoid constraining our future selves, is very nice and expedient.