Tracking issue for RFC 1861: Extern types

[aturon]

This is a tracking issue for RFC 1861 "Extern types".

Steps:

• Implement the RFC (#44295)
• Adjust documentation (see instructions on forge)
• Stabilization PR (see instructions on forge)

Unresolved questions:

• Should we allow generic lifetime and type parameters on extern types?
  If so, how do they effect the type in terms of variance?

• In std's source, it is mentioned that LLVM expects i8* for C's void*.
  We'd need to continue to hack this for the two c_voids in std and libc.
  But perhaps this should be done across-the-board for all extern types?
  Somebody should check what Clang does. Also see #59095.

[jethrogb]

This is not explicitly mentioned in the RFC, but I'm assuming different instances of extern type are actually different types? Meaning this would be illegal:

extern {
    type A;
    type B;
}

fn convert_ref(r: &A) -> &B { r }

[canndrew]

@jethrogb That's certainly the intention, yes.

[glaebhoerl]

Relatedly, is deciding whether we want to call it extern type or extern struct something that can still be done as part of the stabilization process, or is the extern type syntax effectively final as of having accepted the RFC?

EDIT: rust-lang/rfcs#2071 is also relevant here w.r.t. the connotations of type "aliases". In stable Rust a type declaration is "effect-free" and just a transparent alias for some existing type. Both extern type and type Foo = impl Bar would change this by making it implicitly generate a new module-scoped existential type or type constructor (nominal type) for it to refer to.

[Ericson2314]

Can we get a bullet for the panic vs DynSized debate?

[plietar]

I've started working on this, and I have a working simple initial version (no generics, no DynSized).

I've however noticed a slight usability issue. In FFI code, it's frequent for raw pointers to be initialized to null using std::ptr::null/null_mut. However, the function only accepts sized type arguments, since it would not be able to pick a metadata for the fat pointer.

Despite being unsized, extern types are used through thin pointers, so it should be possible to use std::ptr::null.

It is still possible to cast an integer to an extern type pointer, but this is not as nice as just using the function designed for this. Also this can never be done in a generic context.

extern {
    type foo;
}
fn null_foo() -> *const foo {
    0usize as *const foo
}

Really we'd want is a new trait to distinguish types which use thin pointers. It would be implemented automatically for all sized types and extern types. Then the cast above would succeed whenever the type is bounded by this trait. Eg, the function std::ptr::null becomes :

fn null<T: ?Sized + Thin>() -> *const T {
    0usize as *const T
}

However there's a risk of more and more such traits creeping up, such as DynSized, making it confusing for users. There's also some overlap with the various custom RFCs proposals which allow arbitrary metadata. For instance, instead of Thin, Referent<Meta=()> could be used

[SimonSapin]

I think we can add extern types now and live with str::ptr::null not supporting them for a while until we figure out what to do about Thin/DynSized/Referent<Meta=…> etc.

[plietar]

@SimonSapin yeah, it's definitely a minor concern for now.

I do think this problem of not having a trait bound to express "this type may be unsized be must have a thin pointer" might crop up in other places though.

[SimonSapin]

Oh yeah, I agree we should solve that eventually too. I’m only saying we might not need to solve all of it before we ship any of it.

[plietar]

I've pushed an initial implementation in #44295

[kennytm]

@plietar In #44295 you wrote

Auto traits are not implemented by default, since the contents of extern types is unknown. This means extern types are !Sync, !Send and !Freeze. This seems like the correct behaviour to me. Manual unsafe impl Sync for Foo is still possible.

While it is possible for Sync, Send, UnwindSafe and RefUnwindSafe, doing impl Freeze for Foo is not possible as it is a private trait in libcore. This means it is impossible to convince the compiler that an extern type is cell-free.

Should Freeze be made public (even if #[doc(hidden)])? cc @eddyb #41349.

Or is it possible to declare an extern type is safe-by-default, which opt-out instead of opt-in?

extern {
    #[unsafe_impl_all_auto_traits_by_default]
    type Foo;
}
impl !Send for Foo {}

[eddyb]

@kennytm What's the usecase? The semantics of extern type are more or less that of a hack being used before the RFC, which is struct Opaque(UnsafeCell<()>);, so the lack of Freeze fits.
That prevents rustc from telling LLVM anything different from what C signatures in clang result in.

[kennytm]

@eddyb Use case: Trying to see if it's possible to make CStr a thin DST.

I don't see anything related to a cell in #44295? It is reported to LLVM as an i8 similar to str. And the places where librustc_trans involves the Freeze trait reads the real type, not the LLVM type, so LLVM treating all extern type as i8 should be irrelevant?

[eddyb]

@kennytm So with extern type CStr;, writes through &CStr would be legal, and you don't want that?
The Freeze trait is private because it's used to detect UnsafeCell and not meant to be overriden.

[shepmaster]

Comments from jethrogb and Ericson2314 mention that LLVM uses a special type opaque instead of Rust's current empty type, but I can't find any follow up on it. Rust still uses an empty type:

Rust

1.33.0-nightly 2019-01-04 f381a96

extern {
    type nuffin_t;

    fn myalloc() -> *mut nuffin_t;
    fn myfree(_: *mut nuffin_t);
}

%"::nuffin_t" = type {}

declare %"::nuffin_t"* @myalloc() unnamed_addr #0
declare void @myfree(%"::nuffin_t"*) unnamed_addr #0

C

clang version 8.0.0 (trunk 350447)

struct nuffin_t;

struct nuffin_t *myalloc();
void myfree(struct nuffin_t*);

%struct.nuffin_t = type opaque

declare dso_local void @myfree(%struct.nuffin_t*) #1
declare dso_local %struct.nuffin_t* @myalloc(...) #1

[Ericson2314]

Good catch!

[eddyb]

@shepmaster I independently came across that (well, @RalfJung pointed it out) and I opened #58271 last week, which I'm not sure how to fix.

If what I did is different from type opaque, that might be why it fails.

But also, note that type {} is just a tuple with no fields, i.e. LLVM's (), not an "empty type" in the Rust sense of enum Void {}.

[raphaelcohn]

I've been trying to use extern type to model some variably-sized data that needs to co-operate with a C API.

I've come across an issue where I'd like to define a method which involves casting from various raw pointers, come of which are Sized and some of which are extern type. In trying to make the code common, I need to use T: ?Sized, but that then creates a problem, as it 'pulls in' all types, including fat pointers, which I don't want (the underlying C code assumes that all pointers are usize in size).

Is there any way to use the type system to differentiate extern type from just ?Sized (similar problems have bit me in the past - it would be nice to use the type system to restrict based on sizes or alignment patterns or repr(u16) or things being enums with specific layouts - but ho-hum).

[alexreg]

@raphaelcohn I believe the plan is to add a DynSized, so you could use T: DynSized to include basically all types except extern types. See rust-lang/rfcs#2255 for more info. I'm not the most knowledgeable about this though.

[jethrogb]

@alexreg yeah so that's the opposite of what @raphaelcohn wants I think :)

[SimonSapin]

rust-lang/rfcs#2580 is another proposal in somewhat the same space as DynSized. With that implemented, you could use T: ?Sized + Pointee<Metadata=()> to restrict a type parameter to types that have thin pointers pointing to them, including extern types.

[alexreg]

@jethrogb It's exactly what he wants, the way I read it. Or maybe he needs both. Regardless, negative bounds for auto traits exist, so e.g. !DynSized would work.

[kennytm]

@alexreg You could write impl !Send for Stuff {} but that doesn't mean where Stuff: !Send is a valid bound.

[raphaelcohn]

@alexreg, @jethrogb @SimonSapin @kennytm Thank you all.
Essentially, I need a bound that says T: thin pointers. I think @SimonSapin's solution T: ?Sized + Pointee<Metadata=()> would work, but it does seem rather non-intuitive.

[SimonSapin]

Indeed, the RFC also proposes a trait alias pub trait Thin = Pointee<Metadata=()>; in the std::ptr module. But at this point it’s only a proposal.

[alexreg]

@kennytm It's not? Well, it needs to be!

[Ericson2314]

For the record, I think we need Pointee and DynSized. Thing + ?Sized still means you are responsible for coming up with the dynamic size and allocation from a (), which means panicking, which is no good. Likewise ?DynSized without Thin means you can do unsafe operations, but have no means to ensure the ABI is correct, so generic FFI stuff cannot be written. Both is wonderful, and will making custom DSTs so much easier to discuss because the unnecessary degrees of freedom don't exist.

[raphaelcohn]

And anything that lets me safely coerce a slice to an iovec would be most welcome...