Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue for RFC 1861: Extern types #43467

Open
aturon opened this issue Jul 25, 2017 · 200 comments
Open

Tracking issue for RFC 1861: Extern types #43467

aturon opened this issue Jul 25, 2017 · 200 comments

Comments

@aturon
Copy link
Member

@aturon aturon commented Jul 25, 2017

This is a tracking issue for RFC 1861 "Extern types".

Steps:

Unresolved questions:

  • Should we allow generic lifetime and type parameters on extern types?
    If so, how do they effect the type in terms of variance?

  • In std's source, it is mentioned that LLVM expects i8* for C's void*.
    We'd need to continue to hack this for the two c_voids in std and libc.
    But perhaps this should be done across-the-board for all extern types?
    Somebody should check what Clang does. Also see #59095.

@aturon aturon changed the title Tracking issue for RFC 1861: Extern tyupes Tracking issue for RFC 1861: Extern types Jul 25, 2017
@jethrogb
Copy link
Contributor

@jethrogb jethrogb commented Jul 25, 2017

This is not explicitly mentioned in the RFC, but I'm assuming different instances of extern type are actually different types? Meaning this would be illegal:

extern {
    type A;
    type B;
}

fn convert_ref(r: &A) -> &B { r }
@canndrew
Copy link
Contributor

@canndrew canndrew commented Jul 25, 2017

@jethrogb That's certainly the intention, yes.

@glaebhoerl
Copy link
Contributor

@glaebhoerl glaebhoerl commented Jul 25, 2017

Relatedly, is deciding whether we want to call it extern type or extern struct something that can still be done as part of the stabilization process, or is the extern type syntax effectively final as of having accepted the RFC?

EDIT: rust-lang/rfcs#2071 is also relevant here w.r.t. the connotations of type "aliases". In stable Rust a type declaration is "effect-free" and just a transparent alias for some existing type. Both extern type and type Foo = impl Bar would change this by making it implicitly generate a new module-scoped existential type or type constructor (nominal type) for it to refer to.

@Ericson2314
Copy link
Contributor

@Ericson2314 Ericson2314 commented Jul 25, 2017

Can we get a bullet for the panic vs DynSized debate?

@plietar
Copy link
Contributor

@plietar plietar commented Aug 10, 2017

I've started working on this, and I have a working simple initial version (no generics, no DynSized).

I've however noticed a slight usability issue. In FFI code, it's frequent for raw pointers to be initialized to null using std::ptr::null/null_mut. However, the function only accepts sized type arguments, since it would not be able to pick a metadata for the fat pointer.

Despite being unsized, extern types are used through thin pointers, so it should be possible to use std::ptr::null.

It is still possible to cast an integer to an extern type pointer, but this is not as nice as just using the function designed for this. Also this can never be done in a generic context.

extern {
    type foo;
}
fn null_foo() -> *const foo {
    0usize as *const foo
}

Really we'd want is a new trait to distinguish types which use thin pointers. It would be implemented automatically for all sized types and extern types. Then the cast above would succeed whenever the type is bounded by this trait. Eg, the function std::ptr::null becomes :

fn null<T: ?Sized + Thin>() -> *const T {
    0usize as *const T
}

However there's a risk of more and more such traits creeping up, such as DynSized, making it confusing for users. There's also some overlap with the various custom RFCs proposals which allow arbitrary metadata. For instance, instead of Thin, Referent<Meta=()> could be used

@SimonSapin
Copy link
Contributor

@SimonSapin SimonSapin commented Aug 10, 2017

I think we can add extern types now and live with str::ptr::null not supporting them for a while until we figure out what to do about Thin/DynSized/Referent<Meta=…> etc.

@plietar
Copy link
Contributor

@plietar plietar commented Aug 10, 2017

@SimonSapin yeah, it's definitely a minor concern for now.

I do think this problem of not having a trait bound to express "this type may be unsized be must have a thin pointer" might crop up in other places though.

@SimonSapin
Copy link
Contributor

@SimonSapin SimonSapin commented Aug 11, 2017

Oh yeah, I agree we should solve that eventually too. I’m only saying we might not need to solve all of it before we ship any of it.

@plietar
Copy link
Contributor

@plietar plietar commented Sep 3, 2017

I've pushed an initial implementation in #44295

bors added a commit that referenced this issue Sep 4, 2017
Implement RFC 1861: Extern types

A few notes :

- Type parameters are not supported. This was an unresolved question from the RFC. It is not clear how useful this feature is, and how variance should be treated. This can be added in a future PR.

- `size_of_val` / `align_of_val` can be called with extern types, and respectively return 0 and 1. This differs from the RFC, which specified that they should panic, but after discussion with @eddyb on IRC this seems like a better solution.
If/when a `DynSized` trait is added, this will be disallowed statically.

- Auto traits are not implemented by default, since the contents of extern types is unknown. This means extern types are `!Sync`, `!Send` and `!Freeze`. This seems like the correct behaviour to me.
Manual `unsafe impl Sync for Foo` is still possible.

- This PR allows extern type to be used as the tail of a struct, as described by the RFC :
```rust
extern {
    type OpaqueTail;
}

#[repr(C)]
struct FfiStruct {
    data: u8,
    more_data: u32,
    tail: OpaqueTail,
}
```

However this is undesirable, as the alignment of `tail` is unknown (the current PR assumes an alignment of 1). Unfortunately we can't prevent it in the general case as the tail could be a type parameter :
```rust
#[repr(C)]
struct FfiStruct<T: ?Sized> {
    data: u8,
    more_data: u32,
    tail: T,
}
```

Adding a `DynSized` trait would solve this as well, by requiring tail fields to be bound by it.

- Despite being unsized, pointers to extern types are thin and can be casted from/to integers. However it is not possible to write a `null<T>() -> *const T` function which works with extern types, as I've explained here : #43467 (comment)

- Trait objects cannot be built from extern types. I intend to support it eventually, although how this interacts with `DynSized`/`size_of_val` is still unclear.

- The definition of `c_void` is unmodified
bors added a commit that referenced this issue Sep 10, 2017
Implement RFC 1861: Extern types

A few notes :

- Type parameters are not supported. This was an unresolved question from the RFC. It is not clear how useful this feature is, and how variance should be treated. This can be added in a future PR.

- `size_of_val` / `align_of_val` can be called with extern types, and respectively return 0 and 1. This differs from the RFC, which specified that they should panic, but after discussion with @eddyb on IRC this seems like a better solution.
If/when a `DynSized` trait is added, this will be disallowed statically.

- Auto traits are not implemented by default, since the contents of extern types is unknown. This means extern types are `!Sync`, `!Send` and `!Freeze`. This seems like the correct behaviour to me.
Manual `unsafe impl Sync for Foo` is still possible.

- This PR allows extern type to be used as the tail of a struct, as described by the RFC :
```rust
extern {
    type OpaqueTail;
}

#[repr(C)]
struct FfiStruct {
    data: u8,
    more_data: u32,
    tail: OpaqueTail,
}
```

However this is undesirable, as the alignment of `tail` is unknown (the current PR assumes an alignment of 1). Unfortunately we can't prevent it in the general case as the tail could be a type parameter :
```rust
#[repr(C)]
struct FfiStruct<T: ?Sized> {
    data: u8,
    more_data: u32,
    tail: T,
}
```

Adding a `DynSized` trait would solve this as well, by requiring tail fields to be bound by it.

- Despite being unsized, pointers to extern types are thin and can be casted from/to integers. However it is not possible to write a `null<T>() -> *const T` function which works with extern types, as I've explained here : #43467 (comment)

- Trait objects cannot be built from extern types. I intend to support it eventually, although how this interacts with `DynSized`/`size_of_val` is still unclear.

- The definition of `c_void` is unmodified
bors added a commit that referenced this issue Sep 13, 2017
Implement RFC 1861: Extern types

A few notes :

- Type parameters are not supported. This was an unresolved question from the RFC. It is not clear how useful this feature is, and how variance should be treated. This can be added in a future PR.

- `size_of_val` / `align_of_val` can be called with extern types, and respectively return 0 and 1. This differs from the RFC, which specified that they should panic, but after discussion with @eddyb on IRC this seems like a better solution.
If/when a `DynSized` trait is added, this will be disallowed statically.

- Auto traits are not implemented by default, since the contents of extern types is unknown. This means extern types are `!Sync`, `!Send` and `!Freeze`. This seems like the correct behaviour to me.
Manual `unsafe impl Sync for Foo` is still possible.

- This PR allows extern type to be used as the tail of a struct, as described by the RFC :
```rust
extern {
    type OpaqueTail;
}

#[repr(C)]
struct FfiStruct {
    data: u8,
    more_data: u32,
    tail: OpaqueTail,
}
```

However this is undesirable, as the alignment of `tail` is unknown (the current PR assumes an alignment of 1). Unfortunately we can't prevent it in the general case as the tail could be a type parameter :
```rust
#[repr(C)]
struct FfiStruct<T: ?Sized> {
    data: u8,
    more_data: u32,
    tail: T,
}
```

Adding a `DynSized` trait would solve this as well, by requiring tail fields to be bound by it.

- Despite being unsized, pointers to extern types are thin and can be casted from/to integers. However it is not possible to write a `null<T>() -> *const T` function which works with extern types, as I've explained here : #43467 (comment)

- Trait objects cannot be built from extern types. I intend to support it eventually, although how this interacts with `DynSized`/`size_of_val` is still unclear.

- The definition of `c_void` is unmodified
bors added a commit that referenced this issue Oct 1, 2017
Implement RFC 1861: Extern types

A few notes :

- Type parameters are not supported. This was an unresolved question from the RFC. It is not clear how useful this feature is, and how variance should be treated. This can be added in a future PR.

- `size_of_val` / `align_of_val` can be called with extern types, and respectively return 0 and 1. This differs from the RFC, which specified that they should panic, but after discussion with @eddyb on IRC this seems like a better solution.
If/when a `DynSized` trait is added, this will be disallowed statically.

- Auto traits are not implemented by default, since the contents of extern types is unknown. This means extern types are `!Sync`, `!Send` and `!Freeze`. This seems like the correct behaviour to me.
Manual `unsafe impl Sync for Foo` is still possible.

- This PR allows extern type to be used as the tail of a struct, as described by the RFC :
```rust
extern {
    type OpaqueTail;
}

#[repr(C)]
struct FfiStruct {
    data: u8,
    more_data: u32,
    tail: OpaqueTail,
}
```

However this is undesirable, as the alignment of `tail` is unknown (the current PR assumes an alignment of 1). Unfortunately we can't prevent it in the general case as the tail could be a type parameter :
```rust
#[repr(C)]
struct FfiStruct<T: ?Sized> {
    data: u8,
    more_data: u32,
    tail: T,
}
```

Adding a `DynSized` trait would solve this as well, by requiring tail fields to be bound by it.

- Despite being unsized, pointers to extern types are thin and can be casted from/to integers. However it is not possible to write a `null<T>() -> *const T` function which works with extern types, as I've explained here : #43467 (comment)

- Trait objects cannot be built from extern types. I intend to support it eventually, although how this interacts with `DynSized`/`size_of_val` is still unclear.

- The definition of `c_void` is unmodified
bors added a commit that referenced this issue Oct 28, 2017
Implement RFC 1861: Extern types

A few notes :

- Type parameters are not supported. This was an unresolved question from the RFC. It is not clear how useful this feature is, and how variance should be treated. This can be added in a future PR.

- `size_of_val` / `align_of_val` can be called with extern types, and respectively return 0 and 1. This differs from the RFC, which specified that they should panic, but after discussion with @eddyb on IRC this seems like a better solution.
If/when a `DynSized` trait is added, this will be disallowed statically.

- Auto traits are not implemented by default, since the contents of extern types is unknown. This means extern types are `!Sync`, `!Send` and `!Freeze`. This seems like the correct behaviour to me.
Manual `unsafe impl Sync for Foo` is still possible.

- This PR allows extern type to be used as the tail of a struct, as described by the RFC :
```rust
extern {
    type OpaqueTail;
}

#[repr(C)]
struct FfiStruct {
    data: u8,
    more_data: u32,
    tail: OpaqueTail,
}
```

However this is undesirable, as the alignment of `tail` is unknown (the current PR assumes an alignment of 1). Unfortunately we can't prevent it in the general case as the tail could be a type parameter :
```rust
#[repr(C)]
struct FfiStruct<T: ?Sized> {
    data: u8,
    more_data: u32,
    tail: T,
}
```

Adding a `DynSized` trait would solve this as well, by requiring tail fields to be bound by it.

- Despite being unsized, pointers to extern types are thin and can be casted from/to integers. However it is not possible to write a `null<T>() -> *const T` function which works with extern types, as I've explained here : #43467 (comment)

- Trait objects cannot be built from extern types. I intend to support it eventually, although how this interacts with `DynSized`/`size_of_val` is still unclear.

- The definition of `c_void` is unmodified
@kennytm
Copy link
Member

@kennytm kennytm commented Nov 16, 2017

@plietar In #44295 you wrote

Auto traits are not implemented by default, since the contents of extern types is unknown. This means extern types are !Sync, !Send and !Freeze. This seems like the correct behaviour to me. Manual unsafe impl Sync for Foo is still possible.

While it is possible for Sync, Send, UnwindSafe and RefUnwindSafe, doing impl Freeze for Foo is not possible as it is a private trait in libcore. This means it is impossible to convince the compiler that an extern type is cell-free.

Should Freeze be made public (even if #[doc(hidden)])? cc @eddyb #41349.

Or is it possible to declare an extern type is safe-by-default, which opt-out instead of opt-in?

extern {
    #[unsafe_impl_all_auto_traits_by_default]
    type Foo;
}
impl !Send for Foo {}
@eddyb
Copy link
Member

@eddyb eddyb commented Nov 16, 2017

@kennytm What's the usecase? The semantics of extern type are more or less that of a hack being used before the RFC, which is struct Opaque(UnsafeCell<()>);, so the lack of Freeze fits.
That prevents rustc from telling LLVM anything different from what C signatures in clang result in.

@kennytm
Copy link
Member

@kennytm kennytm commented Nov 16, 2017

@eddyb Use case: Trying to see if it's possible to make CStr a thin DST.

I don't see anything related to a cell in #44295? It is reported to LLVM as an i8 similar to str. And the places where librustc_trans involves the Freeze trait reads the real type, not the LLVM type, so LLVM treating all extern type as i8 should be irrelevant?

@eddyb
Copy link
Member

@eddyb eddyb commented Nov 16, 2017

@kennytm So with extern type CStr;, writes through &CStr would be legal, and you don't want that?
The Freeze trait is private because it's used to detect UnsafeCell and not meant to be overriden.

@mjbshaw
Copy link
Contributor

@mjbshaw mjbshaw commented Jul 7, 2019

Are the "Unresolved questions" in the OP still blocking this from stabilization? Are there any other unresolved questions or issues that are also blocking this?

@Kixunil
Copy link
Contributor

@Kixunil Kixunil commented Apr 12, 2020

If this doesn't support generics, I'd expect people to hack it like this:

extern {
type Helper;
}

pub struct MyExternType<'a, T> {
 _phantom: core::marker::PhantomData<&'a T>,
_helper: Helper,
}

Since the suggested syntax doesn't support it directly, I think that the idea with marker is superior. Maybe worth revisiting.

@eddyb
Copy link
Member

@eddyb eddyb commented Apr 12, 2020

@Kixunil Btw you probably want &'a mut &'a T (or *mut &'a T), to be invariant over those parameters, unless you really want variance (it's wrong if you use &MyExternType<'x, Y> for any mutation that involves types like &'x Y, although I'm not sure what all the conditions are - better make it invariant if you're not sure either).

But yeah, that's a good way to describe some FFI APIs, although without rust-lang/rfcs#2770, it's more painful than it has to be.

@Kixunil
Copy link
Contributor

@Kixunil Kixunil commented Apr 12, 2020

@eddyb oh, I was unclear, I didn't mean &'a T specifically, I meant "this is the syntax one would use to implement any variance needed using PhantomData".

@eddyb
Copy link
Member

@eddyb eddyb commented Apr 12, 2020

One advantage extern { type Foo<'a, T>; } syntax would have, is that it could only be invariant.
So it'd be "correct by default".

@Kixunil
Copy link
Contributor

@Kixunil Kixunil commented Apr 12, 2020

Would it be possible to override it if one knows what he's doing though?

@eddyb
Copy link
Member

@eddyb eddyb commented Apr 12, 2020

No, you'd need to do it like in your example if you wanted something weaker.

bors bot added a commit to rust-analyzer/ungrammar that referenced this issue Aug 24, 2020
Merge #10
10: Allow type aliases in extern blocks r=jonas-schievink a=jonas-schievink

This is for the unstable feature rust-lang/rust#43467, which rustc uses internally

Co-authored-by: Jonas Schievink <jonasschievink@gmail.com>
bors bot added a commit to rust-analyzer/rust-analyzer that referenced this issue Aug 24, 2020
Merge #5861
5861: Support extern types r=matklad a=jonas-schievink

This is a currently unstable feature tracked at rust-lang/rust#43467

Co-authored-by: Jonas Schievink <jonasschievink@gmail.com>
@skade
Copy link
Contributor

@skade skade commented Dec 17, 2020

I want to add a note to this discussion: the current implementation seems to be !Unpin. I think this is correct, but haven't seen that mentioned explicitly.

@SimonSapin
Copy link
Contributor

@SimonSapin SimonSapin commented Jan 18, 2021

Update on null and null_mut for extern types: #42847 (comment)

@Ericson2314
Copy link
Contributor

@Ericson2314 Ericson2314 commented Jan 18, 2021

@SimonSapin in #42847 (comment) rightly points out that Sized now should imply other things, and so no longer is a root of the trait hierarchy. I hope this can help dispel the resistance to DynSized.

resisting math is futile, accept Sized: Thin + DynSized.

@SimonSapin
Copy link
Contributor

@SimonSapin SimonSapin commented Jan 18, 2021

You’re putting words in my mouth. That the trait resolver should be able to deduce Metadata == () from T: Sized is not an argument for adding a new opt-out ?Trait, nor does it alleviate the practical concerns with those proposals.

@Ericson2314
Copy link
Contributor

@Ericson2314 Ericson2314 commented Jan 19, 2021

rust-lang/rfcs#2984 (comment) what I wrote that I don't want to twist your words, I just see that chipping away at the argument against effectively.

@chorman0773
Copy link

@chorman0773 chorman0773 commented Jan 20, 2021

I would like to raise an issue about this.
extern type does not currently have a way to specify the type class of the extern type. This does not pose an issue in rust, but may affect the ability to use this with a C abi.

In ISO 9899 N1124, §6.2.5 Clause 26 defines the classes of pointer types which are required to be compatible. Notably, pointers in the following classes MUST have the same representation and alignment-requirements with others in the same class, and no further requirements are imposed:

  • Pointers to void and pointers to character types (char, unsigned char, and signed char. i8 and u8 I believe are guaranteed to be in this class)
  • Pointers to any structure types.
  • Pointers to any union types (yes this is distinct from the first category)
  • Pointers to any otherwise compatible types (IE. const/volatile qualified, signed/unsigned integer types)

This proposal seems to only allow one, presumably unified ABI (if not, there is no indication of which of the listed classes extern types are guaranteed to be included in). This is not generally compatible with the C abi. Unless rust further restricts compatible C platforms, there needs to be a way to select at least between the classes of struct and union. In a discussion on the Rust Programming Language Community [Discord] Server, a conversion made mention that the class of character types would also be a good idea (though this can be suplemented by a newtype arround u8 or i8, or a repr(u8)/repr(i8) enum). This would solve the c_void problem.

@mjbshaw
Copy link
Contributor

@mjbshaw mjbshaw commented Jan 20, 2021

@chorman0773 Are there any existing ABIs that differentiate between pointer types? Strictly speaking C doesn't have or define an ABI, only implementations of C's "abstract machine" have an ABI (i.e. OS and ISA). If there are ABIs that really do differentiate between pointer types then I suppose something like extern { struct Foo; union Bar; } would probably be sufficient.

@Tamschi
Copy link

@Tamschi Tamschi commented Jan 20, 2021

@mjbshaw An option for void/character-like extern types would also be highly appreciated, at least on my part.

It's not strictly necessary to correctly interface with C due to *mut c_void being available, but it still makes for a fairly large ergonomics improvement when FFI-binding APIs that take/provide *void with effectively constrained lifetime. (e.g. an OOP API that holds user data pointers in destructible instances)

These parameters/return values could then be written as extern type references (which creates appropriate speed bumps when using the API), instead of adding the lifetime constraints to the /// documentation and hoping that everyone reads them thoroughly all the time.

(I'm mostly talking about myself here, though, since I'm a bit prone to these types of mistakes if I don't add the constraints from the get-go.)

@chorman0773
Copy link

@chorman0773 chorman0773 commented Jan 20, 2021

Are there any existing ABIs that differentiate between pointer types? Strictly speaking C doesn't have or define an ABI, only implementations of C's "abstract machine" have an ABI (i.e. OS and ISA). If there are ABIs that really do differentiate between pointer types then I suppose something like extern { struct Foo; union Bar; } would probably be sufficient.

I am not aware of any, though I am only familiar with the System V x86_64 abi. I also raised a similar issue in https://github.com/rust-lang/unsafe-coding-guidelines/issues/266, and one of the comments pointed out control-flow integrety schemes. I don't know that this is mandated in any ABI, though as it mentions, clang supports it with a software implementation (and it seems one of the santizers does check indirect function calls, though I don't see anything for direct calls to a free function).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet