other-reprs: Null-pointer-optimized enums are FFI safe unless repr(C)

[geofft]

See also the improper_ctypes lint, specifically is_repr_nullable_ptr in
src/librustc_lint/types.rs.

[geofft]

Actually the more I read the code, it looks like the improper_ctypes lint considers it FFI-safe to have an enum with #[repr(C)] or #[repr(uN)], as long as all variants contain FFI-safe data. Presumably the layout is effectively struct SomeEnum {uintN_t discriminant; union { ... } data;}; or so.

But the nomicon is claiming that "tagged unions ... are never FFI safe." Should this text be fixed, or is "tagged union" intended to mean something other than an enum with data?

[ubsan]

They aren't yet FFI-safe, but they should probably be made so.

[steveklabnik]

Enums are tagged unions, so this feels a bit weird, I would phrase it as something like "adds the tag back in" or something, dunno.

[Gankro]

This is way too much information to squish into a single-sentence aside.

To be clear, OptionLike<u32> isn't FFI-safe, but OptionLike<&T> absolutely must be (same repr as *const T). It's possible OptionLike<Box<T>> could be too, but it's unclear to me whether FFI'ing boxes is a very good idea.

[ubsan]

@Gankro I have used it in the past. It's useful for passing ownership to and from C (you return a Box<YourType> from rust, then take &YourTypes in all the functions, then have a dropper function which takes Box<YourType>. From C, it all looks like YourType [const]*, where YourType is an opaque type in C)

[geofft]

They aren't yet FFI-safe, but they should probably be made so.

@ubsan What needs to be done to make this happen?

In particular, if they're not yet FFI-safe, then it seems to me that the improper_ctypes lint should be fixed to align with the current state of things.

To be clear, OptionLike<u32> isn't FFI-safe, but OptionLike<&T> absolutely must be (same repr as *const T). It's possible OptionLike<Box<T>> could be too, but it's unclear to me whether FFI'ing boxes is a very good idea.

FWIW the improper_ctypes lint currently warms about both Option<i32> and Option<Box<i32>>, but not about Option<&i32>.

I've pushed some slightly longer text that phrases this as Option being a specific special case. ("subject to the null pointer optimization" and "is defined like Option, including repr(rust)" are synonyms, right?) I agree that making Box FFI-safe is useful, but it seems reasonable not to promise that yet.

[ubsan]

@geofft basically, define the layout of

#[repr(INT_TYPE)]
enum Foo {
  T0(...),
  T1(...),
   ...
}

to be

struct Foo {
  enum class Tag: INT_TYPE {
    T0,
    T1,
    ...
  } tag;
  union {
    struct { ... } T0;
    struct { ... } T1;
    ...
  };
};

or something of that order

[geofft]

I suspect that's what its representation is right now - what needs to be done process-wise to formalize this? Would this be an RFC or something?

[ubsan]

@geofft not sure. Probably an RFC, but also I'd like to note that the representation of the specific union members isn't specified, so you'd have to figure that out as well.

[steveklabnik]

Thanks!