Tracking issue for RFC 2645, "Transparent Unions" (formerly: and Enums)

[Centril]

This is a tracking issue for the RFC "Transparent Unions and Enums" (rust-lang/rfcs#2645).

Steps:

• Implement the RFC (cc @rust-lang/compiler -- can anyone write up mentoring instructions?)
• Adjust documentation (see instructions on forge)
• Stabilization PR (see instructions on forge) [done for enums]

Unresolved questions:

The role of #[repr(transparent)] in nonnull-style optimizations is not entirely clear. Specifically, it is unclear whether the user can rely on these optimizations to be performed when they make a type transparent. Transparent unions somewhat complicate the matter. General concensus seems to be that the compiler is free to decide where and when to perform nonnull-style optimizations on unions (regardless of whether or not the union is transaprent), and no guarantees are made to the user about when and if those optimizations will be applied. It is still an open question exactly what guarantees (if any) Rust makes about transparent structs (and enums) and nonnull-style optimizations.

This RFC doesn't propose any changes to transparent structs, and so does not strictly depend on this question being resolved. But since this RFC is attempting to round out the #[repr(transparent)] feature, it seems reasonable to dedicate some time to attempting to round out the guarantees about #[repr(transparent)] on structs.

Also it is not clear if transparent unions can even be implemented on LLVM without seriously restricting our semantics for unions overall.

[mjbshaw]

I plan on attempting to implement the feature by adapting my previous implementation. I'd appreciate it if someone could take a look at that commit and let me know if it's horribly wrong :)

[Centril]

@pnkfelix Up for reviewing ^--- ?

[Centril]

Implementation of both feature gates was done in #60463 which landed on 2019-06-11.
The PR was written by @mjbshaw and reviewed by @varkor and @rkruppe.

[Centril]

Should we apply #[repr(transparent)] to MaybeUninit now?

[mjbshaw]

I'll submit a PR to update MaybeUninit.

[gnzlbg]

I ran into this today:

#[repr(C)] struct S(u8, u32);

#[repr(transparent)]
union U { _x: S, y: () }

unsafe extern "C" fn foo(u: U) {
    let pad = (&u as *const _ as *const u8).add(1).read();
    assert_eq!(pad, 13); // Should fail when called below
}

pub unsafe fn bar() {
  let mut u: U = U { y: () };
  (&mut u as *mut _ as *mut u8).add(1).write(42);
  foo(u);
}

Because of (), all bit-patterns of union U are valid. The particular bit-pattern being constructed in bar is [undef, 42, undef, undef, undef, undef, undef, undef]. I expected the assert in foo to fail when called from bar: that bit-pattern is a valid bit-pattern for U, foo reads its first byte, and the test always returns false (42 != 13).

However, due to repr(transparent), this is not correct. repr(transparent) gives U the same ABI as S. That is, U will be passed by value "as if" it were an S. The byte of S at offset 1 is padding, and it does not need to be passed when calling foo.

For example, the ABI could state that S.0 is passed in a 1-byte register (e.g. al), and that S.1 is passed in a 32-bit register (e.g. ecx). This is not exactly, but close to what happens when that snippet is compiled for, e.g., i686-pc-windows-msvc (godbolt). The above is compiled in debug mode to:

core::ptr::write:
        ;; writes 42 to the address at ecx
        mov     byte ptr [ecx], 42
        ret
core::ptr::<impl *mut T>::add:
        ;; increments the address at ecx by 1
        lea     eax, [ecx + 1]
        ret
example::bar:
        sub     esp, 24
        ;; our union is at esp + 8, create ptr:
        lea     ecx, [esp + 8]
        ;; increment its address by 1
        call    core::ptr::<impl *mut T>::add
        mov     ecx, eax
        ;; writes 42 to the address (esp + 9)
        call    core::ptr::<impl *mut T>::write
        ;; now passing the union to foo begins
        ;; S.0 is put in `al` - padding be damned
        mov     al, byte ptr [esp + 8]
        ;; S.1 is put in `ecx`
        mov     ecx, dword ptr [esp + 12]
        ;; then these are moved back to the stack (esp + 16) 
        mov     byte ptr [esp + 16], al
        ;; the padding at esp +17/18/19 is uninitialized
        mov     dword ptr [esp + 20], ecx
        ;; and then moved back and forth a couple of times
        movsd   xmm0, qword ptr [esp + 16]
        movsd   qword ptr [esp], xmm0
        ;; and then we call foo, which never sees 42
        call    example::foo
        add     esp, 24
        ret

If this happens, then the bytes at offset 1,2, and 3 of the union U will not be passed to foo, and will become uninitialized. This would allow the compiler to optimize the assert_eq! inside foo away. This is exactly what happens if we crank-up the opt-level to 3 (godbolt):

example::bar:
        ret

If the union U has the same call ABI as S, then the bytes 1, 2, and 3 cannot be part of the "value representation" (cc @ubsan) of the union - they do not need to be memcpy'ed on move, etc. If these bytes need to be part of the "value representation" of the union, then the union cannot have the same call ABI as S - instead, the union would need the same ABI as, e.g., #[repr(C)] struct(i32,i32) so that those bytes get passed by value on copy/move/etc.

AFAICT, these two constraints are incompatible.

cc @eddyb @rust-lang/wg-unsafe-code-guidelines

[RalfJung]

Ouch. Nice catch!

The reason we implemented a "transparent" repr for unions (re-using the ABI of the only non-ZST field if possible) was because of code that used MybeUninit<SimdType> getting a lot slower compared to using mem::uninitialized, where there was no "wrapper". We had to switch MaybeUninit to use the SIMD ABI in that case to get the old performance back. At least that was our interpretation back then, I remember we did more than one thing to fix the perf problems, and I don't remember if we confirmed which one was "it". Here's the relevant PR.

So, a way to keep that while also achieving that "unions preserve all bytes when copied around" would be to only use the ABI of the only non-ZST field if that ABI is an "Integer" or "vector" one, where we know all bytes are preserved.

That would not really be repr(transparent) any more, that flag would have to be removed.

[mjbshaw]

@gnzlbg I think that's UB, with or without repr(transparent). With repr(transparent). Without repr(transparent). The end results are exactly the same (interestingly, repr(C) is different).

I'm not aware of any guarantees or requirements that a wrapper type (whether it's a union or a struct) preserves padding bytes, or that reading padding bytes is valid/safe. If Rust makes any guarantees about that I'd love to know where I can read about them.

[RalfJung]

I meant stricter in the sense that less code is legal under those rules.

Oh I see.

Yes: at rust-lang/unsafe-code-guidelines#73 some people propose that with a union like

#[repr(C)]
union U {
  f1: (bool, u8),
  f2: bool,
}

it should be UB to have the first byte of the union be anything but 0x00 or 0x01. Basically "if some byte of the union is subject to a restriction for all fields, the restriction also applies to the union as a whole". That would allow enum optimizations for unions. (When accessing a field that is already UB because then the field's type's rule apply, but we are talking about handling the union wholesale.)

But I think defining this precisely is too complicated and we should rather provide people with the tools to define their own "niches" for enum optimizations.

[Centril]

We discussed this briefly on the language team meeting today. Attendance was low. However, those present agreed that it would be OK to move towards stabilizing repr(transparent) on univariant enums specifically, while leaving the hint on unions unstable for the time being. I will write up a short report & PR.

[Centril]

Stabilization report & PR for enums is up at #68122.

[RalfJung]

This is now just tracking transparent unions.

I think I just realized there is a problem with transparent unions if we want to provide "bag of bytes" semantics for unions:

#[repr(transparent)]
union U { f: u32 }

As a union, U should be just a bag of bytes. If I make one of the bytes poison, then after copying the union around, that should properly preserve which byte is poison and which one is not.

However, to my knowledge, LLVM actually makes an i32 either fully poison or not poison at all -- so the moment such a partially poisoned U gets loaded as an i32 in LLVM, the remaining bytes would lose their content and reset to poison as well. Ouch.

[mjbshaw]

I have two questions:

1. Why is that not a problem if you omit #[repr(transparent)]?
2. Why is this so bad? You just poisoned f; I'm not sure what else you'd expect to happen, with or without #[repr(transparent)].

I'm still not sold on the "bag of bits" idea. I've tried to put an alternative forward but instead of receiving a good rebuttal for why my alternative is inferior it instead go sidetracked by "let's decide what padding bits are" when I think we're pretty much already in agreement on what "padding bits" means. Is this something that a chat on Zulip or something could solve? I think some real-time vocal communication could resolve this in minutes, whereas asynchronous text comms will take days (or weeks) (and GitHub is a suboptimal forum due to it hiding replies and whatnot).

[hanna-kruppe]

Why is that not a problem if you omit #[repr(transparent)]?

Since repr(Rust) unions have no ABI commitments, we can just represent them as a literal byte array. That allows each byte to be poison-or-not independently.

[mjbshaw]

@hanna-kruppe That's true, but I'm not sure it's useful to allow such granularity for poison because you still can't read f. You can only memcpy the non-poison bytes of f, which I'm not sure Rust should guarantee you can do because it's such a big footgun (and I can't think of any real value it provides).

[hanna-kruppe]

That goes more into your second question (why/how this observation matters), which I deliberately didn't go into because it's a more complex topic and my time is currently very limited.

[RalfJung]

Why is this so bad? You just poisoned f; I'm not sure what else you'd expect to happen, with or without #[repr(transparent)].

#[repr(transparent)]
union U { f: u32 }

let mut u = U { f: 0 };
(&mut u as *mut _ as *mut MaybeUninit<u8>).add(1).write(MaybeUninit::uninit());

let u2 = u;
let v = (&u2 as *const _ as *const u8).read();
println!("{}", v);

The Rust semantics as they exist in my head and as it is drafted here would guarantee that this program prints 0. This is what "unions are just bags of bytes" means.

But it turns out LLVM actually says "nope this is UB as all of u2 is poisoned". So, the desired semantics is not implementable with LLVM while upholding repr(transparent) guarantees.

Of course we could try to adjust our semantics, but (a) that will make the semantics of unions significantly more complicated, and (b) it seems like a shame that LLVM would force us to cripple our semantics like that, for no good reason. If LLVM's type system was not quite so restrictive, we could just tell LLVM to load 4 bytes at once and preserve which byte is poison and which is not. But LLVM conflates "bytes" with "integers that have arithmetic operations", and then a lot of sadness ensues.

[mjbshaw]

Thanks for the demo, @RalfJung. I personally think this should have the same behavior as #[repr(transparent)] struct S { f: u32 }. I'll take some time to read through the draft you linked to, though.

[RalfJung]

@mjbshaw do you still think that if we replace U by MaybeUninit<u32>? Since MaybeUninit is repr(transparent), that's basically the same type.

[alercah]

Doesn't writing MaybeUninit::uninit() still write undef and not poison? Unless I'm quite mistaken, in this playground, it's undef and not poison that gets passed to write_to_float. And since undef is tracked bitwise, this all works out doesn't it?

Of course, if LLVM does change uninitialized memory to poison, then we would run into this problem, without bitwise semantics.

[RalfJung]

Indeed, this was written assuming LLVM would switch to poison eventually.

Cc #94428

[alercah]

Is the concern in this comment about poison spreading to entire union fields not equally valid for #[repr(C)] unions that must be passed as integers per platform ABIs?

[RalfJung]

Yeah sure, that's the same situation.

Judging from this discussion, it seems like LLVM will get a "freezing load" operation. Whenever we have a load that would allow partially uninit data, we could use the freezing load and be sure that the data is preserved correctly. This does lose some information, but at least it would resolve the concern about spreading poison to neighboring bytes.

[clarfonthey]

So, poking around the various open issues and PRs and this looks like the best place to mention this.

I mentioned in #101179 that I think that allowing DSTs in MaybeUninit might be a way to improve the API for slices (by allowing MaybeUninit<[T]>), but it appears that DSTs are still not allowed in unions, even #[repr(transparent)] ones like MaybeUninit. The closest this was to being implemented was #47650 which got closed because the author didn't have time to finish it.

Was there ever an explicit reason to disallow this, or was it just not implemented/considered since no one had discussed it much?

[RalfJung]

The problem is custom DST. If CStr becomes a DST, then size_of_val might have to actually read the data behind the reference to determine the length of the value -- but clearly with a MaybeUninit<CStr> that would be bad.

[clarfonthey]

Ah, right -- we haven't fully eliminated the possibility of thin DSTs. I guess that we're still unsure what a proper custom DST RFC would look like, although my guess is that CStr will be a thorn in the side for most of them. C strings really are the worst. (I say this sarcastically, although I'm not surprised to find another reason why they're problematic.)

My gut feeling is to say that any RFC which would permit thin DSTs (which, as demonstrated by CStr, would be something we want) is that we'd also want a mechanism to filter them out, precisely for cases like this. But I guess that this isn't a strong enough argument to be able to simply allow them now and deal with the consequences later.