Layout of repr(C) unions has padding

[gnzlbg]

In this thread and the comments that follow, the following new information was discovered.

repr(C) unions have padding bits of the form:

• trailing padding: if the union size is larger than the size of its largest field, all union_size - largest_field_size trailing bits are padding bits.
• interior padding: if the bit i of all union fields is a padding bit, the bit i of the inion is a padding bit

Note: because all repr(C) union fields are at offset 0, there is no padding before any field.

The content of padding bits of repr(C) unions is always uninitialized. That is, they are not required to be preserved on copy / move / pass by value, etc. The implementation of the call ABI can exploit this knowledge.

For example, Rust, clang, and GCC all implement the SysV64 ABI, and when passing a #[repr(C)] union U { x: (u8, u32) } around by value, @eddyb mentioned that Rust and GCC pass the bottom 32-bits (where the u8 is stored) while clang passes the bottom 8-bits. Both implementations are allowed. @comex also mentioned that in some ABIs like RISC-V ELF "appears to require callers to zero- or sign-extend
arguments in registers in a particular way. In other words, it requires the upper bits (which correspond to padding bytes) to have a specific value, and the callee can assume that they do have that value". That would be incompatible with allowing users to use the padding bits.

That is, repr(C) unions are not and cannot be just "bags of bits" where one could write to any bit, and that bit value would need to be preserved on copy / move / pass-by-value.

We should document this for repr(C) unions in the Unsafe Code Guidelines, so I'm re-opening this issue until that is resolved.

[BurntSushi]

I think you meant to ping @comex here. :-) (Not comes.)

[RalfJung]

That is, repr(C) unions are not and cannot be just "bags of bits" where one could write to any bit, and that bit value would need to be preserved on copy / move / pass-by-value.

I do not understand why this is a consequence. Your summary (and thanks for the writeup!) says that "Rust and GCC pass the bottom 32-bits". So it is possible to implement this in a way that preserves all bits. Correct?

Note: I am not concerned about bits getting preserved when Rust calls a C function and then C passes data back to Rust. At that point, C rules apply. But I am concerned about the case where Rust code calls an extern "C" function that is also implemented in Rust.

[gnzlbg]

I do not understand why this is a consequence. Your summary (and thanks for the writeup!) says that "Rust and GCC pass the bottom 32-bits". So it is possible to implement this in a way that preserves all bits. Correct?

So that's a good question. Consider this:

#[repr(C)] union U { x: (u8, u32 ) }
extern "C" {
      fn foo() -> U;
}

If the foo implementation is C code compiled with a C compiler, then the padding bytes of the returned U will probably be undef because the C compiler does not need to put anything initialized in there.

OTOH, if we were to somehow make Rust to always copy padding bytes for repr(C) unions, and enforce that in extern "C" fn foo() -> U { ... } function definitions, then those padding bytes could be initialized to something meaningful.

We can't tell whether the code at the other side is Rust or C, so if we want to be able to treat a repr(C) union as a bag of bytes here, we would always need to freeze the union when doing FFI.

I tend to think that an extern "C" declaration should be assumed to follow C rules (e.g. no panics), and that most people use them for interfacing with C, and that therefore we should aim to optimize according to those rules. In this particular case, it would mean that we don't have to copy padding when dealing with these unions, however minor this optimization is for this example, or even in general.

[RalfJung]

My reading of rust-lang/rust#60405 (comment) is that it is impossible to achieve "raw bit copy" for extern "C" with a repr(C) union, but I might be misinterpreting that.

[gnzlbg]

@comex mentions there that Rust->Rust could be extended to support this.

[comex]

I said that it would be possible for some ABIs but not all.

[eddyb]

@RalfJung If you have 8 bytes or more of padding you might not have where to put them.

When only some bytes of an u64 group (passed as one register) are padding, you can pass them alongside the non-padding bytes, but if they're all padding, they won't get a register at all.

[comes]

I think you meant to ping @comex here. :-) (Not comes.)

enjoy what you're doing guys and have a super great time while that.

comes

[RalfJung]

In MiniRust I am now handling this by making unions "chunked": a union is not simply a bag of bytes as large as its size, there are multiple chunks with that size with possible gaps between them. The data within the chunks is preserved verbatim, data between the chunks is lost as padding.

Actually computing the chunks is left to the frontend. I hope we can make it so that repr(Rust) unions only ever have a single chunk that matches their full size, but this also lets us represent C unions that have "gaps" in them that are not being preserved. (We could have also encoded those as structs of unions. But that would probably just be unnecessarily confusing.)

[alercah]

Apologies for the poorly organized long post.

I think @comex misread the RISC-V calling convention spec (which appears to have moved) slightly, but I don't think it really changes much:

Aggregates whose total size is no more than XLEN bits are passed in a register, with the fields laid out as though they were passed in memory. If no register is available, the aggregate is passed on the stack. Aggregates whose total size is no more than 2×XLEN bits are passed in a pair of registers; if only one register is available, the first XLEN bits are passed in a register and the remaining bits are passed on the stack. If no registers are available, the aggregate is passed on the stack. Bits unused due to padding, and bits past the end of an aggregate whose size in bits is not divisible by XLEN, are undefined.

So while I think it means there's no obligation to sign-extend when an integer followed by padding occupies an entire register, it is also extremely clear that padding is not preserved.

The ARM ABI, on the other hand, appears to me to require aggregates to simply be copied, which might imply preservation of padding, but description of the way that arguments are obtained actually clarifies this (section 8.2):

For C, each argument is formed from the value specified in the source code, except that an array is passed by passing the address of its first element.

So this means that the value is copied, meaning padding can take an unspecified value. Unlike what Apple claims, the caller does seem to be responsible for sign- or zero-extension in the latest platform ABI, at least for fundamental integral types. But specifically not for aggregate types. So I think the same logic applies and while the caller is not obligated to pass any particular padding values, it is not obligated to preserve them either.

But if we look at the C standard, it's actually even worse, I think. See section 6.2.6.1
(apologies if the numbers don't match up to the latest official version of the standard):

7. When a value is stored in a member of an object of union type, the bytes of the object
   representation that do not correspond to that member but do correspond to other members
   take unspecified values.

8. ... Where a value is stored in an object using a type that has more than one object representation for
   that value, it is unspecified which representation is used, but a trap representation shall
   not be generated.

and section 6.2.7.1:

14. ... The value of at most one of the members can be stored in a union object at any time.

Thus we arrive at the classic "you can't use a union to cast" problem: the actual value space of a C union, in the most pedantic and hostile interpretation for the programmer, is a true sum type, and copying a union copies the value, not the object representation, which can clobber any bytes that are padding for that particular value. So the example of union U { (u8, u32), (u64) } isn't even guaranteed to preserve padding of the first variant—not even in functions taking an arbitrary U as an argument, because the variant might be known to the compiler through inlining or otherwise.

Combined with the particular language in the ARM ABI especially which clearly talks about passing the value, rather than the object representation, I think that a maximally defensive definition of #[repr(C)] union semantics would need to take this into account.

(C++, interestingly, is much more clear about the implicit sum type nature of a union, but also requires padding to be preserved if I read it correctly.)

tl;dr I'm not convinced that any ABI requires padding bytes to take specific values. But I think that, in C, which bytes of a union are padding are based on the last value properly stored into the union, which is not represented in memory at all.

[alercah]

Actually, I am apparently wrong about C++, according to a note on this page:

For a union with bits that participate in the value representations of some members but not the others, compare-and-exchange might always fail because such padding bits have indeterminate values when they do not participate in the value representation of the active member.

[RalfJung]

But if we look at the C standard, it's actually even worse, I think.

I don't think those parts of the C standard are relevant for ABI discussions. ABIs are unavoidably defined on the assembly level.
The C standard was not intended to also specify ABIs (even though it is now heavily involved in that business), and large parts of it IMO make little sense for that purpose.

So I don't think that the actual ABI of a union is ever any worse than "copy this range of bytes but possibly leave some gaps where data gets lost". I have recently adjusted MiniRust to support such "chunked" unions, and I think this is good enough for the ABIs out there. Or do you know of a counterexample?

[RalfJung]

So the example of union U { (u8, u32), (u64) } isn't even guaranteed to preserve padding of the first variant—not even in functions taking an arbitrary U as an argument, because the variant might be known to the compiler through inlining or otherwise.

This is true if that code is written in C. But I don't see how that has any consequence on Rust code calling Rust code via the C ABI -- and that is the case we are concerned about here.

Combined with the particular language in the ARM ABI especially which clearly talks about passing the value, rather than the object representation, I think that a maximally defensive definition of #[repr(C)] union semantics would need to take this into account.

repr(C) on a struct doesn't mean that all the value representation nonsense works exactly like it does in C. It means that we use the same layout algorithm as C. I think the same is true of unions. We don't promise "everything that is UB in C is also UB in Rust", that would be a terrible promise to make.

tl;dr I'm not convinced that any ABI requires padding bytes to take specific values. But I think that, in C, which bytes of a union are padding are based on the last value properly stored into the union, which is not represented in memory at all.

Yeah I could believe that. Lucky enough I think it has no relevance for Rust.

[alercah]

I think, in the end, I agree with everything you said here.

[JakobDegen]

It's not entirely clear if this question is just asking for documentation or for a resolution to the Rust level issue about what the semantics of #[repr(C)] unions are, but assuming it's the second

[RalfJung]

In MiniRust this issue is resolved by saying that the value of a union consists of several "chunks" of raw memory, that will have Uninit but in between them. There is logic here that computes these chunks for Rust union types, when translating MIR to MiniRust.

As far as I know, this is sufficient to encode the behavior of these repr(C) unions.

I think all that remains to be done here is somehow documenting this as our consensus solution to the problem. But of course this interacts with other questions around the value representation of unions, of which there are a lot. In particular, do we make union bytes "noundef" if they are "noundef" in each variant?

It might be worth opening a new issue that specifically tracks these questions.

[RalfJung]

Closing in favor of #438.