Refactor type memory layouts and ABIs, to be more general and easier to optimize.

[eddyb]

To combat combinatorial explosion, type layouts are now described through 3 orthogonal properties:

• Variants describes the plurality of sum types (where applicable)
  • Single is for one inhabited/active variant, including all C structs and unions
  • Tagged has its variants discriminated by an integer tag, including C enums
  • NicheFilling uses otherwise-invalid values ("niches") for all but one of its inhabited variants
• FieldPlacement describes the number and memory offsets of fields (if any)
  • Union has all its fields at offset 0
  • Array has offsets that are a multiple of its stride; guarantees all fields have one type
  • Arbitrary records all the field offsets, which can be out-of-order
• Abi describes how values of the type should be passed around, including for FFI
  • Uninhabited corresponds to no values, associated with unreachable control-flow
  • Scalar is ABI-identical to its only integer/floating-point/pointer "scalar component"
  • ScalarPair has two "scalar components", but only applies to the Rust ABI
  • Vector is for SIMD vectors, typically #[repr(simd)] structs in Rust
  • Aggregate has arbitrary contents, including all non-transparent C structs and unions

Size optimizations implemented so far:

• ignoring uninhabited variants (i.e. containing uninhabited fields), e.g.:
  • Option<!> is 0 bytes
  • Result<T, !> has the same size as T
• using arbitrary niches, not just 0, to represent a data-less variant, e.g.:
  • Option<bool>, Option<Option<bool>>, Option<Ordering> are all 1 byte
  • Option<char> is 4 bytes
• using a range of niches to represent multiple data-less variants, e.g.:
  • enum E { A(bool), B, C, D } is 1 byte

Code generation now takes advantage of Scalar and ScalarPair to, in more cases, pass around scalar components as immediates instead of indirectly, through pointers into temporary memory, while avoiding LLVM's "first-class aggregates", and there's more untapped potential here.

Closes #44426, fixes #5977, fixes #14540, fixes #43278.

[rust-highfive]

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

[eddyb]

Started a crater run (we'll run cargobomb later, but that takes longer).

[Gankro]

holy shit

[scottmcm]

Wow! I think this might fix #43278 too.

[arielb1]

Why is this always 1? Because you do a pointercast instead of a gep in the offset 0 case?

[eddyb]

Yes, and the assert ensures it.

[arielb1]

Why are you removing this test?

[eddyb]

This is one of those things that don't trigger a typeck error but is totally broken, and I didn't want to deal with it.

[sgrif]

What does "totally broken" mean in this context?

[eddyb]

Something hid the ICE and #30276 was closed, but that was accidental. This is more or less a bug in WF-checking of function arguments in signatures, which @nikomatsakis somewhat insisted on keeping around, which allows fn(Unsized) to even exist (it shouldn't, for now).

[arielb1]

Is the case this test was intended to check unreachable?

[eddyb]

Now, yeah, ScalarPair only works by having the placement of the second scalar at an ABI-aligned offset, so it's incompatible with packing.

[arielb1]

Why do "general" enums use FieldPlacement::Union for the discriminant only and "niche" enums use FieldPlacement::Arbitrary?

[eddyb]

Because a tag is always at offset 0 whereas a niche can be at some offset - we could have a variant of FieldPlacement that's just one Size to avoid the Vec but that's about it.

[arielb1]

But why we can't use FieldPlacement::Arbitrary for both of them? "Normal" structs with one field don't suddenly use FieldPlacement::Union because they can.

[eddyb]

I mean, FieldPlacement::Arbitrary is a bit of a waste, and I considered using FieldPlacement::Union for normal structs too, but then I have checks for Union in some places which would conflict with that now. I think you're right and I should make the change here.

EDIT: done.

EDIT2: That's funny, not using FieldPlacement::Union results in some missed optimizations (codegen/{alloc-optimisation,vec-optimizes-away}). I'll try to see what I can do about it.

EDIT3: None of the tricks I've tried work, I'll just leave it like this to avoid pessimizing tagged enums for now.

[arielb1]

Maybe add a comment to that effect?

[eddyb]

I've just moved Uninhabited from Variants to Abi - that way, we can still compute field types for uninhabited types, but also signal the fact that there's no valid values anyway.

[eddyb]

@scottmcm Your original example from #43278 compiles to a single ret i32 %0 with this PR.

[mrhota]

@eddyb also would seem to fix #5977

any relevance to #14540? #6791? And am I right in thinking this work puts us on a path for doing some of the optimizations here rust-lang/rfcs#1230?

For travelers from the future, also see the older, far less ambitious PRs #31215 and #36237.

[bors]

☔️ The latest upstream changes (presumably #45031) made this pull request unmergeable. Please resolve the merge conflicts.

[eddyb]

@mrhota Are you sure you meant #6791? That issue was closed more than three years ago.

The problem with #31215 and #36237 was that they only added code where this PR is a net negative.

This PR implements most of rust-lang/rfcs#1230, the description there is confused in places (i.e. it's a list of special cases, but several of them happen to be the exact same optimization, just scattered).
The cases I couldn't do here:

• Option<Option<(&T, &U)>> requires us to initialize the second &U with a non-zero value for the inner Option, which we don't do right now (and could be expensive in general)
• enum E { A(&T / Enum / etc., A...), B(B...), C(C...) } (where A... takes up at least as much space as each of B... and C...) requires trickier search logic, and would be hard/impossible to integrate with the current rustc_trans::mir::constant, we'd have to wait for miri to experiment with it
• enum F { A(T, A), B(T, B) } as (T, enum G { A(A), B(B) }) where we already are able to optimize G - this includes enum Foo { A(&u8, &u8), B(&u8) } and many others - requires effectively a 2D layout process (although, just like the previous case, it's tricky and there may be many valid approaches) and will likely need a bunch more caching, around niches probably

EDIT: for an updated view on these hard cases, see rust-lang/rfcs#1230 (comment).

My approach with this PR has been to avoid doing anything that would pessimize codegen, and one example of this is that enum X { A(bool), B, C } and enum Y { A, B, C(bool) } use (invalid for bool) values 2 and 3 for the nullary variants, but enum Z { A, B(bool), C } uses 2 and 4, wasting 3, which (if you were to transmute it) matches B, to avoid a more runtime-expensive value remapping when the discriminant value is needed (e.g. for every match).

[eddyb]

I can't reproduce the Travis failure locally, even after commenting out most of my config.toml 😞

[eddyb]

@alexcrichton @Mark-Simulacrum I hope you don't mind that I just pushed a temporary commit to build on Travis without LLVM 3.7, to check if it's some old LLVM bug getting triggered.

[eddyb]

Confirmed successful build with newer (4.0?) LLVM. I'll try to reproduce locally.

[eddyb]

@alexcrichton @Mark-Simulacrum May I suggest outputting a signal number by default?
Currently, all I see (for example, in Travis CI) is:

error: Could not compile `quote`.

But if I turn verbose mode on, there's an extra (manually wrapped for your convenience):

Caused by:
  process didn't exit successfully: `
/home/eddy/Projects/rust-2/build/bootstrap/debug/rustc --crate-name quote
/home/eddy/.cargo/registry/src/github.com-1ecc6299db9ec823/quote-0.3.15/src/lib.rs
--crate-type lib --emit=dep-info,link -C opt-level=2 -C metadata=396e074f31421a34
-C extra-filename=-396e074f31421a34
--out-dir /home/eddy/Projects/rust-2/build/x86_64-unknown-linux-gnu/stage2-tools/release/deps
-L dependency=/home/eddy/Projects/rust-2/build/x86_64-unknown-linux-gnu/stage2-tools/release/deps
--cap-lints allow` (signal: 11, SIGSEGV: invalid memory reference)

Printing just "rustc exited with signal: 11, SIGSEGV: invalid memory reference" by default would be really great, although I'd understand if it's too much of an edge case to bother with.