Compute LLVM-agnostic type layouts in rustc.

[eddyb]

Layout for monomorphic types, and some polymorphic ones (e.g. &T where T: Sized),
can now be computed by rustc without involving LLVM in the actual process.

This gives rustc the ability to evaluate size_of or align_of, as well as obtain field offsets.
MIR-based CTFE will eventually make use of these layouts, as will MIR trans, shortly.

Layout computation also comes with a [breaking-change], or two:

• "data-layout" is now mandatory in custom target specifications, reverting the decision from #27076.
  This string is needed because it describes endianness, pointer size and alignments for various types.
  We have the first two and we could allow tweaking alignments in target specifications.
  Or we could also extract the data layout from LLVM and feed it back into rustc.
  However, that can vary with the LLVM version, which is fragile and undermines stability.
  For built-in targets, I've added a check that the hardcoded data-layout matches LLVM defaults.
• transmute calls are checked in a stricter fashion, which fixes #32377

To expand on transmute, there are only 2 allowed patterns: between types with statically known sizes and between pointers with the same potentially-unsized "tail" (which determines the type of unsized metadata they use, if any).
If you're affected, my suggestions are:

• try to use casts (and raw pointer deref) instead of transmutes
• really try to avoid transmute where possible
• if you have a structure, try working on individual fields and unpack/repack the structure instead of transmuting it whole, e.g. transmute::<RefCell<Box<T>>, RefCell<*mut T>>(x) doesn't work, but RefCell::new(Box::into_raw(x.into_inner())) does (and Box::into_raw is just a transmute)

[rust-highfive]

r? @nrc

(rust_highfive has picked a reviewer for you, use r? to override)

[arielb1]

No Layout::Unit? Also Layout::Empty might be nice.

[eddyb]

I don't really see the need for a dedicated unit layout, but Layout::Empty would probably be useful in refining the set of variants, to make sizeof(Result<T, Void>) just sizeof(T).

However, just like invalid value range reusal (i.e. extending non-zero optimizations), it cannot be done before moving trans to use only ty::Layout.

I was hoping we can test the worst-case transmute approach before the beta, but there were complications I had to address.

[solson]

@eddyb How were these defaults chosen? When do they get used?

[eddyb]

They're LLVM's defaults: http://llvm.org/docs/LangRef.html#data-layout.

[Aatch]

Looks like the defaults match LLVMs defaults for the data layout string. The data layout string isn't required to specify every value, so the defaults are used to fill in the gaps.

[solson]

That makes sense, thanks.

[alexcrichton]

My main worry on these data layouts was just that no one actually understands them and they're just opaque blobs we copy around without any thought, and then once that caused a bug one day and LLVM had reasonable defaults for everything it seemed "why not just use that?"

If we validate that our data layout is the same as LLVM, however, then this seems fine to me. I'd be curious if LLVM actually changes anything here between releases (wouldn't that be a huge breaking change for C as well?), but doing this ourselves also seems fine to me.

[solson]

This PR will be a big help for implementing Miri more properly, which hopefully will lead to MIR-based CTFE as @eddyb mentioned in the description.

In general, I think it's a good idea to make rustc less dependent on LLVM where it isn't too difficult to do so.

[eddyb]

@alexcrichton They're documented, though: http://llvm.org/docs/LangRef.html#data-layout.
Also, they used to be much more verbose for redundancy's sake, but they're manageably small now.
The bug you mention was precisely my point: if we get the datalayout from LLVM and there's a bug fix, we don't get notified.

[eddyb]

I've taken a look at the preliminary list of regressions, and even though almost half of them were build failures (timeouts?), the rest displayed a clear pattern:

• in postgres-binary-copy-0.2.1 Option<&'a T> is transmuted to Option<&'b T> - we could try to accommodate for this by comparing the two types, with regions substituted away - or handle Option<&T> like *const T (see below)
• everywhere else, the errors come from a transmute between two pointer types, some of which are wrapped in newtypes, both potentially-fat with the exact same "tail", which is a type parameter

That is, in the latter case, you might have &T -> Rc<RefCell<T>> with T: ?Sized (not that you could make use of that, but it's an extreme example), and both of those types are potentially-fat pointers with the same metadata, depending only on T, now and in the future.

Assuming we want to allow the pattern of casting between maybe-newtyped pointers to maybe-unsized maybe-wrapped type parameters, how should we go about doing it?

The cleanest solution I have in mind, although not the most principled one, is to have two checks:

• first, we try to obtain static sizes for the source and destination and compare them
• if that fails, we attempt to extract potentially-unsized "pointer skeletons" from both types, and compare the pointee "tail" type; this technique also allows transmutes between pointers to ?Sized associated types

While it may be possible to integrate unknown sizes into the regular Layout to get a more principled(?) solution, it would complicate any code working with layouts in monomorphic code.

EDIT: I should mention that if breaking all of those crates is an option, most of them can use pointer casts and dereferences instead - even when a newtype or Option is involved.

[Aatch]

• in postgres-binary-copy-0.2.1 Option<&'a T> is transmuted to Option<&'b T> - we could try to accommodate for this by comparing the two types, with regions substituted away - or handle Option<&T> like *const T (see below)

I think regions should be ignored/substituted here. Makes more general sense, since the regions shouldn't affect the layout of the type.

[nrc]

r? @nikomatsakis

[nikomatsakis]

@eddyb

Assuming we want to allow the pattern of casting between maybe-newtyped pointers to maybe-unsized maybe-wrapped type parameters, how should we go about doing it?

I think we do.

While it may be possible to integrate unknown sizes into the regular Layout to get a more principled(?) solution, it would complicate any code working with layouts in monomorphic code.

I don't have an informed enough opinion yet. I'll try to take a look at your code but it may not happen until Monday, since I'm kind of busy today with other (non-Rust-related) things. But it seems to me like we could probably use a similar trick to what we do today, where for things where T: ?Sized we compute the layout multiple times, under different assumptions (T is definitely sized, T is definitely unsized, etc). But maybe your suggestion turns out nicer -- it just seems like it may wind up duplicating some layout logic. But perhaps not.

[nikomatsakis]

in postgres-binary-copy-0.2.1 Option<&'a T> is transmuted to Option<&'b T> - we could try to accommodate for this by comparing the two types, with regions substituted away - or handle Option<&T> like *const T (see below)

Oh, and clearly regions shouldn't affect the result, but I would expect this equivalency to fallout in a more general way than strict type equality?

[eddyb]

@nikomatsakis Just trying out combinations is pretty fragile, if we ever want to add more kinds of fat pointer metadata (or if we make it fully custom).

The "perfect" solution IMO is to enforce static sizes because that would allow a size_of::<T>() == size_of::<U>() bound on transmute in the future, but I am open to non-static, yet strict, solutions.

That's because we could end up simplifying, e.g. size_of::<*mut T>() == size_of::<Option<&T>>() to true based on the "pointer skeleton" rule mentioned above.

[nikomatsakis]

@eddyb

Just trying out combinations is pretty fragile, if we ever want to add more kinds of fat pointer metadata (or if we make it fully custom).

I see. Good point.

[nikomatsakis]

I don't really care, but I imagine the use of privacy here was to help ensure people don't do stupid things like let mut size = ...; size.bytes *= 2;, but rather encourage them to go through these (presumably more careful) APIs.

[eddyb]

I actually initially wanted to use the highest bit to indicate that the rest of the size is the base size of an unsized type, instead of an exact size, but I ended up dealing with unsized types differently.
I could go back to private raw and add more checks for 2^61 overflow if you want to.

[bors]

📌 Commit c7d564d has been approved by nikomatsakis

[bors]

⌛️ Testing commit c7d564d with merge 542c7d1...

[bors]

💔 Test failed - auto-win-msvc-32-opt

[eddyb]

@bors retry

[bors]

⌛️ Testing commit c7d564d with merge 6ece144...

[GuillaumeGomez]

Oh nice! Thanks for adding it! \o/

[eddyb]

I only moved it.

[bors]

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-cargotest, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-opt-mir, auto-linux-64-opt-rustbuild, auto-linux-64-x-android-t, auto-linux-64-x-armhf, auto-linux-64-x-armsf, auto-linux-64-x-freebsd, auto-linux-64-x-netbsd, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-nopt-t, auto-mac-64-opt, auto-mac-64-opt-rustbuild, auto-mac-ios-opt, auto-win-gnu-32-nopt-t, auto-win-gnu-32-opt, auto-win-gnu-32-opt-rustbuild, auto-win-gnu-64-nopt-t, auto-win-gnu-64-opt, auto-win-msvc-32-cross-opt, auto-win-msvc-32-opt, auto-win-msvc-64-cargotest, auto-win-msvc-64-opt, auto-win-msvc-64-opt-mir, auto-win-msvc-64-opt-rustbuild

[eddyb]

Nominated for backporting to beta to fix #32377.

[solson]

@eddyb In Miri I use rustc::infer::normalize_associated_type. Its source code is slightly different and has a warning about only being callable from trans, but most of it is duplicated. Could they be unified somehow?

[eddyb]

See how I call ty.layout(&infcx) from trans. The effect is the same in that case, it's only different elsewhere (intrinsicck).

[pnkfelix]

after a week of exercising in the nightly release, the compiler team has decided to accept this for beta, largely because it fixes a real regression (#32377), and that outweighed the relative risk of backporting such a largish change to beta.