Experiment with pipelining rustc invocations via metadata

[alexcrichton]

In talking with the Gecko team at the most recent work week one of the difficulties with building and managing Rust code is dealing with incremental changes and how they can force a recompilation of many crates in the crate graph. Notably when you incrementally change a crate you'll typically have a linear and single-threaded compilation back up to the root (hopefully not as we add more parallelism to rustc, but still). This issue, however, is a different strategy for hopefully even more quickly using up all the cores on a build machine.

Today rustc requires that when a crate is recompiled then everything that transitively depends on it is also recompiled. With incremental compilation these recompilations hopefully won't take too too long, but if you do actually change code that affects upstream crates you want to get to the point of compiling them as quickly as possible. The canonical example we were using looks like:

bitflags -> nsstring -> nserror

That is, the nserror crate depends on nsstring, and nsstring depends on the bitflags crate. Cargo today, when bitflags changes, will recompile bitflags, wait for it to finish entirely, and then start the compilation of nsstring. (and so on and so forth down the graph).

Since these are all rlib compilations, however, the compiler doesn't actually need the full rlib of bitflags to start compiling nsstring! Instead rustc actually only needs the metadata output and then it's good to go to start compiling nsstring.

So instead of a graph like above, we could imagine something like:

bitflags metadata -> nsstring metadata -> nserror metadata -> nserror rlib
                  \                      \                   
                   bitflags rlib          nsstring rlib        

The thinking is that we could invoke rustc twice per crate instead of once per crate. The first invocation would be --emit metadata and the second invocation would be --emit link (like Cargo does today). There's two crucial bits to this which could provide some nice speedups:

• Metadata omisson could be done with incremental enabled, ideally speeding up the rlib generating rustc command by quite a bit.
• Compilation of downstream crates can happen in parallel as soon as metadata is available. That means that bitflags rlib generation can happen in parallel with other compilations.

The extra parallelism opportunities here could provide some nice wins in terms of comiple times, although they'll largely be dependent on how incremental the rlib generation can be from the previous metadata generation. In any case this is likely a great avenue to get started on for optimizing build times!

cc @luser
cc @mshal
cc @michaelwoerister

---

FWIW we've also toyed around with the idea here in rustc itself. One way we could implement this in a Cargo specific fashion (aka something external build systems wouldn't benefit from) is an OS pipe from rustc to Cargo to allow Cargo to get notified when rustc is finished emitting the metadata. That way we'd invoke rustc only once and would have a 100% guarantee about how quickly the rlib generation can start (aka immediately). This may not be worth it in the initial testing however.

[chmanchester]

I was hacking up a prototype of this to see what sort of wins we might see on the rust in mozilla-central, and I got fairly far, but it appears the ability to link an rlib from metadata only has regressed (or I'm misunderstanding some mechanic of this proposal):

$ rustc --crate-type rlib --emit metadata foo.rs 
$ rustc --extern c=libfoo.rmeta --crate-type rlib bar.rs

where foo.rs and bar.rs are trivial crates, bar depending on foo, succeeds on rust 1.25 but fails on 1.26 and later with the following:

error: internal compiler error: librustc_mir/monomorphize/collector.rs:746: Cannot create local mono-item for DefId(11/0:3 ~ c[8787]::foo[0])

which looks like rust-lang/rust#50865, although neither that issue nor related issues mention trying to link to rmeta files.

[eddyb]

@chmanchester That won't work without -Zalways-encode-mir - see also rust-lang/rust#38913

[luser]

Here's a Twitter thread where some Swift folks talk about doing essentially the same build optimization for Swift projects. And some profiler output comparing builds with/without the optimization.

[alexcrichton]

Moving this to #6660 to have a modern take on things.