rustc: Enable LTO and multiple codegen units #44783

alexcrichton · 2017-09-23T03:32:13Z

This commit is a refactoring of the LTO backend in Rust to support compilations
with multiple codegen units. The immediate result of this PR is to remove the
artificial error emitted by rustc about -C lto -C codegen-units-8, but longer
term this is intended to lay the groundwork for LTO with incremental compilation
and ultimately be the underpinning of ThinLTO support.

The problem here that needed solving is that when rustc is producing multiple
codegen units in one compilation LTO needs to merge them all together.
Previously only upstream dependencies were merged and it was inherently relied
on that there was only one local codegen unit. Supporting this involved
refactoring the optimization backend architecture for rustc, namely splitting
the optimize_and_codegen function into optimize and codegen. After an LLVM
module has been optimized it may be blocked and queued up for LTO, and only
after LTO are modules code generated.

Non-LTO compilations should look the same as they do today backend-wise, we'll
spin up a thread for each codegen unit and optimize/codegen in that thread. LTO
compilations will, however, send the LLVM module back to the coordinator thread
once optimizations have finished. When all LLVM modules have finished optimizing
the coordinator will invoke the LTO backend, producing a further list of LLVM
modules. Currently this is always a list of one LLVM module. The coordinator
then spawns further work to run LTO and code generation passes over each module.

In the course of this refactoring a number of other pieces were refactored:

Management of the bytecode encoding in rlibs was centralized into one module
instead of being scattered across LTO and linking.
Some internal refactorings on the link stage of the compiler was done to work
directly from CompiledModule structures instead of lists of paths.
The trans time-graph output was tweaked a little to include a name on each
bar and inflate the size of the bars a little

rust-highfive · 2017-09-23T03:32:17Z

r? @arielb1

(rust_highfive has picked a reviewer for you, use r? to override)

alexcrichton · 2017-09-23T03:32:21Z

r? @michaelwoerister

estebank · 2017-09-23T05:36:50Z

tidy error: /checkout/src/librustc_trans/back/bytecode.rs: incorrect license

hanna-kruppe

The changes to storing and loading bitcode in rlibs LGTM. I don't know enough about the whole work items and job server stuff to review the rest, though I did skim it.

hanna-kruppe · 2017-09-23T11:07:09Z

src/librustc_trans/back/bytecode.rs

+//! elsewhere, so we currently compress the bytecode via deflate to avoid taking
+//! up too much space on disk.
+//!
+//! After compressing the bytcode we then have the rest of the format to


Typo: s/bytcode/bytecode/

hanna-kruppe · 2017-09-23T11:07:23Z

src/librustc_trans/back/bytecode.rs

+//! up too much space on disk.
+//!
+//! After compressing the bytcode we then have the rest of the format to
+//! basically deal with various bugs in various archive implemenatations. THe


Typo: s/THe/The/

hanna-kruppe · 2017-09-23T11:14:55Z

src/librustc_trans/back/link.rs

@@ -1310,7 +1225,7 @@ fn add_upstream_rust_crates(cmd: &mut Linker,
        archive.update_symbols();

        for f in archive.src_files() {
-            if f.ends_with("bytecode.deflate") || f == METADATA_FILENAME {
+            if f.ends_with("bytecode.encoded") || f == METADATA_FILENAME {


Since you're already updating all the places that involve this magic suffix, maybe pull the string out into a constant?

hanna-kruppe · 2017-09-23T11:20:52Z

src/librustc_trans/back/lto.rs

        llvm::LLVMRustRunRestrictionPass(llmod,
                                         ptr as *const *const libc::c_char,
-                                         arr.len() as libc::size_t);
+                                         symbol_white_list.len() as libc::size_t);
+        cgcx.save_temp_bitcode(&module, "lto.after-restrictoin");


Typo: s/restrictoin/restriction/

hanna-kruppe · 2017-09-23T12:31:32Z

src/rustllvm/RustWrapper.cpp

+
+  std::string Err;
+  raw_string_ostream Stream(Err);
+  DiagnosticPrinterRawOStream DP(Stream);


This seems unused?

hanna-kruppe · 2017-09-23T12:50:09Z

src/librustc_trans/back/bytecode.rs

+            return Err(format!("bytecode corrupted"))
+        }
+        let identifier_len = unsafe {
+            u32::from_le(ptr::read_unaligned(data.as_ptr() as *const u32)) as usize


I wish this code used byteorder. Oh well, a refactoring for another day.

michaelwoerister · 2017-09-23T16:43:05Z

Nice! I'll review this after the weekend.

kennytm · 2017-09-25T08:58:26Z

src/test/run-pass/lto-many-codegen-units.rs

+// option. This file may not be copied, modified, or distributed
+// except according to those terms.
+
+// compile-flags: -C lto -C codegen-units=8


🤔

[00:49:33] ---- [run-pass] run-pass/lto-many-codegen-units.rs stdout ---- [00:49:33] [00:49:33] error: compilation failed! [00:49:33] status: exit code: 101 [00:49:33] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/run-pass/lto-many-codegen-units.rs" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass" "--target=x86_64-unknown-linux-gnu" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass/lto-many-codegen-units.stage2-x86_64-unknown-linux-gnu" "-Crpath" "-O" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-C" "lto" "-C" "codegen-units=8" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-pass/lto-many-codegen-units.stage2-x86_64-unknown-linux-gnu.run-pass.libaux" [00:49:33] stdout: [00:49:33] ------------------------------------------ [00:49:33] [00:49:33] ------------------------------------------ [00:49:33] stderr: [00:49:33] ------------------------------------------ [00:49:33] error: cannot prefer dynamic linking when performing LTO [00:49:33] [00:49:33] note: only 'staticlib', 'bin', and 'cdylib' outputs are supported with LTO [00:49:33] [00:49:33] error: aborting due to previous error [00:49:33] [00:49:33] thread '<unnamed>' panicked at 'Box<Any>', /checkout/src/librustc_trans/back/write.rs:521:65 [00:49:33] [00:49:33] ------------------------------------------ [00:49:33] [00:49:33] thread '[run-pass] run-pass/lto-many-codegen-units.rs' panicked at 'explicit panic', /checkout/src/tools/compiletest/src/runtest.rs:2433:8 [00:49:33] note: Run with `RUST_BACKTRACE=1` for a backtrace. [00:49:33] [00:49:33] [00:49:33] failures: [00:49:33] [run-pass] run-pass/lto-many-codegen-units.rs [00:49:33] [00:49:33] test result: FAILED. 2762 passed; 1 failed; 8 ignored; 0 measured; 0 filtered out

michaelwoerister

Looks great, thanks @alexcrichton!

r=me with the comments addressed.

michaelwoerister · 2017-09-25T08:39:16Z

src/librustc_trans/back/bytecode.rs

+//! up too much space on disk.
+//!
+//! After compressing the bytecode we then have the rest of the format to
+//! basically deal with various bugs in various archive implemenatations. The


implementations

michaelwoerister · 2017-09-25T08:40:45Z

src/librustc_trans/back/bytecode.rs

+
+    // Next is the LLVM module deflate compressed, prefixed with its length. We
+    // don't know its length yet, so fill in 0s
+    let deflated_size = encoded.len();


The name of the binding here is a bit misleading. Could you change that to something like deflated_size_pos?

michaelwoerister · 2017-09-25T08:53:24Z

src/librustc_trans/back/lto.rs

+    // bitcode. All modules were translated in their own LLVM context, however,
+    // and we want to move everything to the same LLVM context. Currently the
+    // way we know of to do that is to serialize them to a string and them parse
+    // them later. Not great but hey, that's why it's "fat" LTO, right?


Yeah, this won't be the most performant thing to do but my guess is that it's still cheap compared to optimizing the resulting mega-module. Plus, it's a corner case anyway. So: 👍

michaelwoerister · 2017-09-25T09:06:48Z

src/librustc_trans/back/write.rs

@@ -1304,6 +1365,8 @@ fn start_executing_work(tcx: TyCtxt,
        let mut compiled_modules = vec![];
        let mut compiled_metadata_module = None;
        let mut compiled_allocator_module = None;
+        let mut needs_lto = Vec::new();


Would mind updating the big comment above with how LTO fits into the picture. It's not immediately clear to me what work gets executed when and where -- e.g. it seems that a whole bunch of work gets done on the scheduler thread via generate_lto_work. That deserves to be mentioned, I think.

michaelwoerister · 2017-09-25T09:10:56Z

src/librustc_trans/base.rs

-    // this as we're not working with this dual "rlib/dylib" functionality.
-    let allocator_module = if tcx.sess.lto() {
-        None
-    } else if let Some(kind) = tcx.sess.allocator_kind.get() {


Oh, that's nice that we can get rid of the special casing here. If it's easy, it would be nice to make sure we don't serialize the big "actual-code-module" and merge that into the tiny allocator module. Although it's a special case to even have an allocator module, right?

Good point! I'll just switch LTO to merging everything into the "costliest" module

alexcrichton · 2017-09-25T14:52:20Z

@bors: r=michaelwoerister

bors · 2017-09-25T14:52:23Z

📌 Commit bf3be56 has been approved by michaelwoerister

kennytm · 2017-09-25T16:11:08Z

src/rustllvm/RustWrapper.cpp

+LLVMRustModuleCost(LLVMModuleRef M) {
+  Module &Mod = *unwrap(M);
+  uint64_t cost = 0;
+  for (auto &GO : Mod.global_objects()) {


Cannot build rustllvm, at least for the CI (LLVM 3.7):

[00:04:04] cargo:warning=../rustllvm/RustWrapper.cpp: In function 'uint64_t LLVMRustModuleCost(LLVMModuleRef)': [00:04:04] cargo:warning=../rustllvm/RustWrapper.cpp:1462:23: error: 'class llvm::Module' has no member named 'global_objects' [00:04:04] cargo:warning= for (auto &GO : Mod.global_objects()) { [00:04:04] cargo:warning= ^ [00:04:04] exit code: 1

alexcrichton · 2017-09-25T16:54:02Z

@bors: r-

alexcrichton · 2017-09-25T16:55:30Z

@bors: r=michaelwoerister

bors · 2017-09-25T16:55:31Z

📌 Commit 6154e04 has been approved by michaelwoerister

bors · 2017-09-25T20:41:49Z

☔ The latest upstream changes (presumably #44085) made this pull request unmergeable. Please resolve the merge conflicts.

alexcrichton · 2017-09-25T22:51:38Z

@bors: r=michaelwoerister

bors · 2017-09-25T22:51:39Z

📌 Commit c3c6592 has been approved by michaelwoerister

kennytm · 2017-09-26T06:22:54Z

CI is still failing, not sure if legit.

[00:48:29] ---- [codegen] codegen/lto-removes-invokes.rs stdout ----
[00:48:29] 	
[00:48:29] error: verification with 'FileCheck' failed
[00:48:29] status: exit code: 2
[00:48:29] command: "/usr/lib/llvm-3.7/bin/FileCheck" "--input-file" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/codegen/lto-removes-invokes.ll" "/checkout/src/test/codegen/lto-removes-invokes.rs"
[00:48:29] stdout:
[00:48:29] ------------------------------------------
[00:48:29] 
[00:48:29] ------------------------------------------
[00:48:29] stderr:
[00:48:29] ------------------------------------------
[00:48:29] Could not open input file '/checkout/obj/build/x86_64-unknown-linux-gnu/test/codegen/lto-removes-invokes.ll': No such file or directory
[00:48:29] 
[00:48:29] ------------------------------------------
[00:48:29] 
[00:48:29] thread '[codegen] codegen/lto-removes-invokes.rs' panicked at 'explicit panic', /checkout/src/tools/compiletest/src/runtest.rs:2433:8
[00:48:29] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[00:48:29] 
[00:48:29] 
[00:48:29] failures:
[00:48:29]     [codegen] codegen/lto-removes-invokes.rs
[00:48:29] 
[00:48:29] test result: FAILED. 37 passed; 1 failed; 11 ignored; 0 measured; 0 filtered out

alexcrichton · 2017-09-26T15:58:21Z

@bors: r-

definitely legit

alexcrichton · 2017-09-26T16:05:31Z

@bors: r=michaelwoerister

bors · 2017-09-26T16:05:32Z

📌 Commit 3547f2d has been approved by michaelwoerister

bors · 2017-09-28T08:44:20Z

⌛ Testing commit 3547f2d8e3cf4ab63041b7463046a6d9d893205a with merge 3d3506dc9f13c3dc72ceb531e62011af86774cbe...

bors · 2017-09-28T09:10:09Z

💔 Test failed - status-travis

kennytm · 2017-09-28T09:16:04Z

src/librustc_trans/back/bytecode.rs

+    encoded[deflated_size_pos + 4] = (bytecode_len >> 32) as u8;
+    encoded[deflated_size_pos + 5] = (bytecode_len >> 40) as u8;
+    encoded[deflated_size_pos + 6] = (bytecode_len >> 48) as u8;
+    encoded[deflated_size_pos + 7] = (bytecode_len >> 56) as u8;


bytecode_len is an usize. It should be cast to u64, or the last 4 statements should be #[cfg]'ed out.

Failed on `i686-gnu`:

[00:19:51] error: bitshift exceeds the type's number of bits [00:19:51] --> /checkout/src/librustc_trans/back/bytecode.rs:88:38 [00:19:51] | [00:19:51] 88 | encoded[deflated_size_pos + 4] = (bytecode_len >> 32) as u8; [00:19:51] | ^^^^^^^^^^^^^^^^^^^^ [00:19:51] | [00:19:51] = note: #[deny(exceeding_bitshifts)] on by default [00:19:51] [00:19:51] error: bitshift exceeds the type's number of bits [00:19:51] --> /checkout/src/librustc_trans/back/bytecode.rs:89:38 [00:19:51] | [00:19:51] 89 | encoded[deflated_size_pos + 5] = (bytecode_len >> 40) as u8; [00:19:51] | ^^^^^^^^^^^^^^^^^^^^ [00:19:51] [00:19:51] error: bitshift exceeds the type's number of bits [00:19:51] --> /checkout/src/librustc_trans/back/bytecode.rs:90:38 [00:19:51] | [00:19:51] 90 | encoded[deflated_size_pos + 6] = (bytecode_len >> 48) as u8; [00:19:51] | ^^^^^^^^^^^^^^^^^^^^ [00:19:51] [00:19:51] error: bitshift exceeds the type's number of bits [00:19:51] --> /checkout/src/librustc_trans/back/bytecode.rs:91:38 [00:19:51] | [00:19:51] 91 | encoded[deflated_size_pos + 7] = (bytecode_len >> 56) as u8; [00:19:51] | ^^^^^^^^^^^^^^^^^^^^ [00:19:51] [00:19:51] error: aborting due to 4 previous errors [00:19:51] [00:19:51] error: Could not compile `rustc_trans`.

alexcrichton · 2017-09-28T13:55:57Z

@bors: r=michaelwoerister

bors · 2017-09-28T13:55:58Z

📌 Commit b377366 has been approved by michaelwoerister

bors · 2017-09-29T02:52:24Z

⌛ Testing commit b377366f7bb60653cb10b56ddd61313aca67906d with merge e1e340a524902f341944f9c68ef57b6aa69c4b39...

bors · 2017-09-29T03:54:08Z

💔 Test failed - status-travis

kennytm · 2017-09-29T04:40:08Z

The LTO run-pass tests seg-faulted on i686-gnu.

[00:57:29] ---- [run-pass] run-pass/lto-many-codegen-units.rs stdout ----
[00:57:29] 	
[00:57:29] error: compilation failed!
[00:57:29] status: signal: 11
[00:57:29] command: "/checkout/obj/build/i686-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/run-pass/lto-many-codegen-units.rs" "-L" "/checkout/obj/build/i686-unknown-linux-gnu/test/run-pass" "--target=i686-unknown-linux-gnu" "-o" "/checkout/obj/build/i686-unknown-linux-gnu/test/run-pass/lto-many-codegen-units.stage2-i686-unknown-linux-gnu" "-Crpath" "-O" "-Lnative=/checkout/obj/build/i686-unknown-linux-gnu/native/rust-test-helpers" "-C" "lto" "-C" "codegen-units=8" "-L" "/checkout/obj/build/i686-unknown-linux-gnu/test/run-pass/lto-many-codegen-units.stage2-i686-unknown-linux-gnu.run-pass.libaux"
[00:57:29] stdout:
[00:57:29] ------------------------------------------
[00:57:29] 
[00:57:29] ------------------------------------------
[00:57:29] stderr:
[00:57:29] ------------------------------------------
[00:57:29] 
[00:57:29] ------------------------------------------
[00:57:29] 
[00:57:29] thread '[run-pass] run-pass/lto-many-codegen-units.rs' panicked at 'explicit panic', /checkout/src/tools/compiletest/src/runtest.rs:2433:8
[00:57:29] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[00:57:29] 
...
[00:57:29] 
[00:57:29] 
[00:57:29] failures:
[00:57:29]     [run-pass] run-pass/lto-many-codegen-units.rs
[00:57:29]     [run-pass] run-pass/panic-runtime/lto-abort.rs
[00:57:29]     [run-pass] run-pass/panic-runtime/lto-unwind.rs
[00:57:29]     [run-pass] run-pass/sepcomp-lib-lto.rs
[00:57:29]     [run-pass] run-pass/stack-probes-lto.rs
[00:57:29] 
[00:57:29] test result: FAILED. 2763 passed; 5 failed; 4 ignored; 0 measured; 0 filtered out

alexcrichton · 2017-09-29T07:40:37Z

I spy.... at least one use after free

alexcrichton · 2017-09-29T07:43:02Z

@bors: r=michaelwoerister

bors · 2017-09-29T07:43:03Z

📌 Commit c077360 has been approved by michaelwoerister

bors · 2017-09-29T09:22:43Z

⌛ Testing commit c077360147a7fe2cc03dfdca647f7c7b95614284 with merge 0b4c7760c55425321b2944ebfa6fc4f7c0fe1884...

bors · 2017-09-29T10:10:12Z

💔 Test failed - status-travis

kennytm · 2017-09-29T11:30:40Z

The compile-fail tests involving ASM did not fail in x86_64-gnu-llvm-3.7.

[00:43:50] ---- [compile-fail] compile-fail/asm-src-loc-codegen-units.rs stdout ----
[00:43:50] 	
[00:43:50] error: compile-fail test compiled successfully!
[00:43:50] status: exit code: 0
[00:43:50] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/compile-fail/asm-src-loc-codegen-units.rs" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail/asm-src-loc-codegen-units.stage2-x86_64-unknown-linux-gnu" "-Crpath" "-O" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-C" "codegen-units=2" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail/asm-src-loc-codegen-units.stage2-x86_64-unknown-linux-gnu.compile-fail.libaux" "-A" "unused"
[00:43:50] stdout:
[00:43:50] ------------------------------------------
[00:43:50] 
[00:43:50] ------------------------------------------
[00:43:50] stderr:
[00:43:50] ------------------------------------------
[00:43:50] 
[00:43:50] ------------------------------------------
[00:43:50] 
[00:43:50] thread '[compile-fail] compile-fail/asm-src-loc-codegen-units.rs' panicked at 'explicit panic', /checkout/src/tools/compiletest/src/runtest.rs:2433:8
[00:43:50] note: Run with `RUST_BACKTRACE=1` for a backtrace.
[00:43:50] 
[00:43:50] ---- [compile-fail] compile-fail/asm-src-loc.rs stdout ----
[00:43:50] 	
[00:43:50] error: compile-fail test compiled successfully!
[00:43:50] status: exit code: 0
[00:43:50] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/compile-fail/asm-src-loc.rs" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail/asm-src-loc.stage2-x86_64-unknown-linux-gnu" "-Crpath" "-O" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail/asm-src-loc.stage2-x86_64-unknown-linux-gnu.compile-fail.libaux" "-A" "unused"
[00:43:50] stdout:
[00:43:50] ------------------------------------------
[00:43:50] 
[00:43:50] ------------------------------------------
[00:43:50] stderr:
[00:43:50] ------------------------------------------
[00:43:50] 
[00:43:50] ------------------------------------------
[00:43:50] 
[00:43:50] thread '[compile-fail] compile-fail/asm-src-loc.rs' panicked at 'explicit panic', /checkout/src/tools/compiletest/src/runtest.rs:2433:8
[00:43:50] 
[00:43:50] 
[00:43:50] failures:
[00:43:50]     [compile-fail] compile-fail/asm-src-loc-codegen-units.rs
[00:43:50]     [compile-fail] compile-fail/asm-src-loc.rs
[00:43:50] 
[00:43:50] test result: FAILED. 2748 passed; 2 failed; 13 ignored; 0 measured; 0 filtered out

bors · 2017-09-29T13:02:19Z

☔ The latest upstream changes (presumably #44853) made this pull request unmergeable. Please resolve the merge conflicts.

This commit is a refactoring of the LTO backend in Rust to support compilations with multiple codegen units. The immediate result of this PR is to remove the artificial error emitted by rustc about `-C lto -C codegen-units-8`, but longer term this is intended to lay the groundwork for LTO with incremental compilation and ultimately be the underpinning of ThinLTO support. The problem here that needed solving is that when rustc is producing multiple codegen units in one compilation LTO needs to merge them all together. Previously only upstream dependencies were merged and it was inherently relied on that there was only one local codegen unit. Supporting this involved refactoring the optimization backend architecture for rustc, namely splitting the `optimize_and_codegen` function into `optimize` and `codegen`. After an LLVM module has been optimized it may be blocked and queued up for LTO, and only after LTO are modules code generated. Non-LTO compilations should look the same as they do today backend-wise, we'll spin up a thread for each codegen unit and optimize/codegen in that thread. LTO compilations will, however, send the LLVM module back to the coordinator thread once optimizations have finished. When all LLVM modules have finished optimizing the coordinator will invoke the LTO backend, producing a further list of LLVM modules. Currently this is always a list of one LLVM module. The coordinator then spawns further work to run LTO and code generation passes over each module. In the course of this refactoring a number of other pieces were refactored: * Management of the bytecode encoding in rlibs was centralized into one module instead of being scattered across LTO and linking. * Some internal refactorings on the link stage of the compiler was done to work directly from `CompiledModule` structures instead of lists of paths. * The trans time-graph output was tweaked a little to include a name on each bar and inflate the size of the bars a little

alexcrichton · 2017-09-30T07:22:44Z

@bors: r=michaelwoerister

bors · 2017-09-30T07:22:45Z

📌 Commit ded38db has been approved by michaelwoerister

bors · 2017-09-30T15:01:41Z

⌛ Testing commit ded38db with merge c6884b1...

…ster rustc: Enable LTO and multiple codegen units This commit is a refactoring of the LTO backend in Rust to support compilations with multiple codegen units. The immediate result of this PR is to remove the artificial error emitted by rustc about `-C lto -C codegen-units-8`, but longer term this is intended to lay the groundwork for LTO with incremental compilation and ultimately be the underpinning of ThinLTO support. The problem here that needed solving is that when rustc is producing multiple codegen units in one compilation LTO needs to merge them all together. Previously only upstream dependencies were merged and it was inherently relied on that there was only one local codegen unit. Supporting this involved refactoring the optimization backend architecture for rustc, namely splitting the `optimize_and_codegen` function into `optimize` and `codegen`. After an LLVM module has been optimized it may be blocked and queued up for LTO, and only after LTO are modules code generated. Non-LTO compilations should look the same as they do today backend-wise, we'll spin up a thread for each codegen unit and optimize/codegen in that thread. LTO compilations will, however, send the LLVM module back to the coordinator thread once optimizations have finished. When all LLVM modules have finished optimizing the coordinator will invoke the LTO backend, producing a further list of LLVM modules. Currently this is always a list of one LLVM module. The coordinator then spawns further work to run LTO and code generation passes over each module. In the course of this refactoring a number of other pieces were refactored: * Management of the bytecode encoding in rlibs was centralized into one module instead of being scattered across LTO and linking. * Some internal refactorings on the link stage of the compiler was done to work directly from `CompiledModule` structures instead of lists of paths. * The trans time-graph output was tweaked a little to include a name on each bar and inflate the size of the bars a little

bors · 2017-09-30T18:11:22Z

☀️ Test successful - status-appveyor, status-travis
Approved by: michaelwoerister
Pushing c6884b1 to master...

Mark-Simulacrum · 2017-09-30T18:27:53Z

This was actually merged.

rust-highfive assigned arielb1 Sep 23, 2017

rust-highfive assigned michaelwoerister and unassigned arielb1 Sep 23, 2017

estebank added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Sep 23, 2017

hanna-kruppe reviewed Sep 23, 2017

View reviewed changes

alexcrichton force-pushed the lto-codegen-units branch from 1077714 to 6ea1259 Compare September 23, 2017 16:08

kennytm reviewed Sep 25, 2017

View reviewed changes

michaelwoerister reviewed Sep 25, 2017

View reviewed changes

alexcrichton force-pushed the lto-codegen-units branch from 6ea1259 to bf3be56 Compare September 25, 2017 14:51

kennytm reviewed Sep 25, 2017

View reviewed changes

alexcrichton mentioned this pull request Sep 25, 2017

rustc: Implement ThinLTO #44841

Merged

alexcrichton force-pushed the lto-codegen-units branch from bf3be56 to 6154e04 Compare September 25, 2017 16:55

alexcrichton force-pushed the lto-codegen-units branch from 6154e04 to c3c6592 Compare September 25, 2017 22:51

alexcrichton force-pushed the lto-codegen-units branch from c3c6592 to 3547f2d Compare September 26, 2017 16:05

kennytm requested changes Sep 28, 2017

View reviewed changes

alexcrichton force-pushed the lto-codegen-units branch from 3547f2d to b377366 Compare September 28, 2017 13:55

alexcrichton force-pushed the lto-codegen-units branch from b377366 to c077360 Compare September 29, 2017 07:42

alexcrichton force-pushed the lto-codegen-units branch from c077360 to ded38db Compare September 30, 2017 07:22

Mark-Simulacrum closed this Sep 30, 2017

alexcrichton deleted the lto-codegen-units branch October 7, 2017 04:31

rustc: Enable LTO and multiple codegen units #44783

rustc: Enable LTO and multiple codegen units #44783

Conversation

alexcrichton commented Sep 23, 2017

rust-highfive commented Sep 23, 2017

alexcrichton commented Sep 23, 2017

estebank commented Sep 23, 2017

hanna-kruppe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelwoerister commented Sep 23, 2017

kennytm Sep 25, 2017 • edited Loading

Choose a reason for hiding this comment

michaelwoerister left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexcrichton commented Sep 25, 2017

bors commented Sep 25, 2017

Choose a reason for hiding this comment

alexcrichton commented Sep 25, 2017

alexcrichton commented Sep 25, 2017

bors commented Sep 25, 2017

bors commented Sep 25, 2017

alexcrichton commented Sep 25, 2017

bors commented Sep 25, 2017

kennytm commented Sep 26, 2017

alexcrichton commented Sep 26, 2017

alexcrichton commented Sep 26, 2017

bors commented Sep 26, 2017

bors commented Sep 28, 2017

bors commented Sep 28, 2017

kennytm Sep 28, 2017 • edited Loading

Choose a reason for hiding this comment

alexcrichton commented Sep 28, 2017

bors commented Sep 28, 2017

bors commented Sep 29, 2017

bors commented Sep 29, 2017

kennytm commented Sep 29, 2017

alexcrichton commented Sep 29, 2017

alexcrichton commented Sep 29, 2017

bors commented Sep 29, 2017

bors commented Sep 29, 2017

bors commented Sep 29, 2017

kennytm commented Sep 29, 2017

bors commented Sep 29, 2017

alexcrichton commented Sep 30, 2017

bors commented Sep 30, 2017

bors commented Sep 30, 2017

bors commented Sep 30, 2017

Mark-Simulacrum commented Sep 30, 2017

kennytm Sep 25, 2017 •

edited

Loading

kennytm Sep 28, 2017 •

edited

Loading