rustc: Prepare to enable ThinLTO by default #46382

alexcrichton · 2017-11-29T21:52:32Z

This commit almost enables ThinLTO and multiple codegen units in release mode by
default but is blocked on #46346 now before pulling the trigger.

rust-highfive · 2017-11-29T21:52:36Z

r? @arielb1

(rust_highfive has picked a reviewer for you, use r? to override)

alexcrichton · 2017-11-29T21:52:46Z

r? @michaelwoerister

cc @rust-lang/compiler

sfackler · 2017-11-29T22:12:56Z

Is this a blocker? #46346

alexcrichton · 2017-11-29T22:16:45Z

Are we sure that's related to ThinLTO? I've tested everything locally and wasn't able to reproduce myself.

alexcrichton · 2017-11-29T22:58:15Z

Urgh it may very well be. I swear to god.

kennytm · 2017-11-29T23:25:06Z

src/rustllvm/PassWrapper.cpp

+  // summaries in the index, and we basically just only want to ensure that dead
+  // symbols are internalized. Otherwise everything that's already external
+  // linkage will stay as external, and internal will stay as internal.
+  std::set<GlobalValue::GUID> ExportedGUIDs;


Why not use a std::unordered_set here.

cuviper · 2017-11-30T02:26:17Z

I see that LLVMRustThinLTOAvailable gates on just LLVM_VERSION_GE(4, 0). How well are you expecting this to work when using an external LLVM-4.0?
(IOW, are there rust-lang/llvm patches that I definitely need here?)

alexcrichton · 2017-11-30T05:06:14Z

@cuviper IIRC rust-lang/llvm@e45c75d is the main (and possibly only) ThinLTO related patch (although it fixes a preexisting bug).

In general though I lose track over time what patch was for what on our fork, although everything that matters is upstream in some form so it's just a matter of time until we update to get it.

cuviper · 2017-11-30T05:11:09Z

Ok, thanks, we already grabbed that one. I'll look back over the branch again later myself. I do appreciate that everything is fixed upstream first!

michaelwoerister · 2017-11-30T09:52:04Z

Awesome! Will review shortly. Am I reading this right, this is still blocked on #46346?

alexcrichton · 2017-11-30T13:29:05Z

@michaelwoerister unfortunately yes :(

michaelwoerister

The PR looks good to me. Let's hope it doesn't bit rot too much until #46346 is resolved.

Can you explain to me why you don't use addPreservedGUID anymore?

michaelwoerister · 2017-11-30T14:23:01Z

src/librustc/session/mod.rs

+    }
+
+    /// Returns whether ThinLTO is enabled for this compilation
+    /// compilation


compilation compilation

michaelwoerister · 2017-11-30T14:25:31Z

src/libstd/sys_common/backtrace.rs

-    // First validate the symbol. If it doesn't look like anything we're
+pub fn demangle(writer: &mut Write, mut s: &str, format: PrintFormat) -> io::Result<()> {
+    // During ThinLTO LLVM may import and rename internal symbols, so strip out
+    // those endings first as they're on of the last manglings applied to symbol


michaelwoerister · 2017-11-30T14:36:02Z

src/rustllvm/PassWrapper.cpp

  for (int i = 0; i < num_symbols; i++) {
-    addPreservedGUID(Ret->Index,
-                     Ret->GUIDPreservedSymbols,
-                     GlobalValue::getGUID(preserved_symbols[i]));


Was this too conservative before?

Previously we were too eagerly exporting almost all symbols used in ThinLTO which can cause a whole host of problems downstream! This commit instead fixes this error by aligning more closely with `lib/LTO/LTO.cpp` in LLVM's codebase which is to only change the linkage of summaries which are computed as dead. Closes rust-lang#46374

Helps to avoid hitting path limits on Windows

This commit prepares to enable ThinLTO and multiple codegen units in release mode by default. We've still got a debuginfo bug or two to sort out before actually turning it on by default.

alexcrichton · 2017-11-30T15:20:48Z

@michaelwoerister ok I've updated the listing of commits and addressed comments. The commits now to not enable ThinLTO by default but instead retain retain today's default. The repo is left in a state though to make it a one line change to turn ThinLTO on by default in the future.

To answer your question about GUIDPreservedSymbols though there's more info on #46374 but it boils down to two things:

In the first location where GUIDPreservedSymbols was used the logic of addPreservedGUID wasn't necessary. The same result is computed whether we call addPreservedGUID or not.
In the second location it was incorrect to call addPreservedGUID in the sense that it added the transitive closure of symbols to that set. That in turn caused ThinLTO exposes too many symbols #46374 where too many symbols were exported (aka all of them) rather than just the set we wanted. By avoiding addPreservedGUID and tweaking the definition of isExported this bug was fixed though.

michaelwoerister · 2017-11-30T16:18:06Z

Thanks for the explanation!

@bors r+

bors · 2017-11-30T16:18:09Z

📌 Commit 855f6d1 has been approved by michaelwoerister

bors · 2017-12-02T22:27:30Z

⌛ Testing commit 855f6d1 with merge 088f328...

rustc: Prepare to enable ThinLTO by default This commit *almost* enables ThinLTO and multiple codegen units in release mode by default but is blocked on #46346 now before pulling the trigger.

bors · 2017-12-03T00:54:47Z

☀️ Test successful - status-appveyor, status-travis
Approved by: michaelwoerister
Pushing 088f328 to master...

In rust-lang#46382 the logic around linkage preservation with ThinLTO ws tweaked but the loop that registered all otherwise exported GUID values as "don't internalize me please" was erroneously too conservative and only asking "external" linkage items to not be internalized. Instead we actually want the inversion of that condition, everything *without* "local" linkage to be internalized. This commit updates the condition there, adds a test, and... Closes rust-lang#46543

rustc: Further tweak linkage in ThinLTO In #46382 the logic around linkage preservation with ThinLTO ws tweaked but the loop that registered all otherwise exported GUID values as "don't internalize me please" was erroneously too conservative and only asking "external" linkage items to not be internalized. Instead we actually want the inversion of that condition, everything *without* "local" linkage to be internalized. This commit updates the condition there, adds a test, and... Closes #46543

rust-highfive assigned arielb1 Nov 29, 2017

rust-highfive assigned michaelwoerister and unassigned arielb1 Nov 29, 2017

kennytm added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Nov 29, 2017

kennytm reviewed Nov 29, 2017

View reviewed changes

michaelwoerister reviewed Nov 30, 2017

View reviewed changes

alexcrichton added 3 commits November 30, 2017 07:14

Rename long test names to be a bit shorter

7df4683

Helps to avoid hitting path limits on Windows

rustc: Prepare to enable ThinLTO by default

855f6d1

This commit prepares to enable ThinLTO and multiple codegen units in release mode by default. We've still got a debuginfo bug or two to sort out before actually turning it on by default.

alexcrichton force-pushed the thinlto-default branch from e527fa6 to 855f6d1 Compare November 30, 2017 15:18

alexcrichton changed the title ~~rustc: Enable ThinLTO by default~~ rustc: Prepare to enable ThinLTO by default Nov 30, 2017

kennytm added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 30, 2017

bors merged commit 855f6d1 into rust-lang:master Dec 3, 2017

alexcrichton mentioned this pull request Dec 7, 2017

rustc: Further tweak linkage in ThinLTO #46549

Merged

alexcrichton mentioned this pull request Dec 16, 2017

Tracking issue for enabling multiple CGUs in release mode by default #45320

Closed

11 tasks

pnkfelix mentioned this pull request Jan 29, 2018

heisenbug: debug builds Integer::repr_discr has transient ICE on compile-fail/enum-discrim-too-small2.rs #47381

Closed

alexcrichton mentioned this pull request Feb 1, 2018

LTO: 10x regression #47866

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rustc: Prepare to enable ThinLTO by default #46382

rustc: Prepare to enable ThinLTO by default #46382

alexcrichton commented Nov 29, 2017 •

edited

Loading

rust-highfive commented Nov 29, 2017

alexcrichton commented Nov 29, 2017

sfackler commented Nov 29, 2017

alexcrichton commented Nov 29, 2017

alexcrichton commented Nov 29, 2017

kennytm Nov 29, 2017

cuviper commented Nov 30, 2017

alexcrichton commented Nov 30, 2017

cuviper commented Nov 30, 2017

michaelwoerister commented Nov 30, 2017

alexcrichton commented Nov 30, 2017

michaelwoerister left a comment

michaelwoerister Nov 30, 2017

michaelwoerister Nov 30, 2017

michaelwoerister Nov 30, 2017

alexcrichton commented Nov 30, 2017

michaelwoerister commented Nov 30, 2017

bors commented Nov 30, 2017

bors commented Dec 2, 2017

bors commented Dec 3, 2017

rustc: Prepare to enable ThinLTO by default #46382

rustc: Prepare to enable ThinLTO by default #46382

Conversation

alexcrichton commented Nov 29, 2017 • edited Loading

rust-highfive commented Nov 29, 2017

alexcrichton commented Nov 29, 2017

sfackler commented Nov 29, 2017

alexcrichton commented Nov 29, 2017

alexcrichton commented Nov 29, 2017

kennytm Nov 29, 2017

Choose a reason for hiding this comment

cuviper commented Nov 30, 2017

alexcrichton commented Nov 30, 2017

cuviper commented Nov 30, 2017

michaelwoerister commented Nov 30, 2017

alexcrichton commented Nov 30, 2017

michaelwoerister left a comment

Choose a reason for hiding this comment

michaelwoerister Nov 30, 2017

Choose a reason for hiding this comment

michaelwoerister Nov 30, 2017

Choose a reason for hiding this comment

michaelwoerister Nov 30, 2017

Choose a reason for hiding this comment

alexcrichton commented Nov 30, 2017

michaelwoerister commented Nov 30, 2017

bors commented Nov 30, 2017

bors commented Dec 2, 2017

bors commented Dec 3, 2017

alexcrichton commented Nov 29, 2017 •

edited

Loading