LLVM upgrade #34743

badboy · Jul 9, 2016

As discussed in https://internals.rust-lang.org/t/need-help-with-emscripten-port/3154/46 I'm trying to update the used LLVM checkout in Rust.

I basically took @shepmaster's code and applied it on top (though I did the commits manually, the original commits have better descriptions.

With these changes I was able to build rustc. make check throws one last error on run-pass/issue-28950.rs. Output: https://gist.github.com/badboy/bcdd3bbde260860b6159aa49070a9052

I took the metadata changes as is and they seem to work, though it now uses the module in another step. I'm not sure if this is the best and correct way.

Things to do:

~~Make run-pass/issue-28950.rs pass~~ unrelated
Find out how the PositionIndependentExecutable setting is now used
Is the llvm::legacy still the right way to do these things?

cc @brson @alexcrichton

rust-highfive · Jul 9, 2016

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @pnkfelix (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

badboy · Jul 9, 2016

Ah, forgot to mention that I am using a LLVM checkout of http://llvm.org/git/llvm.git

frewsxcv · Jul 9, 2016

frewsxcv reviewed Jul 9, 2016

View changes

frewsxcv · Jul 9, 2016

src/rustllvm/PassWrapper.cpp

@@ -267,7 +267,7 @@ LLVMRustAddLibraryInfo(LLVMPassManagerRef PMB,
 // similar code in clang's BackendUtil.cpp file.
 extern "C" void
 LLVMRustRunFunctionPassManager(LLVMPassManagerRef PM, LLVMModuleRef M) {
-    FunctionPassManager *P = unwrap<FunctionPassManager>(PM);
+    llvm::legacy::FunctionPassManager *P = unwrap<llvm::legacy::FunctionPassManager>(PM);


llvm::legacy::FunctionPassManager

Tangential comment to this pull request: Does anyone know what this got replaced with since it's presumably legacy now?

LLVM pass manager infrastructure is currently getting rewritten to be more flexible. See https://github.com/llvm-mirror/llvm/blob/d1aa4ea020d932ffc6a3c3e4d1bbab659391c725/tools/opt/NewPMDriver.cpp#L50 for a usage example for the new API. That said, the rewrite isn't complete yet.

Looks like clang is also still using this, so we probably don't need to update just yet until they do at least.

shepmaster · Jul 9, 2016

shepmaster reviewed Jul 9, 2016

View changes

shepmaster · Jul 9, 2016

src/rustllvm/PassWrapper.cpp

@@ -188,7 +188,7 @@ LLVMRustCreateTargetMachine(const char *triple,
    }

    TargetOptions Options;
-    Options.PositionIndependentExecutable = PositionIndependentExecutable;
+    //Options.PositionIndependentExecutable = PositionIndependentExecutable;


I'm pretty sure this is a hack that only makes sense in my fork (there's no PIE for microcontrollers :-)).

Even if we wanted to remove this (doubtful), it should be removed and not commented-out. I believe that LLVM applies this flag at a more granular level now, so looking downward for another flag to apply it to seems reasonable.

Right, this is one of the things mentioned above. I left it in, so I remember to fix that.

This was deleted here which appears to be replaced by this which is a new setPIELevel function on the LLVM module itself.

The LLVM module isn't available at this time so we'll just have to arrange for it to be set later. Perhaps something like this though:

Leave this block, but gated by #if LLVM_VERSION_MINOR <= 8 for compatibility with older versions.

Add LLVMRustSetModulePIELevel as a function which is a noop on pre-LLVM-3.9 and afterwards calls the setPIELevel function on the LLVM module.

Did you mean we have a LLVMRustSetModulePIELevel with the #ifdef for calling the setPIELevel method?
If so, at what point do we call it?

If so, at what point do we call it?

I'd believe you want to call it every time an LLVM module is created.

shepmaster · Jul 9, 2016

shepmaster reviewed Jul 9, 2016

View changes

shepmaster · Jul 9, 2016

src/librustc_trans/debuginfo/mod.rs

@@ -168,7 +168,7 @@ pub fn finalize(cx: &CrateContext) {
    }

    debug!("finalize");
-    let _ = compile_unit_metadata(cx);
+    //let _ = compile_unit_metadata(cx);


Shouldn't commit commented-out code. This appears to be moved to the constructor, so I believe it's safe and correct to just remove this.

Right. This will be removed.

Although thinking of what @alexcrichton was saying, it might have to just be placed in an #if guard to maintain compatibility with earlier LLVM...

Yeah we may want this for compatibility with older llvm versions, but I'm not sure if the fix here is backwards compatible (it may be?)

It will definitely need to change, as we changed the function.
However, the new version might even work with old LLVM, doesn't it?

I think maybe yeah? I'm not actually sure what changed here, it looks like just a bit of shuffling around to me...

It's indeed possible that the updated code (that creates the metadata early) will work with older LLVMs. Specifically, LLVM inverted the way the debug information graph is built. All the builders inside LLVM magically handle this, as long as there is something to add the debug information to. Creating the debug information as early as possible enables this.

shepmaster · Jul 9, 2016

shepmaster reviewed Jul 9, 2016

View changes

shepmaster · Jul 9, 2016

src/rustllvm/PassWrapper.cpp

@@ -359,7 +359,17 @@ LLVMRustAddAlwaysInlinePass(LLVMPassManagerBuilderRef PMB, bool AddLifetimes) {
 extern "C" void
 LLVMRustRunRestrictionPass(LLVMModuleRef M, char **symbols, size_t len) {
    llvm::legacy::PassManager passes;
-    passes.add(llvm::createInternalizePass());
+
+    auto PreserveFunctions = [=](const GlobalValue &GV) {


How did this work before? symbols wasn't even being used...

It was used in the old version, see 0763041#diff-2030f049f8df2454e3720ea9403bc14aL363.

I should have squashed that.

shepmaster · Jul 9, 2016

I'm no core contributor, but I don't see the value in commit X introducing less-than-great variable names and then commit X+1 improving them. I'd just squash those two together.

badboy · Jul 9, 2016

@shepmaster yes, I should have squashed them beforehand. I can still do this.

alexcrichton · Jul 10, 2016

alexcrichton reviewed Jul 10, 2016

View changes

alexcrichton · Jul 10, 2016

src/librustc_trans/debuginfo/metadata.rs

-    let work_dir = &cx.sess().working_dir;
-    let compile_unit_name = match cx.sess().local_crate_source_file {
-        None => fallback_path(cx),
+pub fn compile_unit_metadata(scc: &::context::SharedCrateContext, debug_context: &CrateDebugContext, sess: &::session::Session) -> DIDescriptor {


Right now our line limit is 100 characters so I think this will trigger make tidy, could you also add imports instead of using absolute paths here?

alexcrichton · Jul 10, 2016

alexcrichton reviewed Jul 10, 2016

View changes

alexcrichton · Jul 10, 2016

src/rustllvm/PassWrapper.cpp

+        return false;
+    };
+
+    passes.add(llvm::createInternalizePass(PreserveFunctions));


Can you wrap this in a #if to only execute this new part on LLVM version >= 9 and keep the old code around for 8 and below? That should help us keep compiling against older versions of LLVM.

Come to think of that, can you also ensure that the llvm::legacy:: above compiles on LLVM 3.7 and 3.8?

alexcrichton · Jul 10, 2016

alexcrichton reviewed Jul 10, 2016

View changes

alexcrichton · Jul 10, 2016

src/llvm

@@ -1 +1 @@
-Subproject commit 80ad955b60b3ac02d0462a4a65fcea597d0ebfb1
+Subproject commit c90294dc24393d973ed18bbc8bc87b73cb67e484


This is currently the rust-llvm-2016-03-13 branch in the rust-lang/llvm repo, we'll want to make a new branch which points to this LLVM version and also has some of our custom patches, which includes.

I've done this at the rust-lang-2016-07-09 branch as well as including some of our own patches (mostly just related to compiling LLVM itself). Can you update this commit to that patch?

alexcrichton · Jul 10, 2016

A failure on issue-28950 looks like it may be specific to your system or something like that, I doubt that the LLVM changes would cause it to fail. Are you sure that it succeeds before this PR on your local computer?

badboy · Jul 10, 2016

@alexcrichton Indeed, with a fresh checkout of rust-lang/rust I see the same error on issue-28950

alexbool · Jul 11, 2016

This is supposed to fix #34119 ?

badboy · Jul 11, 2016

@alexbool If it is fixed by something in LLVM upstream, then yes

brson · Jul 11, 2016

cc @kripken Here's Rust's LLVM upgrade.

kripken · Jul 11, 2016

@brson: cool, emscripten's merges of that same LLVM commit are in https://github.com/kripken/emscripten-fastcomp/tree/next-merge and https://github.com/kripken/emscripten-fastcomp-clang/tree/next-merge .

The tracking issue is emscripten-core/emscripten#4430. Looks like only some SIMD issues block it, which probably wouldn't be a problem for rust, so it could be usable for testing already.

shepmaster · Jul 11, 2016

I'd strongly encourage having links to the original LLVM issues / commits that are forcing these changes. These links should be in the commit message that modifies the relevant aspect of the code. That should make it much easier for people in the future to track down the why.

shepmaster · Jul 11, 2016

That should make it much easier for people in the future to track down the why.

I say this having been one of those people to track down such information in the past.

alexcrichton · Jul 11, 2016

@shepmaster to clarify, you mean the changes to LLVM bindings and why they're changing? (e.g. this)

shepmaster · Jul 11, 2016

you mean the changes to LLVM bindings and why they're changing

Yeah; sorry to be unclear. Having the commit message be something like

[LLVM 3.9] Configure PIE at the module level instead of compilation unit level

This was deleted here[linky] which appears to be replaced by this[linky] which is a new setPIELevel function on the LLVM module itself.

would make that commit 💯 in my book. Otherwise the commit is like "moving some functions around!"

Again, I stress that I hold no power to enforce these suggestions; I just would have found commits like this to be useful when I was digging into the existing code to make these kind of changes. 🌴

alexcrichton · Jul 12, 2016

Sounds like a good plan to me!

badboy · Jul 12, 2016

I will rebase the commits and change accordingly (force-push incoming!)

@shepmaster Thanks for mentioning this, it's indeed a good idea, especially in the long run

solson · Jul 13, 2016

@alexcrichton @eddyb Can this PR also be placed under the "Launch MIR into Orbit" milestone, since it's blocking #34119?

alexcrichton · Jul 13, 2016

@solson sure!

badboy · Jul 14, 2016

I just force-pushed with the new commits

Reorganized all commits into logical blocks, including proper descriptions and links
I had to upgrade compiler-rt as well to get it to compile, this is not yet reflected here
I had to re-enable the std::thread thing in LLVM to get it to compile at least once. We probably need to find a proper workaround first.
When choosing anything other than Default in ffc5a8608f1418f6a52f754232ba5edb90427370 I get the following error on compile:

error: linking with `cc` failed: exit code: 1
note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/home/jer/src/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/home/jer/src/rust/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/debug/deps/std-b8d6c224bdd7e5ff.0.o" "/home/jer/src/rust/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/debug/deps/std-b8d6c224bdd7e5ff.1.o" "/home/jer/src/rust/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/debug/deps/std-b8d6c224bdd7e5ff.2.o" "/home/jer/src/rust/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/debug/deps/std-b8d6c224bdd7e5ff.3.o" "-o" "/home/jer/src/rust/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/debug/deps/libstd-b8d6c224bdd7e5ff.so" "/home/jer/src/rust/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/debug/deps/std-b8d6c224bdd7e5ff.metadata.o" "-nodefaultlibs" "-L" "/home/jer/src/rust/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/debug/deps" "-L" "/home/jer/src/rust/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/debug/deps" "-L" "/home/jer/src/rust/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/debug/build/std-26167b16044fc6c3/out/.libs" "-L" "/home/jer/src/rust/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/debug/build/alloc_jemalloc-0986af1f8480d6ae/out/lib" "-L" "/home/jer/src/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "-Wl,--whole-archive" "-l" "backtrace" "-Wl,--no-whole-archive" "-Wl,-Bdynamic" "-l" "dl" "-l" "rt" "-l" "pthread" "-Wl,--whole-archive" "/tmp/rustc.FqfLoqjnVH7z/libcollections-af9a9d510058b25a.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.FqfLoqjnVH7z/libpanic_unwind-17c382f05cff5951.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.FqfLoqjnVH7z/liballoc-3e8adf33e113f98f.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.FqfLoqjnVH7z/liballoc_jemalloc-565df9f061d73123.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.FqfLoqjnVH7z/librustc_unicode-f4dd78d2f1b9c500.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.FqfLoqjnVH7z/libunwind-f70560ece060458e.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.FqfLoqjnVH7z/liblibc-b8c0e2a7b980b655.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.FqfLoqjnVH7z/librand-f0667cf9d06dbb0e.rlib" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "/tmp/rustc.FqfLoqjnVH7z/libcore-d7d9994298ae504c.rlib" "-Wl,--no-whole-archive" "-l" "pthread" "-l" "gcc_s" "-l" "c" "-l" "m" "-l" "rt" "-l" "util" "-shared" "-Wl,-rpath,$ORIGIN/../lib" "-l" "compiler-rt"
note: /usr/bin/ld: /home/jer/src/rust/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/debug/deps/std-b8d6c224bdd7e5ff.0.o: relocation R_X86_64_TPOFF32 against `_ZN3std11collections4hash3map11RandomState3new4KEYS7__getit5__KEY17hc8a582c36b9a9c3eE' can not be used when making a shared object; recompile with -fPIC
/home/jer/src/rust/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/debug/deps/std-b8d6c224bdd7e5ff.0.o: error adding symbols: Bad value
collect2: error: ld returned 1 exit status

error: aborting due to previous error
error: Could not compile `std`.

It seems something is not compiled correctly with -fPIC, but I'm not sure what to do here.

eddyb · Jul 31, 2016

@tmiasko That's interesting. Do you have a backtrace? We couldn't get anything, reproduction seemed to need the concurrent builds, or a specific builder, to trigger.

alexcrichton · Aug 1, 2016

@tmiasko did some digging and discovered that https://reviews.llvm.org/D22858 may be relevant (stack trace gathered). I've updated our LLVM submodule with this commit (rebased on the llvm/release_39 branch) and now it's rust-lang/llvm@d1cc489. @badboy wanna try updating to that and give it a go? Builds in dev are running, but we can do in parallel hopefully :)

badboy · Aug 1, 2016

Done! Upgraded to d1cc489

eddyb · Aug 1, 2016

@bors r+

bors · Aug 1, 2016

📌 Commit 5d1d247 has been approved by eddyb

bors · Aug 1, 2016

⌛️ Testing commit 5d1d247 with merge 2c1612c...

bors · Aug 1, 2016

badboy · Aug 1, 2016

Wuhu! 🎉

eminence · Aug 1, 2016

Outstanding work everyone!

michaelwoerister · Aug 1, 2016

Epic! :)

shepmaster · Aug 1, 2016

Hurrah! Now it's time to rebase the AVR and Emscripten forks and really give it a workout ;-)

badboy · Aug 1, 2016

@shepmaster: That's the next step and I'll try to work on it this week

bstrie · Aug 1, 2016

Heroes, every one of you! On with MIR!

whitequark · Aug 13, 2016

Looks like this missed the LLVM version check in configure, not sure why it didn't break bots:

--- .../rust/configure  Sat Aug 13 01:03:47 2016
+++ .../rust/configure  Sat Aug 13 01:03:50 2016
@@ -1013,7 +1013,7 @@
     LLVM_VERSION=$($LLVM_CONFIG --version)

     case $LLVM_VERSION in
-        (3.[7-8]*)
+        (3.[7-9]*)
             msg "found ok version of LLVM: $LLVM_VERSION"
             ;;
         (*)

shepmaster · Aug 13, 2016

I have a PR out for that - #35594. No idea about the bots.

rust-highfive assigned pnkfelix Jul 9, 2016

frewsxcv referenced this pull request Jul 9, 2016
Closed
[MIR] Stage1 `libcore` crashes with SIGILL on ARM #34427

alexcrichton added this to the Launch MIR into Orbit milestone Jul 13, 2016

badboy force-pushed the badboy:llvm-upgrade branch from 8e955d2 to ffc5a86 Jul 14, 2016

Upgrade LLVM once more to get a bugfix
@tmiasko did some digging and discovered that https://reviews.llvm.org/D22858 may be relevant.

Loading status checks…

5d1d247

This was referenced Aug 1, 2016

Closed

rustc/LLVM DiagnosticKind mismatch causes crash when bootstrapping unoptimized w/ MIR. #35131

Merged

Rollup of 8 pull requests #35163

bors merged commit 5d1d247 into rust-lang:master Aug 1, 2016
2 checks passed

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

homu Test successful
Details

This was referenced Aug 1, 2016

Merged

Add help for target CPUs, features, relocation and code models. #34845

Merged

resolve: diagnostics improvement and groundwork for RFC 1560 #35116

spladug added a commit to spladug/rust that referenced this pull request Aug 17, 2016

Update minimum CMake version in README
The minimum got bumped in the LLVM upgrade of rust-lang#34743.

4254b31

spladug referenced this pull request Aug 17, 2016
Merged
Update minimum CMake version in README #35742

eddyb added a commit to eddyb/rust that referenced this pull request Aug 17, 2016

Rollup merge of rust-lang#35742 - spladug:readme-cmake-version, r=nik…
…omatsakis Update minimum CMake version in README The minimum got bumped in the LLVM upgrade of rust-lang#34743.

4fe6e11

eddyb added a commit to eddyb/rust that referenced this pull request Aug 18, 2016

Rollup merge of rust-lang#35742 - spladug:readme-cmake-version, r=nik…
…omatsakis Update minimum CMake version in README The minimum got bumped in the LLVM upgrade of rust-lang#34743.

7a27444

TimNN referenced this pull request Nov 13, 2016
Closed
panic in program that shouldn't panic when compiling at optimization level 1 or 2 #37747

japaric referenced this pull request Jan 23, 2017
Closed
regression: UB with `extern fn() -> !` and LTO #39253

Mark-Simulacrum referenced this pull request Jun 23, 2017
Open
Prevent "Target-incompatible DataLayout" LLVM asserts. #33446

rust-lang/rust

Join GitHub today

LLVM upgrade #34743

Conversation

badboy commented Jul 9, 2016 • edited

rust-highfive assigned pnkfelix Jul 9, 2016

This comment has been minimized.

rust-highfive commented Jul 9, 2016

This comment has been minimized.

badboy commented Jul 9, 2016

frewsxcv reviewed Jul 9, 2016 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

frewsxcv referenced this pull request Jul 9, 2016

shepmaster reviewed Jul 9, 2016 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

shepmaster reviewed Jul 9, 2016 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

shepmaster reviewed Jul 9, 2016 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

shepmaster commented Jul 9, 2016

This comment has been minimized.

badboy commented Jul 9, 2016

alexcrichton reviewed Jul 10, 2016 View changes

This comment has been minimized.

alexcrichton reviewed Jul 10, 2016 View changes

This comment has been minimized.

alexcrichton reviewed Jul 10, 2016 View changes

This comment has been minimized.

This comment has been minimized.

alexcrichton commented Jul 10, 2016

This comment has been minimized.

badboy commented Jul 10, 2016

This comment has been minimized.

alexbool commented Jul 11, 2016

This comment has been minimized.

badboy commented Jul 11, 2016

This comment has been minimized.

brson commented Jul 11, 2016

This comment has been minimized.

kripken commented Jul 11, 2016

This comment has been minimized.

shepmaster commented Jul 11, 2016

This comment has been minimized.

shepmaster commented Jul 11, 2016

This comment has been minimized.

alexcrichton commented Jul 11, 2016

This comment has been minimized.

shepmaster commented Jul 11, 2016

This comment has been minimized.

alexcrichton commented Jul 12, 2016

This comment has been minimized.

badboy commented Jul 12, 2016 • edited

This comment has been minimized.

solson commented Jul 13, 2016

alexcrichton added this to the Launch MIR into Orbit milestone Jul 13, 2016

This comment has been minimized.

alexcrichton commented Jul 13, 2016

badboy force-pushed the badboy:llvm-upgrade branch from 8e955d2 to ffc5a86 Jul 14, 2016

This comment has been minimized.

badboy commented Jul 14, 2016

This comment has been minimized.

eddyb commented Jul 31, 2016

This comment has been minimized.

alexcrichton commented Aug 1, 2016

This comment has been minimized.

badboy commented Aug 1, 2016

badboy commented Jul 9, 2016 •

edited

frewsxcv reviewed Jul 9, 2016

View changes

shepmaster reviewed Jul 9, 2016

View changes

shepmaster reviewed Jul 9, 2016

View changes

shepmaster reviewed Jul 9, 2016

View changes

alexcrichton reviewed Jul 10, 2016

View changes

alexcrichton reviewed Jul 10, 2016

View changes

alexcrichton reviewed Jul 10, 2016

View changes

badboy commented Jul 12, 2016 •

edited

badboy force-pushed the badboy:llvm-upgrade branch from `8e955d2` to `ffc5a86` Jul 14, 2016

bors merged commit `5d1d247` into rust-lang:master Aug 1, 2016
2 checks passed