Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

-Clinker-plugin-lto doesn't work without extra manual work #60059

Open
glandium opened this issue Apr 17, 2019 · 10 comments
Open

-Clinker-plugin-lto doesn't work without extra manual work #60059

glandium opened this issue Apr 17, 2019 · 10 comments

Comments

@glandium
Copy link
Contributor

@glandium glandium commented Apr 17, 2019

$ cargo new testcase
$ cd testcase
$ RUSTFLAGS="-Clinker-plugin-lto" cargo run --release

Yields the following on mac:

 
   Compiling testcase v0.1.0 (/Users/glandium/testcase)
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-m64" "-Wl,-plugin-opt=O3" "-Wl,-plugin-opt=mcpu=core2" "-L" "/Users/glandium/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "/Users/glandium/testcase/target/release/deps/testcase-d97266bc083c0e9e.testcase.4ank1dht-cgu.0.rcgu.o" "/Users/glandium/testcase/target/release/deps/testcase-d97266bc083c0e9e.testcase.4ank1dht-cgu.1.rcgu.o" "-o" "/Users/glandium/testcase/target/release/deps/testcase-d97266bc083c0e9e" "/Users/glandium/testcase/target/release/deps/testcase-d97266bc083c0e9e.3pvtkuuqiy7p6wge.rcgu.o" "-Wl,-dead_strip" "-nodefaultlibs" "-L" "/Users/glandium/testcase/target/release/deps" "-L" "/Users/glandium/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "/Users/glandium/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libstd-64d1544b9dc8a8d7.rlib" "/Users/glandium/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libpanic_unwind-47702365139f147e.rlib" "/Users/glandium/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libbacktrace_sys-0aefa3a2bfa44649.rlib" "/Users/glandium/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libunwind-c2b22c88cacffeb6.rlib" "/Users/glandium/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/librustc_demangle-fca4484aa9be2d09.rlib" "/Users/glandium/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/liblibc-4728c64ee20d89f8.rlib" "/Users/glandium/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/liballoc-1722fbf72ce989c9.rlib" "/Users/glandium/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_core-0836ff3f3d6a6ee6.rlib" "/Users/glandium/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcore-91c9fbc323ad09b7.rlib" "/Users/glandium/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcompiler_builtins-43b96ba2cdcc7cb3.rlib" "-lSystem" "-lresolv" "-lc" "-lm"
  = note: ld: unknown option: -plugin-opt=O3
          clang: error: linker command failed with exit code 1 (use -v to see invocation)
          

error: aborting due to previous error

error: Could not compile `testcase`.

To learn more, run the command again with --verbose.

On mac, the default linker is ld64. rustc really invokes cc, which invokes ld64 with the flags passed with -Wl and some others depending on the other command line arguments. ld64 does support the LLVM plugin... but doesn't support the -plugin-opt option to pass arguments to it. I know some things can be passed with -Wl,-mllvm,... but I don't know if that includes things that rust is trying to pass here.

On Linux, it's funnier:

   Compiling testcase v0.1.0 (/tmp/testcase)
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-Wl,-plugin-opt=O3" "-Wl,-plugin-opt=mcpu=x86-64" "-L" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/tmp/testcase/target/release/deps/testcase-576934ab8201a4ea.testcase.dl00diw9-cgu.0.rcgu.o" "/tmp/testcase/target/release/deps/testcase-576934ab8201a4ea.testcase.dl00diw9-cgu.1.rcgu.o" "-o" "/tmp/testcase/target/release/deps/testcase-576934ab8201a4ea" "/tmp/testcase/target/release/deps/testcase-576934ab8201a4ea.y29oqvxl224jt9k.rcgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs" "-L" "/tmp/testcase/target/release/deps" "-L" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,--start-group" "-Wl,-Bstatic" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-bbd8cb236ab3b537.rlib" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-334e405e4bdf1791.rlib" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libbacktrace_sys-1e14a089a9f63178.rlib" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-f1aae4818bd13556.rlib" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-a32c94e7da1105b4.rlib" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-e214e2acd110aec9.rlib" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-fbf429991e30afee.rlib" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-1734308ff05fb551.rlib" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-b349c8b817f959a5.rlib" "-Wl,--end-group" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-c4b4b16c70e666d9.rlib" "-Wl,-Bdynamic" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"
  = note: /tmp/testcase/target/release/deps/testcase-576934ab8201a4ea.testcase.dl00diw9-cgu.0.rcgu.o: file not recognized: file format not recognized
          collect2: error: ld returned 1 exit status
          

error: aborting due to previous error

error: Could not compile `testcase`.

To learn more, run the command again with --verbose.

Because cc is gcc, this just plain doesn't work, for no obvious reason. At the very least, it seems rust should try to use clang instead of cc in that case.

But that also fails:

RUSTFLAGS="-Clinker=clang-8 -Clinker-plugin-lto" cargo run --release
   Compiling testcase v0.1.0 (/tmp/testcase)
error: linking with `clang-8` failed: exit code: 1
  |
  = note: "clang-8" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-Wl,-plugin-opt=O3" "-Wl,-plugin-opt=mcpu=x86-64" "-L" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/tmp/testcase/target/release/deps/testcase-e3dc53d119c25549.testcase.ci8upasg-cgu.0.rcgu.o" "/tmp/testcase/target/release/deps/testcase-e3dc53d119c25549.testcase.ci8upasg-cgu.1.rcgu.o" "-o" "/tmp/testcase/target/release/deps/testcase-e3dc53d119c25549" "/tmp/testcase/target/release/deps/testcase-e3dc53d119c25549.61wo3nlknhbro8d.rcgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs" "-L" "/tmp/testcase/target/release/deps" "-L" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,--start-group" "-Wl,-Bstatic" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-bbd8cb236ab3b537.rlib" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-334e405e4bdf1791.rlib" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libbacktrace_sys-1e14a089a9f63178.rlib" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-f1aae4818bd13556.rlib" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-a32c94e7da1105b4.rlib" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-e214e2acd110aec9.rlib" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-fbf429991e30afee.rlib" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-1734308ff05fb551.rlib" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-b349c8b817f959a5.rlib" "-Wl,--end-group" "/home/glandium/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-c4b4b16c70e666d9.rlib" "-Wl,-Bdynamic" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"
  = note: /usr/bin/ld: bad -plugin-opt option
          clang: error: linker command failed with exit code 1 (use -v to see invocation)
          

error: aborting due to previous error

error: Could not compile `testcase`.

To learn more, run the command again with --verbose.

And here, the reason is essentially the same: the underlying linker doesn't support the -plugin-opt flag... except it does, but not when it's not passed -plugin, which happens when the compiler passes it, which happens when -flto was on its command line:

$ RUSTFLAGS="-Clinker=clang-8 -Clinker-plugin-lto -Clink-arg=-flto" cargo run --release   Compiling testcase v0.1.0 (/tmp/testcase)
    Finished release [optimized] target(s) in 0.28s
     Running `target/release/testcase`
Hello, world!

Using lld works too, because it doesn't need an explicit -plugin:

$ RUSTFLAGS="-Clinker=clang-8 -Clinker-plugin-lto -Clink-arg=-fuse-ld=lld" cargo run --release
   Compiling testcase v0.1.0 (/tmp/testcase)
    Finished release [optimized] target(s) in 0.36s
     Running `target/release/testcase`
Hello, world!
@froydnj
Copy link
Contributor

@froydnj froydnj commented Apr 18, 2019

FWIW, you can pass an argument to -Clinker-plugin-lto to give it the name of the plugin to load, which would fix the clang case. (You could make an argument that said requirement is exactly the problem in the clang case.) The OS X and plain cc cases seem like out-and-out bugs, though.

@glandium
Copy link
Contributor Author

@glandium glandium commented Apr 18, 2019

Indeed RUSTFLAGS="-Clinker=clang-8 -Clinker-plugin-lto=/usr/lib/llvm-8/lib/LLVMgold.so" cargo run --release works. It actually even works without -Clinker=clang-8.

@nikomatsakis
Copy link
Contributor

@nikomatsakis nikomatsakis commented Apr 23, 2019

Nominating for (hopefully brief) discussion in the @rust-lang/compiler meeting -- this is holding up Firefox's efforts to use ThinLTO to eliminate glue code. It would be good to get it fixed or, at minimum, to specify what the correct behavior is and figure out who to ping about it (I'm not that familiar with these options).

@glandium
Copy link
Contributor Author

@glandium glandium commented Apr 23, 2019

FWIW, we have a workaround for Firefox, which is that rust is not involved in linking at all. This is more about the general case than Firefox itself.

@nagisa
Copy link
Member

@nagisa nagisa commented Apr 25, 2019

It seems that this issue would be mostly resolved by documenting which cases and linker/flag combinations we know to work. We could then see if there are any changes to the CLI that could be made to improve the experience here.

@nikomatsakis
Copy link
Contributor

@nikomatsakis nikomatsakis commented Apr 25, 2019

@dwightguth
Copy link

@dwightguth dwightguth commented May 15, 2019

I recently ran into this issue and wanted to mention that if you just want to have lto'd libraries that you are using to link an executable with clang, but can't build the rust code you need because crates that build binaries fail to link due to this error on mac OS, you can sidestep the issue by using the following script on the .rlib files generated by cargo build: https://github.com/kframework/llvm-backend/blob/6ec0694fe63f5c552ea3b64022ff8ba197d1252d/bin/llvm-kompile-rust-lto

It will not work if you want to link those libraries with rustc though (although it could be made to work with a small additional effort)

@michaelwoerister
Copy link
Member

@michaelwoerister michaelwoerister commented Aug 27, 2019

Because cc is gcc, this just plain doesn't work, for no obvious reason. At the very least, it seems rust should try to use clang instead of cc in that case.

Should it though? I'm unclear as to what our policy is here? As a user it would make me a bit uneasy if a program like rustc would silently decide to use a different linker. Things would be different if we shipped our own LLVM linker plugin and passed that on to cc.

@vext01
Copy link
Contributor

@vext01 vext01 commented Apr 20, 2021

Just wanted to mention that it's still quite difficult and unobvious to get a linker LTO plugin working with rustc.

I had expected to just pass -C linker_plugin_lto, but as others have noted, it's not that easy.

In case someone else is trying to get this working (including my future self), here's what I had to do (on Linux).

I couldn't see a way to get rustbuild to install all of the necessary llvm-related tools (namely lld and clang), so I ended up building my own. You can build them all directly from the llvm git repo. Once cloned I checked out the llvmorg-12.0.0 release tag and configured the build like this:

$ cmake -DCMAKE_INSTALL_PREFIX=/opt/llvm-12.0.0 \
    -DLLVM_INSTALL_UTILS=On \
    -DCMAKE_BUILD_TYPE=release \
    -DLLVM_ENABLE_PROJECTS="lld;clang" \
    ../llvm

After building and installing llvm, put its bin/ dir in your $PATH.

Then I built a stage 1 rust with this in config.toml:

[target.x86_64-unknown-linux-gnu]
llvm-config = "/opt/llvm-12.0.0/bin/llvm-config"

I added the resulting toolchain to rustup, and built a rust project like this:

$ RUSTFLAGS="-C linker_plugin_lto -C linker=clang -Clink-arg=-fuse-ld=lld" cargo +rust-stage1 build --release

(Note that rust, lld and clang must all be using the same version of LLVM IR. You can't mix tools built with different llvm versions)

It took me a few hours to figure all of this out, so I'm quite keen to either make this easier, or at least document better how to do it. Any thoughts?

@vext01
Copy link
Contributor

@vext01 vext01 commented Apr 20, 2021

I've just learned that I should also have used a recent branch on rust's fork of LLVM, but the above instructions are otherwise still valid.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants