Linker error on a crates that link with system frameworks and add library search paths

[kornelski]

On OS X rustdoc somehow ends up linking the tested crate with ImageIO.framework and libjpeg, even if the crate doesn't link with any library.

If the tested crate's build script merely sets a search path (cargo:rustc-link-search=native=/usr/local/lib) to a path contains a version of libjpeg, rustdoc will link with it. This is problematic, because rustdoc also links with OS X's ImageIO.framework which can't coexist with non-Apple libjpeg in the same program.

$ cargo test --doc

Finished debug [unoptimized + debuginfo] target(s) in 0.0 secs
Doc-tests testing
dyld: Symbol not found: __cg_jpeg_resync_to_restart
Referenced from: /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
Expected in: /usr/local/lib/libJPEG.dylib
in /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
error: Process didn't exit successfully: rustdoc --test /private/tmp/testing/src/lib.rs --crate-name testing -L dependency=/private/tmp/testing/target/debug/deps -L native=/usr/local/lib --extern testing=/private/tmp/testing/target/debug/deps/libtesting.rlib (signal: 5, SIGTRAP: trace/breakpoint trap)

What makes even more bizarre is that if I run the failing command myself from terminal, it works:

$ rustdoc --test /private/tmp/testing/src/lib.rs --crate-name testing -L dependency=/private/tmp/testing/target/debug/deps -L native=/usr/local/lib --extern testing=/private/tmp/testing/target/debug/deps/libtesting.rlib

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured

rustc 1.13.0-nightly (497d67d 2016-09-01)

To reproduce:

brew install libjpeg && brew link libjpeg
cargo new testing

add build.rs with

fn main() {
    println!("cargo:rustc-link-search=native=/usr/local/lib");
}

cargo test --doc

[Luthaf]

I investigated this a bit, and it look like this is not a rustdoc bug (as running the command from a separated process works just fine).

It looks like this is caused by rustup on my machine. Setting RUSTC and RUSTDOC to the full path to the binaries (~/.multirust/toolchains/nightly-x86_64-apple-darwin/bin/rust{c,doc}) make the error go away.

The good new here is that using the latest master of rustup also fixes the error! (Maybe due to rust-lang/rustup@b851f7c ?)

Ping @brson for rustup

[casey]

I ran into this error myself while linking git2-rs and glium, and originally reported it here: https://github.com/alexcrichton/git2-rs/issues/175

I created a repo with the bare minimum needed to trigger the error here:

https://github.com/casey/rust-dynamic-linker-error

Note that in the above repo, the symbol and library in the error message are different, but I believe the cause to be the same.

What's happening is that the build script is emitting cargo:rustc-link-search=native=/opt/local/lib, which causes cargo to run rustc with -L native=/opt/local/lib.

On my system, /opt/local is a macports installation, and I happen to have libiconv installed via macports.

I have an extern block in main.rs that causes the binary to link against the ApplicationServices framework.

When the binary runs, the loader tries to dynamically link the ApplicationServices framework, which eventually pulls in the LanguageModeling framework, which tries to link against libiconv.dylib. Normally, it would link against /usr/lib/libiconv.dylib, however it finds /opt/local/lib/libiconv.dylib and tries to link that, but errors out with a missing symbol since it's the wrong version.

Most of the specifics here aren't super important. This could happen with macports or homebrew, and any number of frameworks which link against system-supplied dynamic libraries that happen to be found in an alternate location.

Unfortunately I'm not sure what exactly fix is here. macOS frameworks expect to link dynamic libraries in /usr/lib, and in some cases against dynamic libraries shipped inside the framework itself. For example, ImageIO wants to link against /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib.

[casey]

@alexcrichton might have some insight here

[casey]

Also, I'm using rustup myself, but I'm not sure if it's related. I'm on 0.6.5, which has b851f7c

[alexcrichton]

Unfortunately I may not have many insights beyond I believe your summary is accurate @casey. This isn't really a rust-lang/rust bug AFAIK though, it's just an unfortunate interaction with openssl-sys and libgit2-sys

[casey]

Are you sure it isn't a rust bug? Wanting to link a binary with a dynamic library in a nonstandard location and a framework in a standard location seems like a reasonable thing to do.

Does rust have its own linker or does it use an external linker? If it uses an external linker, is there a way to get the command line that rustc passes to that linker?

Perhaps there's a way of specifying search path precedence in such a way that it will work.

[casey]

The plot thickens!

I was using cargo run to run the binary, which produced an error:

$ cargo run
   Compiling rust-dynamic-linker-error v0.0.0 (file:///Users/rodarmor/src/rust-dynamic-linker-error)
    Finished debug [unoptimized + debuginfo] target(s) in 0.17 secs
     Running `target/debug/rust-dynamic-linker-error`
dyld: Symbol not found: _iconv
  Referenced from: /System/Library/PrivateFrameworks/LanguageModeling.framework/Versions/A/LanguageModeling
  Expected in: /opt/local/lib/libiconv.2.dylib
 in /System/Library/PrivateFrameworks/LanguageModeling.framework/Versions/A/LanguageModeling

However, I happened to try to run the binary directly, and it worked:

$ ./target/debug/rust-dynamic-linker-error
Hello, world!

When I inspected the environment variables that the binary was inheriting when run with cargo run, the problem became clear.

Cargo is setting DYLD_LIBRARY_PATH=/opt/local/lib:/Users/rodarmor/src/rust-dynamic-linker-error/target/debug:/Users/rodarmor/src/rust-dynamic-linker-error/target/debug/deps:/Users/rodarmor/.multirust/toolchains/nightly-x86_64-apple-darwin/lib ./target/debug/rust-dynamic-linker-error

The /opt/local/lib at the beginning is what's causing the problem, and causing system frameworks to load the wrong dynamic libraries at runtime.

In fact, I was able to trigger the same error in a C program compiled with clang by exporting the same variable:

$ DYLD_LIBRARY_PATH=/opt/local/lib ./main
Symbol not found: _iconv
  Referenced from: /System/Library/PrivateFrameworks/LanguageModeling.framework/Versions/A/LanguageModeling
  Expected in: /opt/local/lib/libiconv.2.dylib
 in /System/Library/PrivateFrameworks/LanguageModeling.framework/Versions/A/LanguageModeling
zsh: trace trap  DYLD_LIBRARY_PATH=/opt/local/lib ./main

I'm not sure why cargo is exporting this variable, since it isn't necessary.

So, fortunately it looks like this one has an easy fix, just stop exporting this variable. If it would cause libraries to not be found, we can look at exporting DYLD_FALLBACK_LIBRARY_PATH, which shouldn't cause these problems.

I'll open an issue with cargo. I think this issue can be closed.

[casey]

It looks like perhaps there already is an issue: rust-lang/cargo#2888

Edit: Actually, that issue looks a little too general, so I opened rust-lang/cargo#3366

[alexcrichton]

@casey currently we shell out to gcc (which in turn uses ld) as a linker on OSX at least with a similar strategy on other systems. It's true yeah that we could perhaps solve this with some sort of prioritization or adding different libraries in different places, though.

[Lord-Kamina]

I've said this in a bunch of different issues already in both here and the cargo repo. The root cause of the problem is DYLD_LIBRARY_PATH; people assume it is equivalent to LD_LIBRARY_PATH but it isn't. DYLD_LIBRARY_PATH should never be used because it's a hack and overrides default system search paths. Instead, you should use DYLD_FALLBACK_LIBRARY_PATH as that only searches anything if a suitable library is not found in the default paths.

_{Sent with GitHawk}

[jyn514]

Is this still relevant? It sounds like it was never a rustdoc bug in the first place, is that true?