Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Undefined references for native library #1046

Closed
fpagliughi opened this issue Sep 29, 2017 · 9 comments
Closed

Undefined references for native library #1046

fpagliughi opened this issue Sep 29, 2017 · 9 comments
Labels

Comments

@fpagliughi
Copy link

I'm attempting to get the stock bindgen tutorial working from:
https://github.com/fitzgen/bindgen-tutorial-bzip2-sys

I have several Linux development machines running various flavors of Mint 17 & 18 (Ubuntu 14.04 and 16.04). The weird thing is that everything works fine on one machine, but on all the others machines, the linker is reporting undefined references in the native library whenever I try to create an executable.

This can be seen in an attempt to build and then run the tests for the tutorial:

$ git clone https://github.com/fitzgen/bindgen-tutorial-bzip2-sys.git
Cloning into 'bindgen-tutorial-bzip2-sys'...
remote: Counting objects: 28, done.
remote: Total 28 (delta 0), reused 0 (delta 0), pack-reused 28
Unpacking objects: 100% (28/28), done.
Checking connectivity... done.

$ cd bindgen-tutorial-bzip2-sys/
$ cargo build -j4
    Updating registry `https://github.com/rust-lang/crates.io-index`
 Downloading lazy_static v0.2.9
   Compiling void v1.0.2
   Compiling libloading v0.4.2
   Compiling vec_map v0.8.0
   Compiling log v0.3.8
   Compiling unicode-xid v0.0.4
   Compiling term v0.4.6
   Compiling bitflags v0.9.1
   Compiling glob v0.2.11
   Compiling rustc-serialize v0.3.24
   Compiling libc v0.2.31
   Compiling lazy_static v0.2.9
   Compiling peeking_take_while v0.1.2
   Compiling cfg-if v0.1.2
   Compiling strsim v0.6.0
   Compiling unicode-width v0.1.4
   Compiling regex-syntax v0.4.1
   Compiling utf8-ranges v1.0.0
   Compiling ansi_term v0.9.0
   Compiling bitflags v0.8.2
   Compiling unreachable v1.0.0
   Compiling term_size v0.3.0
   Compiling memchr v1.0.1
   Compiling atty v0.2.2
   Compiling clang-sys v0.19.0
   Compiling thread_local v0.3.4
   Compiling textwrap v0.8.0
   Compiling aho-corasick v0.6.3
   Compiling nom v3.2.0
   Compiling clap v2.26.2
   Compiling syntex_pos v0.58.1
   Compiling regex v0.2.2
   Compiling syntex_errors v0.58.1
   Compiling cexpr v0.2.2
   Compiling syntex_syntax v0.58.1
   Compiling env_logger v0.4.3
   Compiling aster v0.41.0
   Compiling quasi v0.32.0
   Compiling syntex v0.58.1
   Compiling quasi_codegen v0.32.0
   Compiling bindgen v0.26.3
   Compiling bindgen-tutorial-bzip2-sys v0.1.0 (file:///home/fmp/tmp/bindgen-tutorial-bzip2-sys)
    Finished dev [unoptimized + debuginfo] target(s) in 65.0 secs

$ cargo test
   Compiling bindgen-tutorial-bzip2-sys v0.1.0 (file:///home/fmp/tmp/bindgen-tutorial-bzip2-sys)
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/home/fmp/tmp/bindgen-tutorial-bzip2-sys/target/debug/deps/bindgen_tutorial_bzip2_sys-845fbd2219e8ea65.0.o" "-o" "/home/fmp/tmp/bindgen-tutorial-bzip2-sys/target/debug/deps/bindgen_tutorial_bzip2_sys-845fbd2219e8ea65" "/home/fmp/tmp/bindgen-tutorial-bzip2-sys/target/debug/deps/bindgen_tutorial_bzip2_sys-845fbd2219e8ea65.crate.allocator.o" "-Wl,--gc-sections" "-pie" "-nodefaultlibs" "-L" "/home/fmp/tmp/bindgen-tutorial-bzip2-sys/target/debug/deps" "-L" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "bz2" "-Wl,-Bstatic" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libtest-30d4d638be038598.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgetopts-c58b62d995849f6d.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libterm-359c3de19d4def40.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-29fb1fb52cf57377.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-9cbadc6554202be9.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_jemalloc-42efcbbdeb607d44.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-618c266cf9124966.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_system-3c10208cdd7e61cb.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-16f3b02b9a976b94.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-bc70b4efeaeb398c.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_unicode-a0ad42dc8f5856aa.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librand-32f648f7f7567c6c.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-21491ce3d14f1ef2.rlib" "/home/fmp/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-b9713bd7f605c0b2.rlib" "-Wl,-Bdynamic" "-l" "dl" "-l" "rt" "-l" "pthread" "-l" "pthread" "-l" "gcc_s" "-l" "c" "-l" "m" "-l" "rt" "-l" "pthread" "-l" "util"
  = note: /home/fmp/tmp/bindgen-tutorial-bzip2-sys/target/debug/deps/bindgen_tutorial_bzip2_sys-845fbd2219e8ea65.0.o: In function `bindgen_tutorial_bzip2_sys::tests::round_trip_compression_decompression':
          /home/fmp/tmp/bindgen-tutorial-bzip2-sys/src/lib.rs:21: undefined reference to `BZ2_bzCompressInit'
          /home/fmp/tmp/bindgen-tutorial-bzip2-sys/src/lib.rs:38: undefined reference to `BZ2_bzCompress'
          /home/fmp/tmp/bindgen-tutorial-bzip2-sys/src/lib.rs:49: undefined reference to `BZ2_bzCompressEnd'
          /home/fmp/tmp/bindgen-tutorial-bzip2-sys/src/lib.rs:58: undefined reference to `BZ2_bzDecompressInit'
          /home/fmp/tmp/bindgen-tutorial-bzip2-sys/src/lib.rs:74: undefined reference to `BZ2_bzDecompress'
          /home/fmp/tmp/bindgen-tutorial-bzip2-sys/src/lib.rs:86: undefined reference to `BZ2_bzDecompressEnd'
          clang: error: linker command failed with exit code 1 (use -v to see invocation)
          

error: aborting due to previous error

error: Could not compile `bindgen-tutorial-bzip2-sys`.

To learn more, run the command again with --verbose.

I can't figure out why it's working on one computer but none of the others.
Thanks

@fitzgen
Copy link
Member

fitzgen commented Sep 29, 2017

Is bzip2 installed on the system? And is the shared library on the system's usual search path for shared libraries?

Aside: @emilio do you think we should move the tutorial into this repo to test it for breakage in CI and keep it up to date and all that?

@fpagliughi
Copy link
Author

Yes, the dev package for bzip2 is installed via the package manager, apt-get.

$ apt list --installed | egrep '(bzip2|bz2)'
bzip2/xenial,now 1.0.6-8 amd64 [installed]
libbz2-1.0/xenial,now 1.0.6-8 amd64 [installed]
libbz2-dev/xenial,now 1.0.6-8 amd64 [installed]

and I can link to it with a C program:

$ clang -o bz2tst bz2tst.c -lbz2

@fitzgen fitzgen added the bug label Sep 29, 2017
@emilio
Copy link
Contributor

emilio commented Oct 3, 2017

Aside: @emilio do you think we should move the tutorial into this repo to test it for breakage in CI and keep it up to date and all that?

That sounds fine to me.

@fpagliughi
Copy link
Author

It appears that this may be due to the #[link_name = "..."] attributes which are being specified for each of the native functions in the bindings.rs file, like:

extern "C" {
    #[link_name = "_Z18BZ2_bzCompressInit"]
    pub fn BZ2_bzCompressInit(strm: *mut bz_stream,
                              blockSize100k: ::std::os::raw::c_int,
                              verbosity: ::std::os::raw::c_int,
                              workFactor: ::std::os::raw::c_int)
     -> ::std::os::raw::c_int;
}

If I manually remove them or comment them all out, then the tests seem to link and run properly, like:

extern "C" {
    //#[link_name = "_Z18BZ2_bzCompressInit"]
    pub fn BZ2_bzCompressInit(strm: *mut bz_stream,
                              blockSize100k: ::std::os::raw::c_int,
                              verbosity: ::std::os::raw::c_int,
                              workFactor: ::std::os::raw::c_int)
     -> ::std::os::raw::c_int;
}

Why is it mangling the names like that?
Is it required by some other compiler or operating system?
Is there a way to turn it off, or conditionally specify it on targets where it's necessary?

Thanks

@lilianmoraru
Copy link

lilianmoraru commented Nov 15, 2017

Yep, same issue here, also because of wrong link_name.
The interesting thing is that at home it works, at work it doesn't...
Maybe has to do with the version of clang that I use(and the one that Rust uses)?
With bindgen(I've written the bindings by hand before, and it works just fine, but for maintainability I ported the code to bindgen) it was completely failing to compile(not finding stddef.h) when I did not have clang installed.

At home I have clang-6.0, at work clang-4.0(the one I installed after it was failing to compile because of the stddef.h header).

@fitzgen
Copy link
Member

fitzgen commented Nov 15, 2017

Maybe has to do with the version of clang that I use(and the one that Rust uses)?

Rust doesn't use libclang (although it is using LLVM) -- I think that is a red herring.

bindgen is definitely affected by what version of libclang is being used, however. Right now, we test with 3.8, 3.9, and 4.0. 5.0 has been known to work well in the past, but no one has taken the time to set up our CI with it yet. Last I heard 6.0 was still trunk, has that changed?

The other thing to double check is that you have the LIBCLANG_PATH env var properly set, to be sure that clang-sys is using the correct libclang.

@fpagliughi
Copy link
Author

OK. It looks like this might be the root cause of the issue.

All of my Linux machines have multiple versions of clang, including 3.6, 3.8, and 3.9. Even the machine that "works" has v3.6, but for some reason it... works.

The clang-sys README states that it supports Cargo features like "clang_3_5", "clang_3_6", and so on up to "clang_5_0" to force it to us the specified version or newer, and that:

If you do not enable one of these features, the API provided by libclang 3.5 will be available by default.

So perhaps it is (somewhat inconsistently?) seeking the oldest libclang that it can find?

I tried to see if I could set the Cargo features, but had no luck. Admittedly I have no idea what I'm doing.

But if I set the LIBCLANG_PATH to v3.9, that appears to work, like:

LIBCLANG_PATH=/usr/lib/llvm-3.9/lib cargo build

@fpagliughi
Copy link
Author

Yes, it appears that the old clang version is causing problems, although somewhat inconsistently which made it difficult to diagnose. This issue appears to be a repeat of #528, and possibly a few others.

Setting LIBCLANG_PATH seems the correct fix, but this places a burden on users with a silent failure when things go wrong.

There is even a work-around already in place for older clang's in the bindgen::Builder to fix this: trust_clang_mangling(false) as in:

let bindings = bindgen::Builder::default()
    .trust_clang_mangling(false)
    .header("wrapper.h")
    .generate()
    .expect("Unable to generate bindings");

I hope this is harmless if we're wrapping a straight C library, which shouldn't require mangling?

I guess this issue can be closed, but it might be nice for bindgen to issue a warning on build if it detects a clang library that is unsupported.

Thanks for the help and this awesome tool.

@fitzgen
Copy link
Member

fitzgen commented Nov 27, 2017

libclang versioning is tricky, but we have hopes of eventually side-stepping the issue: #918

I guess this issue can be closed, but it might be nice for bindgen to issue a warning on build if it detects a clang library that is unsupported.

We do have a warning, but only in the executable, not in the library: https://github.com/rust-lang-nursery/rust-bindgen/blob/master/src/main.rs#L46

Would love to receive a PR that moves it somewhere shared by both the executable and library.

@fitzgen fitzgen closed this as completed Nov 27, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants