Linker error in 1.4

[dancooke]

I've just upgraded in 1.4 with Homebrew and I'm getting the following linker error:

Undefined symbols for architecture x86_64:
  "_BZ2_bzBuffToBuffCompress", referenced from:
      _cram_compress_by_method in libhts.a(cram_io.o)
  "_BZ2_bzBuffToBuffDecompress", referenced from:
      _cram_uncompress_block in libhts.a(cram_io.o)
  "_lzma_code", referenced from:
      _cram_uncompress_block in libhts.a(cram_io.o)
  "_lzma_easy_buffer_encode", referenced from:
      _cram_compress_by_method in libhts.a(cram_io.o)
  "_lzma_easy_decoder_memusage", referenced from:
      _cram_uncompress_block in libhts.a(cram_io.o)
  "_lzma_end", referenced from:
      _cram_uncompress_block in libhts.a(cram_io.o)
  "_lzma_stream_buffer_bound", referenced from:
      _cram_compress_by_method in libhts.a(cram_io.o)
  "_lzma_stream_decoder", referenced from:
      _cram_uncompress_block in libhts.a(cram_io.o)
ld: symbol(s) not found for architecture x86_64
clang-4.0: error: linker command failed with exit code 1 (use -v to see invocation)

Compiled with Clang 4.0.

Dan

[mp15]

If I had to guess I would say homebrew have either not added -lbz2 and -llzma to their cmake file or got the linking order wrong. I would suggest you report this to Homebrew in the first instance as their CMake is not of our making.

…

On 20 Mar 2017, at 19:56, Daniel Cooke ***@***.***> wrote: I've just upgraded in 1.4 with Homebrew and I'm now getting the following linker error: Undefined symbols for architecture x86_64: "_BZ2_bzBuffToBuffCompress", referenced from: _cram_compress_by_method in libhts.a(cram_io.o) "_BZ2_bzBuffToBuffDecompress", referenced from: _cram_uncompress_block in libhts.a(cram_io.o) "_lzma_code", referenced from: _cram_uncompress_block in libhts.a(cram_io.o) "_lzma_easy_buffer_encode", referenced from: _cram_compress_by_method in libhts.a(cram_io.o) "_lzma_easy_decoder_memusage", referenced from: _cram_uncompress_block in libhts.a(cram_io.o) "_lzma_end", referenced from: _cram_uncompress_block in libhts.a(cram_io.o) "_lzma_stream_buffer_bound", referenced from: _cram_compress_by_method in libhts.a(cram_io.o) "_lzma_stream_decoder", referenced from: _cram_uncompress_block in libhts.a(cram_io.o) ld: symbol(s) not found for architecture x86_64 clang-4.0: error: linker command failed with exit code 1 (use -v to see invocation) make[2]: *** [src/octopus] Error 1 make[1]: *** [src/CMakeFiles/octopus.dir/all] Error 2 make: *** [all] Error 2 Compiled with Clang 4.0. Dan — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

-- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

[dancooke]

I've reported this to Homebrew, but I've also tried building myself from the latest Github develop branch and I'm still getting linker errors:

Undefined symbols for architecture x86_64:
  "_BZ2_bzBuffToBuffCompress", referenced from:
      _cram_compress_block in libhts.a(cram_io.o)
      _cram_compress_by_method in libhts.a(cram_io.o)
  "_BZ2_bzBuffToBuffDecompress", referenced from:
      _cram_uncompress_block in libhts.a(cram_io.o)
  "_curl_easy_cleanup", referenced from:
      _libcurl_open in libhts.a(hfile_libcurl.o)
      _libcurl_close in libhts.a(hfile_libcurl.o)
  "_curl_easy_getinfo", referenced from:
      _easy_errno in libhts.a(hfile_libcurl.o)
      _libcurl_open in libhts.a(hfile_libcurl.o)
      _wait_perform in libhts.a(hfile_libcurl.o)
  "_curl_easy_init", referenced from:
      _libcurl_open in libhts.a(hfile_libcurl.o)
  "_curl_easy_pause", referenced from:
      _libcurl_read in libhts.a(hfile_libcurl.o)
      _libcurl_write in libhts.a(hfile_libcurl.o)
      _libcurl_seek in libhts.a(hfile_libcurl.o)
      _libcurl_close in libhts.a(hfile_libcurl.o)
  "_curl_easy_setopt", referenced from:
      _libcurl_open in libhts.a(hfile_libcurl.o)
      _libcurl_seek in libhts.a(hfile_libcurl.o)
  "_curl_global_cleanup", referenced from:
      _hfile_plugin_init_libcurl in libhts.a(hfile_libcurl.o)
      _libcurl_exit in libhts.a(hfile_libcurl.o)
  "_curl_global_init", referenced from:
      _hfile_plugin_init_libcurl in libhts.a(hfile_libcurl.o)
  "_curl_multi_add_handle", referenced from:
      _libcurl_open in libhts.a(hfile_libcurl.o)
      _libcurl_seek in libhts.a(hfile_libcurl.o)
  "_curl_multi_cleanup", referenced from:
      _libcurl_exit in libhts.a(hfile_libcurl.o)
  "_curl_multi_fdset", referenced from:
      _wait_perform in libhts.a(hfile_libcurl.o)
  "_curl_multi_info_read", referenced from:
      _wait_perform in libhts.a(hfile_libcurl.o)
  "_curl_multi_init", referenced from:
      _hfile_plugin_init_libcurl in libhts.a(hfile_libcurl.o)
  "_curl_multi_perform", referenced from:
      _wait_perform in libhts.a(hfile_libcurl.o)
  "_curl_multi_remove_handle", referenced from:
      _libcurl_open in libhts.a(hfile_libcurl.o)
      _libcurl_seek in libhts.a(hfile_libcurl.o)
      _libcurl_close in libhts.a(hfile_libcurl.o)
  "_curl_multi_timeout", referenced from:
      _wait_perform in libhts.a(hfile_libcurl.o)
  "_curl_slist_append", referenced from:
      _libcurl_open in libhts.a(hfile_libcurl.o)
      _parse_va_list in libhts.a(hfile_libcurl.o)
  "_curl_slist_free_all", referenced from:
      _vhopen_libcurl in libhts.a(hfile_libcurl.o)
      _libcurl_open in libhts.a(hfile_libcurl.o)
  "_curl_version_info", referenced from:
      _hfile_plugin_init_libcurl in libhts.a(hfile_libcurl.o)
  "_lzma_code", referenced from:
      _cram_uncompress_block in libhts.a(cram_io.o)
  "_lzma_easy_buffer_encode", referenced from:
      _cram_compress_block in libhts.a(cram_io.o)
      _cram_compress_by_method in libhts.a(cram_io.o)
  "_lzma_easy_decoder_memusage", referenced from:
      _cram_uncompress_block in libhts.a(cram_io.o)
  "_lzma_end", referenced from:
      _cram_uncompress_block in libhts.a(cram_io.o)
  "_lzma_stream_buffer_bound", referenced from:
      _cram_compress_block in libhts.a(cram_io.o)
      _cram_compress_by_method in libhts.a(cram_io.o)
  "_lzma_stream_decoder", referenced from:
      _cram_uncompress_block in libhts.a(cram_io.o)
ld: symbol(s) not found for architecture x86_64
clang-4.0: error: linker command failed with exit code 1 (use -v to see invocation)

[jkbonfield]

What commands are you typing for compilation? Is this straight make, or with configure first?

The missing symbols here are almost certainly due to missing dependencies. The INSTALL file lists the dependencies needed and they should be checked in the configure script too, but not the default Makefile.

[dancooke]

I used the commands given in the README (i.e. with configure first). I can successfully link to 1.3.2 (using either Homebrew or manual install).

[daviesrob]

What output did you get from configure? It should complain if it can't link to libcurl or libbz2.

[dancooke]

Ah, it does indeed complain about being unable to find liblzma. I do however already have xz installed (version 5.2.3).

[dancooke]

Also, Homebrew does seem to be aware of this requirement, on attempting to uninstall xz:

Error: Refusing to uninstall /usr/local/Cellar/xz/5.2.3
because it is required by htslib 1.4, python3 3.6.0, python3 3.6.0_1, r 3.3.3, which are currently installed.

[daviesrob]

This could be due to stale configuration files. Try make distclean and rebuilding the configure script with autoreconf -i. Then run configure again to see if it behaves.

If you're using the version from homebrew, apparantly with CMake files in it, then try instead using a clean checkout from github (or the release tarball) instead. It's likely that CMake has replaced htslib's Makefile with one that won't work, even if you try to reconfigure.

[dancooke]

Still no luck, here is the full output:

dhcp565:htslib dcooke$ ./configure
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for ranlib... ranlib
checking for pkg-config... /usr/local/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking shared library type... Darwin dylib
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for stdlib.h... (cached) yes
checking for unistd.h... (cached) yes
checking for sys/param.h... yes
checking for getpagesize... yes
checking for working mmap... yes
checking for gmtime_r... yes
checking whether fdatasync is declared... no
checking for library containing log... none required
checking for zlib.h... yes
checking for inflate in -lz... yes
checking for bzlib.h... yes
checking for BZ2_bzBuffToBuffCompress in -lbz2... yes
checking for lzma.h... no
checking for lzma_easy_buffer_encode in -llzma... yes
configure: error: liblzma development files not found

The CRAM format may use LZMA2 compression, which is implemented in HTSlib
by using compression routines from liblzma <http://tukaani.org/xz/>.

Building HTSlib requires liblzma development files to be installed on the
build machine; you may need to ensure a package such as liblzma-dev (on Debian
or Ubuntu Linux), xz-devel (on RPM-based Linux distributions or Cygwin), or
xz (via Homebrew on macOS) is installed; or build XZ Utils from source.

Either configure with --disable-lzma (which will make some CRAM files
produced elsewhere unreadable) or resolve this error to build HTSlib.

And to sanity check the library is installed:

dhcp565:~ dcooke$ find /usr/local/include/lzma.*
/usr/local/include/lzma.h

dhcp565:lib dcooke$ find /usr/local/lib/liblzma.*
/usr/local/lib/liblzma.5.dylib
/usr/local/lib/liblzma.a
/usr/local/lib/liblzma.dylib

[jkbonfield]

Our official stance on this is that if your include files are installed somewhere other than the default compiler path then you have to manually modify the compiler flags at configure/make time. Eg ./configure CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/lib".

However I think this is pretty distgusting personally and my own preference was for --with-lzma=DIR type syntax, as well as having a more intelligent search path that looked in more places including "standard" non-system localities such as /usr/local. Maybe we should reconsider the original PR - eg see https://github.com/samtools/htslib/pull/331/files#diff-3928dcc0b7a862396236aae91c495bffR89

[daviesrob]

This is very odd, as configure found the lzma library, but not the header file. As @jkbonfield mentioned, you can use CPPFLAGS and LDFLAGS to override the default search path, but this should not be necessary as /usr/local/include and /usr/local/lib should both be on the default path for gcc anyway.

/usr/local/include/lzma.h will be a symbolic link pointing to somewhere like ../Cellar/xz/5.2.3/include/lzma.h. You should check that the linked to file is present and readable.

You might want to try brew doctor to check that your homebrew installation is OK.

You can check the include search path for gcc using gcc -x c -v -E /dev/null. You should see something like this (along with other information):

#include "..." search starts here:
#include <...> search starts here:
 /usr/llvm-gcc-4.2/bin/../lib/gcc/i686-apple-darwin11/4.2.1/include
 /usr/local/include
 /Applications/Xcode.app/Contents/Developer/usr/llvm-gcc-4.2/lib/gcc/i686-apple-darwin11/4.2.1/include
 /usr/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)
End of search list.

Check that /usr/local/include does appear in the search path.

gcc -Wl,-v /dev/null should print out the library search path:

Library search paths:
	/usr/llvm-gcc-4.2/lib/gcc/i686-apple-darwin11/4.2.1/x86_64
	/Applications/Xcode.app/Contents/Developer/usr/llvm-gcc-4.2/lib/gcc/i686-apple-darwin11/4.2.1/x86_64
	/usr/llvm-gcc-4.2/lib/gcc/i686-apple-darwin11/4.2.1
	/usr/llvm-gcc-4.2/lib/gcc
	/Applications/Xcode.app/Contents/Developer/usr/llvm-gcc-4.2/lib/gcc/i686-apple-darwin11/4.2.1
	/usr/llvm-gcc-4.2/lib
	/Applications/Xcode.app/Contents/Developer/usr/llvm-gcc-4.2/lib
	/usr/lib
	/usr/local/lib
Framework search paths:
	/Library/Frameworks/
	/System/Library/Frameworks/

Check that /usr/local/lib appears somewhere in there.

You may also want to look for lzma-related files in directories above /usr/local in the search lists, in case it's finding a conflicting version somewhere.

You should also look for lzma in config.log to see exactly why the test is failing. It may be for a completely unrelated reason. When the test works, you should see something like this:

configure:4218: checking for lzma.h
configure:4218: gcc -c -g -O2  conftest.c >&5
configure:4218: $? = 0
configure:4218: result: yes
configure:4227: checking for lzma_easy_buffer_encode in -llzma
configure:4252: gcc -o conftest -g -O2   conftest.c -llzma  -lbz2 -lz  >&5
configure:4252: $? = 0
configure:4261: result: yes

When it doesn't, you usually get a lot more output.

[dancooke]

The symlinked header is where it's supposed to be and is readable:

ls -l /usr/local/include/lzma.h 
lrwxr-xr-x  1 dcooke  admin  33 27 Jan 14:24 /usr/local/include/lzma.h -> ../Cellar/xz/5.2.3/include/lzma.h

ls -l /usr/local/Cellar/xz/5.2.3/include/lzma.h 
-rw-r--r--  1 dcooke  admin  9737 30 Dec 11:34 /usr/local/Cellar/xz/5.2.3/include/lzma.h

Also, nothing reported by brew doctor. For some reason there were some standard search paths missing from my bundled Apple Clang (which is what gcc defaults to on OS X). Following the advice found here (running xcode-select --install) resolves this. I've also checked the search paths are present on my Hombrew Clang:

gcc -x c -v -E /dev/null

#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/8.0.0/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
 /usr/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)
End of search list.


gcc -Wl,-v /dev/null

Library search paths:
	/usr/lib
	/usr/local/lib
Framework search paths:
	/Library/Frameworks/
	/System/Library/Frameworks/

/usr/local/opt/llvm/bin/clang -x c -v -E /dev/null

#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /usr/local/Cellar/llvm/4.0.0/bin/../lib/clang/4.0.0/include
 /usr/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)
End of search list.

/usr/local/opt/llvm/bin/clang -Wl,-v /dev/null

Library search paths:
	/usr/lib
	/usr/local/lib
Framework search paths:
	/Library/Frameworks/
	/System/Library/Frameworks/

So I guess that explains why./configure complained previously (it now completes). Unfortunately this hasn't resolved the issue; both Homebrew's htslib, and my manual install have the same link errors as above (including the additional curl link errors from my manual install). I've tried compiling with OS X bundled Apple Clang and with a Hombrew installed Clang. Both give the same errors.

There are no other versions of lzma above /usr/local:

find /usr -name "lzma"
/usr/local/bin/lzma
/usr/local/Cellar/xz/5.2.3/bin/lzma
/usr/local/Cellar/xz/5.2.3/include/lzma
/usr/local/include/lzma

[daviesrob]

I'd ignore the homebrew version and concentrate on the Github develop branch. git status should tell you if it's intact, particularly the Makefile.

Run make distclean followed by ./configure.

It should have made htslib.pc.tmp, config.mk, and config.h (plus some other bits and pieces).
htslib.pc.tmp should contain something like this (static_libs may also mention libcurl):

includedir=@-includedir@
libdir=@-libdir@

# Flags and libraries needed when linking against a static libhts.a
# (used by manual and semi-manual pkg-config(1)-style enquiries).
static_ldflags=
static_libs=-lz -lm -lbz2 -llzma

Name: htslib
Description: C library for high-throughput sequencing data formats
Version: @-PACKAGE_VERSION@
Cflags: -I${includedir}
Libs: -L${libdir} -lhts
Libs.private: -L${libdir}  -lbz2 -lhts -lm -lpthread

Config.mk should include these lines (possibly also with libcurl):

CPPFLAGS = 
CFLAGS   = -g -O2
LDFLAGS  = 
LIBS     = -llzma -lbz2 -lz 

If that's the case then make on its own ought to work. If it still doesn't then we'll need to see the whole make output so we can see exactly what it's trying to run when it goes wrong.

[dancooke]

Resolved! The FindHTSlib.cmake I was naively using wasn't looking for the new htslib dependencies. The Homebrew htslib version doesn't use curl, hence the extra errors on my build. I've attached an updated FindHTSlib.cmake with the new dependencies (minus OpenSSL) in case anyone else runs into this. Many thanks for all your help!!

FindHTSlib.cmake.zip

[ejewett]

Hi Dan,

Thanks for figuring that out. For those of us with the same problem who are not so technically inclined, can you explain where we should put the FindHTSlib.cmake script you created and how we should reference it when running make?

Thanks!

[dancooke]

Hi @ejewett, sorry for the delayed response! I usually but CMake modules in build/cmake/modules. Then in your top level CMakeLists.txt script add set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_SOURCE_DIR}/build/cmake/modules/"), Then just use find_package as normal (e.g find_package (HTSlib 1.4 REQUIRED)). Hope that helps!