linking against htslib 1.8 that was built with libdeflate causes samtools to fail configure

[moonwatcher]

So building htslib 1.8 into a clean root with libdeflate yields a functioning htslib (which is indeed btw preforming much nicer than zlib although not really improving multithreading).

However attempting to build samtools against that root folder with --with-htslib fails:

configure:3267: checking for htslib/sam.h
configure:3267: gcc -c -g -O2  -I/home/lg1883/.pheniqs/pheniqs-trunk/install/include conftest.c >&5
configure:3267: $? = 0
configure:3267: result: yes
configure:3270: checking for hts_version in -lhts
configure:3295: gcc -o conftest -g -O2  -I/home/lg1883/.pheniqs/pheniqs-trunk/install/include  -L/home/lg1883/.pheniqs/pheniqs-trunk/install/lib conftest.c -lhts   >&5
/usr/bin/ld: warning: libdeflate.so, needed by /home/lg1883/.pheniqs/pheniqs-trunk/install/lib/libhts.so, not found (try using -rpath or -rpath-link)
/home/lg1883/.pheniqs/pheniqs-trunk/install/lib/libhts.so: undefined reference to `libdeflate_free_decompressor'
/home/lg1883/.pheniqs/pheniqs-trunk/install/lib/libhts.so: undefined reference to `libdeflate_deflate_compress'
/home/lg1883/.pheniqs/pheniqs-trunk/install/lib/libhts.so: undefined reference to `libdeflate_deflate_decompress'
/home/lg1883/.pheniqs/pheniqs-trunk/install/lib/libhts.so: undefined reference to `libdeflate_alloc_compressor'
/home/lg1883/.pheniqs/pheniqs-trunk/install/lib/libhts.so: undefined reference to `libdeflate_alloc_decompressor'
/home/lg1883/.pheniqs/pheniqs-trunk/install/lib/libhts.so: undefined reference to `libdeflate_crc32'
/home/lg1883/.pheniqs/pheniqs-trunk/install/lib/libhts.so: undefined reference to `libdeflate_free_compressor'
collect2: error: ld returned 1 exit status

I suspect a -ldeflate is missing somewhere... maybe in the m4 file? couldn't figure it out :/

[kyleabeauchamp]

FWIW, I believe the Bioconda packages seem to have it working:

https://anaconda.org/bioconda/samtools/files

https://github.com/bioconda/bioconda-recipes/tree/master/recipes/samtools/build.sh

https://github.com/bioconda/bioconda-recipes/blob/master/recipes/htslib/build.sh

[kyleabeauchamp]

But I did have to do a LOT of fiddling around with it to ensure that it actually linked...

[daviesrob]

The problem is that the linker doesn't know where to find your copy of libdeflate. There are a few ways to fix this:

1. Install libdeflate and htslib where the system will normally find them, e.g. in /usr/local/. Unfortunately this requires root, and note that some linux distributions don't include /usr/local on the default shared library search path so you have to remember to add it.

2. Use LD_LIBRARY_PATH to tell the linker where to find it. You would also have to do this when running the resulting samtools binary so it's not ideal.

3. When building htslib, pass an rpath argument to the linker so that the location of libdeflate.so gets baked into htslib, something like this: LDFLAGS='-L/location/of/libdeflate -Wl,-R/location/of/libdeflate'.

You might also want to do a similar thing when building samtools so you can bake in the location of libhts.so. Then when you run the resulting samtools binary the dynamic linker should be able to resolve all of the dependencies for you without any extra help. N.B.: You would have to configure samtools to build against you installed copy of htslib in this case, and not any other one it comes across, so configure samtools with something like:

./configure --with-htslib=/location/of/installed/htslib LDFLAGS='-Wl,-R/location/of/installed/htslib/lib'

4. When building libdeflate, just build a static library with -fPIC and link that into htslib. This is the approach used by the travis set-up when testing the htslib/libdeflate combination:

git clone --depth 1 https://github.com/ebiggers/libdeflate.git
cd libdeflate
make -j 2 CFLAGS='-fPIC -O3' libdeflate.a

If you configure htslib to build against that then the symbols from libdeflate.a will be included in libhts.so and libhts.a so you don't have to worry about it again. Given that libdeflate doesn't come with an install target this approach can be a bit easier as you can just point at the source directory.

There may be issues with this approach if you then try to link samtools against the static library libhts.a. It may insist on linking against libdeflate even though it doesn't really need to. This is a bit of a tricky problem to solve - the build system should remove any libraries that are themselves statically linked into htslib from the list that programs linking against libhts.a need to include. The trouble with this is you can only find out which were included after building libhts.a, and doing so in a portable way may be difficult. As long as you still have libdeflate around when you build samtools this is likely to only be a minor annoyance.