Linker issues against 1.8.18-1 but not against 1.8.17.

[wasade]

I'm receiving linker errors against 1.8.18-1 (conda-forge) that I was not receiving against 1.8.17-2 (defaults). The linker errors suggest that the HDF5 shared libraries are not being built, however an unpacking of the 1.8.18-1 conda package does show the .so files, and my read of the ./configure in build.sh suggests they are being built as it's enabled by default (at least when I last checked HDF5 source). Is there any known reason for why linking would break between these builds?

EDITED: the issue is actually a discrepancy between defaults and conda-forge feedstocks.

# 1.8.18-1 from conda-forge
# ...snip...
h5c++ -shared -o libssu.so tree.o biom.o unifrac.o cmd.o unifrac_task.o api.o -lc -lhdf5_cpp -L/home/mcdonadt/miniconda3/conda-bld/unifrac_1506038888908/_h_env_placehold_placehold_placehold_placehold_p
lacehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib
/usr/bin/ld: /home/mcdonadt/miniconda3/conda-bld/unifrac_1506038888908/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla
cehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/libhdf5_cpp.a(H5IdComponent.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile w
ith -fPIC
/home/mcdonadt/miniconda3/conda-bld/unifrac_1506038888908/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeh
old_placehold_placehold_placehold_placehold_placehold_/lib/libhdf5_cpp.a: could not read symbols: Bad value
collect2: error: ld returned 1 exit status

# 1.8.17-2 from defaults
# ...snip...
h5c++ -shared -o libssu.so tree.o biom.o unifrac.o cmd.o unifrac_task.o api.o -lc -lhdf5_cpp -L/home/mcdonadt/miniconda3/conda-bld/unifrac_1506053505626/_h_env_placehold_placehold_placehold_placehold_p
lacehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib
cp libssu.so /home/mcdonadt/miniconda3/conda-bld/unifrac_1506053505626/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla
cehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/

I'm getting similar linking issues on 1.10.1.

[jakirkham]

I guess because we didn't build the static libraries with -fPIC. Would you like to submit a PR?

[wasade]

--enable-shared was default yes in 1.8.12, which is the only source I have locally at the moment:

  --enable-shared[=PKGS]  build shared libraries [default=yes]

Do you think HDF5 changed that default given that 1.8.17-11 from conda-forge seems to work? There are .so files though associated with 1.8.18-1, which adds to the confusion. I'll issue a PR momentarily explicitly enabling though.

$ ls /home/mcdonadt/miniconda3/conda-bld/unifrac_1506052330974/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/*hdf5*.so
lrwxrwxrwx 1 dtmcdonald knightlab 17 Sep 21 20:53 /home/mcdonadt/miniconda3/conda-bld/unifrac_1506052330974/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/libhdf5.so -> libhdf5.so.10.2.1
lrwxrwxrwx 1 dtmcdonald knightlab 21 Sep 21 20:53 /home/mcdonadt/miniconda3/conda-bld/unifrac_1506052330974/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/libhdf5_cpp.so -> libhdf5_cpp.so.13.0.0
lrwxrwxrwx 1 dtmcdonald knightlab 25 Sep 21 20:53 /home/mcdonadt/miniconda3/conda-bld/unifrac_1506052330974/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/libhdf5_fortran.so -> libhdf5_fortran.so.10.0.4
lrwxrwxrwx 1 dtmcdonald knightlab 20 Sep 21 20:53 /home/mcdonadt/miniconda3/conda-bld/unifrac_1506052330974/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/libhdf5_hl.so -> libhdf5_hl.so.10.1.1
lrwxrwxrwx 1 dtmcdonald knightlab 24 Sep 21 20:53 /home/mcdonadt/miniconda3/conda-bld/unifrac_1506052330974/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/libhdf5_hl_cpp.so -> libhdf5_hl_cpp.so.11.1.0
lrwxrwxrwx 1 dtmcdonald knightlab 27 Sep 21 20:53 /home/mcdonadt/miniconda3/conda-bld/unifrac_1506052330974/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/libhdf5hl_fortran.so -> libhdf5hl_fortran.so.10.0.3

[wasade]

Might also need --with-pic although, like --enable-shared, I believe it is enabled already (i.e. default=use both).

[jakirkham]

Considering as the shared libraries are already there as we can both see. I don't think that is the issue. If HDF5 is already building with -fPIC, then I'm mystified as to why the linker can't figure out what to do. Is this coming from some other feedstock in conda-forge?

[wasade]

hdf5: 1.8.18-1 conda-forge

In looking back, right now, it looks like the successful build is using 1.8.17-2 from defaults at build. It successfully links at runtime against 1.8.17-11 from conda-forge, which I think drove some confusion -- for a little more context, the forced replacement of hdf5 on that line allowed that repo to use our problematic library. I'm testing 1.8.17 at build from conda-forge right now.

I'm a bit green with recipes and conda build has a learning curve, so apologize if I'm making any naive mistakes. And thank you for the help.

[wasade]

So it does seem to be the case that we can build using 1.8.17-2 from defaults but are unable to build using conda-forge, and tried with 1.8.17, 1.8.18, and 1.10.1.

[jakirkham]

Do you have links to the logs where this issue occurs?

[wasade]

hdf5_1.8.17_conda_forge.txt, generated by $ conda build -c biocore -c conda-forge unifrac &> hdf5_1.8.17_conda_forge.txt
hdf5_1.8.17_defaults.txt, generated by conda build -c biocore unifrac &> hdf5_1.8.17_defaults.txt

Using a modified version of this recipe for the build where I just further reduced the build reqs as there was some fluff, and specified the specific hdf5 version.

$ git diff unifrac/meta.yaml
diff --git a/recipes/unifrac/meta.yaml b/recipes/unifrac/meta.yaml
index 78515fe..9b0b11f 100644
--- a/recipes/unifrac/meta.yaml
+++ b/recipes/unifrac/meta.yaml
@@ -16,16 +16,14 @@ requirements:
     - python 3.5*
     - gcc  # [not osx]
     - llvm  # [osx]
-    - hdf5
-    - h5py
+    - hdf5 ==1.8.17
     - cython
-    - biom-format
-    - scikit-bio
+    - numpy

   run:
     - python 3.5*
     - libgcc  # [not osx]
-    - hdf5
+    - hdf5 ==1.8.17
     - h5py
     - cython
     - biom-format

[jakirkham]

Anything more to do here @wasade? I think we leapfrogged to 1.10.1 for the most part. If you have issues with that version as well, maybe we should create a new issue.

[wasade]

I'm not sure to be honest, this fell off my radar, but I think we still want --enable-shared set at ./configure?

[jakirkham]

I'm pretty sure --enable-shared is default otherwise we wouldn't be getting dynamic/shared objects, which do we have in the package. So don't think that would fix the issue.

[jakirkham]

Added PR ( #82 ), which cherry picks your change and adds PIC flags. I suspect the PIC flags will fix the issue. The HDF5 docs suggest they use both, which I'm not really sure how that works. Normally one uses PIC or they don't. Plus the error message cited above is a typical non-PIC build issue. Would encourage you to retest after we merge that. We can cherry-pick the change to 1.8 builds as well.