Troubleshooting Compilation of libhdf5.a

[jimjeffers]

I'm attempting to install Rhdf5lib on a docker container based on the ubuntu:focal image. I am stuck on a compilation issue regarding the internal hdf5 library included in this package. For some reason the hdf5/src/.libs/libhdf5.a file is not getting generated in the src/.libs directory. I can't find anything in the build.log that is clearly a red flag. Is there something obvious I might be overlooking?

Complete details on the docker environment and specifics of the issue can be found on the repo we're working on:

levisimons/eDNAExplorer#9

But I'll repost a summary here:

Attempting to install phyloseq is consistently failing. As mentioned in #8 the failure occurs during the compilation of the Rhdf5lib package. This package includes a packaged source tar ball that it unpacks and compiles. While no errors are thrown during the compilation of this internal hdf5 library, inspection of the .libs folder indicates that the static assets for the library are not compiled. I am not able to diagnose the reason why. See the screenshot below in which the hdf5/src/.libs/libhdf5.a file is not present in the output .libs folder:

This subsequently causes the build of Rhdf5lib to crash when it attempts to copy the libhdf5.a file in a subsequent build step.

cp ""hdf5/c++/src"/"*.h "/opt/conda/envs/GBIF_env/lib/R/library/00LOCK-Rhdf5lib/00new/Rhdf5lib/include"
cp ""hdf5/hl/src"/"*.h "/opt/conda/envs/GBIF_env/lib/R/library/00LOCK-Rhdf5lib/00new/Rhdf5lib/include"
cp ""hdf5/hl/c++/src"/"*.h "/opt/conda/envs/GBIF_env/lib/R/library/00LOCK-Rhdf5lib/00new/Rhdf5lib/include"
cp ""hdf5/src"/"libhdf5.settings "/opt/conda/envs/GBIF_env/lib/R/library/00LOCK-Rhdf5lib/00new/Rhdf5lib/include"
mkdir -p "/opt/conda/envs/GBIF_env/lib/R/library/00LOCK-Rhdf5lib/00new/Rhdf5lib/lib/"
cp ""hdf5/src/.libs/libhdf5.a"" "/opt/conda/envs/GBIF_env/lib/R/library/00LOCK-Rhdf5lib/00new/Rhdf5lib/lib/"
cp: cannot stat 'hdf5/src/.libs/libhdf5.a': No such file or directory
make: *** [Makevars:44: copy-hdf5] Error 1
ERROR: compilation failed for package 'Rhdf5lib'
* removing '/opt/conda/envs/GBIF_env/lib/R/library/Rhdf5lib'

[nick123pig]

This will depend on your source Dockerfile; could you link that directly?

Likely you will need to apt install -y libhdf5-dev in the Dockerfile

[jimjeffers]

@nick123pig thanks here's my Dockerfile:
https://github.com/SumoCreations/eDNAExplorer/blob/main/Dockerfile

I did install libhdf5-dev via apt-get:

# Use an official Ubuntu as a parent image
FROM ubuntu:latest

# Run package updates and install packages
RUN apt-get clean && \
  apt-get update --fix-missing && \
  apt-get upgrade -y && \
  apt-get install -y \
  wget \
  bzip2 \
  unzip \
  libcurl4-openssl-dev \
  libssl-dev \
  libxml2-dev \
  libudunits2-dev \
  zlib1g-dev \
  libhdf5-dev

[nick123pig]

What's the processor arch of the machine you're running docker on? The install script is making hard binding to x86_64. For instance on my m1 macbook docker, the lib is installed at /usr/lib/aarch64-linux-gnu/hdf5/serial

If you're good on that, could you try without that --with-hdf5 argument?

[jimjeffers]

I'm on an M1 Macbook but this dockerfile is compiled with --platform linux/amd64 so that the project dependencies in the conda environment can compile.

docker build . --platform linux/amd64 -t edna-explorer

I've had the same outcome with or without the --with-hdf5=/usr/lib/x86_64-linux-gnu/hdf5/serial flag. I did confirm that apt-get installed /usr/lib/x86_64-linux-gnu/hdf5/serial/libhdf5.a which was why I tried using that flag but had no luck.

[grimbough]

Where does the --with-hdf5 flag come from? Is that documented somewhere? It's not a feature I ever remember implementing, so if you've found it documented I should probably clear that up. Your build log has the following warning:

#13 21.76 configure: WARNING: unrecognized options: --with-hdf5

The entire point the {Rhdf5lib} package is that you shouldn't need a HDF5 system library. The package provides the library, compiled with a (pretty much) fixed set of options and a known versions, which makes maintaining the {rhdf5} package a lot easier.

---

One thing I note is that are a lot of double quotes in your build log output e.g.

cp ""hdf5/src"/"*.h "/opt/conda/envs/GBIF_env/lib/R/library/00LOCK-Rhdf5lib/00new/Rhdf5lib/include"

On the Bioconductor build report that same line appears as below, which doesn't start with cp ""

cp "hdf5/src"/*.h "/home/biocbuild/bbs-3.17-bioc/R/site-library/00LOCK-Rhdf5lib/00new/Rhdf5lib/include"

Maybe that's a red-herring, but perhaps some paths are getting mangled somehow and this is a symptom.

---

As a general statement, if you're using conda to install R, it's also much easier to install the package from bioconda as well. It's quite common for people to run into issues when trying to mix the two.

[grimbough]

I see I fixed the double quoting issue in 6bef8cf which is available in Bioc 3.17 but not 3.16, which you are using.

[grimbough]

I think this issue was because the package was not automatically identifying the version of ar provided by conda. This in turn was causing libtool to fail to generate the static libraries. Why it did this without any sort of error elludes me.

I've now fixed it in the latest version of Rhdf5lib in both Bioconductor 3.17 and 3.18. Unfortunately I can't backport the change to 3.16, which is what is used by R-4.2 in your container. I think you'll either have to use a newer R, install Rhdf5lib from GitHub, or use the version distributed via bioconda.

[jimjeffers]

Thanks I’ll look into upgrading the environment. 👍🏼Sent from my iPhoneOn Sep 8, 2023, at 1:46 PM, Mike Smith ***@***.***> wrote: I think this issue was because the package was not automatically identifying the version of ar provided by conda. This in turn was causing libtool to fail to generate the static libraries. Why it did this without any sort of error elludes me. I've now fixed it in the latest version of Rhdf5lib in both Bioconductor 3.17 and 3.18. Unfortunately I can't backport the change to 3.16, which is what is used by R-4.2 in your container. I think you'll either have to use a newer R, install Rhdf5lib from GitHub, or use the version distributed via bioconda. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>