New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker HDF5 bug via "import h5py" #213

Closed
CRMacPherson opened this Issue Oct 15, 2015 · 10 comments

Comments

Projects
None yet
2 participants
@CRMacPherson

CRMacPherson commented Oct 15, 2015

Great work on the Docker image, its nice to see examples!

I've run into a "file not found" type error. Full traceback from the mini_test script below. I thought you might not have updated the docker image for HDF5, so I tried (re)installing the hdf5 library, in the container, as per the instructions on your site but no luck.

"""

Generating an EMPTY contigs database ...

Traceback (most recent call last):
File "/usr/local/bin/anvi-gen-contigs-database", line 7, in
import anvio.dbops as dbops
File "/usr/local/lib/python2.7/dist-packages/anvio/dbops.py", line 28, in
import anvio.auxiliarydataops as auxiliarydataops
File "/usr/local/lib/python2.7/dist-packages/anvio/auxiliarydataops.py", line 4, in
import h5py
File "/usr/local/lib/python2.7/dist-packages/h5py/init.py", line 13, in
from . import _errors
ImportError: libhdf5.so.7: cannot open shared object file: No such file or directory
"""

@meren

This comment has been minimized.

Member

meren commented Oct 15, 2015

Crap :/

This is my fault. I am sorry. I went back to my server logs, and it seems I've been testing the docker image for 1.2.0 by running this command:

docker run --rm -it meren/anvio:1.1.0

So I never tested 1.2.0 before uploading it :(

I will be very busy today, and I asked a friend to take a look. If no one responds to this, I will try to resolve it as soon as I can. Meanwhile please try the docker image for 1.1.0 version for now.

@CRMacPherson

This comment has been minimized.

CRMacPherson commented Oct 15, 2015

OK, no problem, thanks for the quick reply. I can confirm the error is not repeatable in tag 1.1.0.

A new error in 1.1.0: running "anvi-profile -i name.bam --list-contigs | head" says that the -c option is required. The documentation for anvi-profile says either -i or -c should be specified. The tutorial (http://merenlab.org/2015/05/02/anvio-tutorial/) says that the minimal command requires both -i and -c, "anvi-profile -i X.bam -c contigs.db".

I'll create an official issue once the 1.2.0 image is stable.

@meren

This comment has been minimized.

Member

meren commented Oct 15, 2015

This is what happens when cute projects really start being used :/ If you bear with us, we will get all of them sorted out. Sorry about these embarrassing hiccups, and thanks for your patience.

@meren

This comment has been minimized.

Member

meren commented Oct 15, 2015

I found the problem with the Docker image. I will upload an updated docker image in 10-15 minutes.

meren added a commit that referenced this issue Oct 15, 2015

meren added a commit that referenced this issue Oct 15, 2015

sorry about the previous commit. I shall never edit anything using th…
…e Github editor. this is the proper Dockerfile now... related #213
@meren

This comment has been minimized.

Member

meren commented Oct 15, 2015

Hi @CRMacPherson,

I just pushed a new 1.2.0 that should be working (the image id is 110b749e53d1). I'd be very happy if you can confirm whether it works. Thanks a lot.

@CRMacPherson

This comment has been minimized.

CRMacPherson commented Oct 15, 2015

@meren
Hey, can confirm the new layer 110b749e53d1 pulled successfully and the mini_test script ran without error. Nicely done and thanks for the prompt response.

Maybe you can mention what the problem was for all those people not using Docker?

@meren

This comment has been minimized.

Member

meren commented Oct 15, 2015

Thanks!

In order to save space in the resulting docker image, we had this command that removes all packages that are necessary for building stuff, but are not used during runtime:

apt-get remove -y \
        binutils \
        build-essential \
        g++ \
        g++-4.8 \
        gcc \
        gcc-4.8 \
        make \
        patch \
        $(dpkg --list | grep "^ii" | grep "\-dev" | awk '{print $2}' | xargs) \

The very last parameter in this one-liner removes every package that ends with -dev to get rid of all the development packages once we're done with compiling things in the image. Along with everything else it was removing libhdf5-dev from the system, and as a result h5py was complaining about the missing shared object (libhdf5.so.7 --which I am not sure whether it should be shipped in a *-dev package, but I will not start a war with anyone over this).

As a quick resolution I removed the line from our Dockerfile, so all *-dev packages stay in the image (which increased the size about 80Mb. For which I will do a better job later).

Thanks for your patience.

@meren meren closed this Oct 15, 2015

@CRMacPherson

This comment has been minimized.

CRMacPherson commented Oct 15, 2015

Great thanks again!

Also Docker related,

Perhaps mention in the docs how to connect the host's browser to the interactive server...

Assuming the use of a the Docker Toolbox on Mac/Windows:

  1. After starting a VM with the quick start terminal take note of the IPAddress. It will look something like this "docker is configured to use the default machine with IP 192.168.99.100"
  2. Start the docker container with docker run -p 8080:8080 -it meren/anvio:1.2.0
  3. Change directory to "anvi-tests"
  4. Run "./run_mini_test.sh". All tests will run and at the end it will say 'The server is now listening the port number "8080"'.
  5. Finally, go to your Chrome browser and type the VM's IPAddress that you noted in (1) and the port 8080.
    5.1. For this example, "192.168.99.100:8080"
@meren

This comment has been minimized.

Member

meren commented Oct 15, 2015

This is a good idea. It forced me to install Docker Tools on my Mac.

I will test the performance, and add a section for people who would like to do it on OSX or Windows.

Thank you very much!

@meren

This comment has been minimized.

Member

meren commented Oct 15, 2015

I updated the post, @CRMacPherson, thanks again for the suggestion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment