Dockerizing fastBMA a step by step tutorial

Welcome to the fastBMAdemo wiki! __ ##Dockerizing fastBMA - a hands-on tutorial##

###Summary# This tutorial was meant to introduce the process of building a Dockerfile to containerize a GitHub project. It is aimed at an audience with little or no GitHub or Docker experience and was originally used in a graduate bioinformatics seminar course at the University of Washington Tacoma. The tutorial will go through a real-life example, starting with a package, fastBMA, which has somewhat complex dependencies, and was originally developed on Fedora. The container will start with an Ubuntu distro. We will download the starting repository, attach the directory to our docker container, update the directory with the necessary dependencies, compile dependencies that are not properly packaged by the distro, modify the Makefile, compile the binary and then generate the final Dockerfile from the history file. ###Start up Docker# Follow the instructions here to start and install Docker ###Clone the fastBMADemo repository# git clone https://github.com/lhhunghimself/fastBMAdemo

This is the software that we are going to Dockerize. The major dependencies are OpenBLAS and boost MPI, both of which are a bit of a pain to install. ###Build the starting Docker Image# cd fastBMAdemo ls

If we examine the Docker file...

cat Dockerfile

We will see that this is a pretty simple Dockerfile that just pulls the baseimage. fastBMA requires OpenBLAS and for some reason the version OpenBLAS packaged with Ubuntu does not seem to work well with fastBMA. The ogrise/openblas image has OpenBLAS compiled from source.

Now let's build it

docker build --tag fastbmampi .

On Linux installations, you need to be the root to do this. You can either change to root by "su root" or preface each of the commands with "sudo". On MacOS and Windows command, when you are logged into a Linux Virtual Machine as root so the "sudo" is unnecessary.

Let's list the images and check that the new image is built

docker images

###Run and log into the Docker container# Now we want to start up the Docker container, log into it, mount our local software directory so that it can be accessed from inside the container and then try to compile it. In doing so, we will figure out and add the necessary dependencies. To do this we type

docker run -ti --rm -v $PWD:/local fastbmampi /bin/bash

-ti starts up an interactive terminal for us --rm cleans up after you quit the terminal -v $PWD:/local mounts the current directory (stored in the environment variable $PWD) to the mountpoint /local inside the container NB some windows systems it seems that pwd without the $ sign is required fastbmampi is the name of the container to run /bin/bash is the starting command so that we will start in a bash shell You should see this:

###Update the container and install dependencies with apt-get# Now we are inside the container, logged in as root. The first thing to do is to update the image and then install some dependencies that we will need.

apt-get update -y && apt-get install -y libmpich-dev wget g++ nano

The tools and dependencies are: g++ - c++ compiler nano - simple text editor (not strictly needed) wget - tool to fetch files from a url libmpich-dev - provides MPI headers and libraries needed by the MPI version of fastBMA

###Compile libraries that are not provided by apt-get# Unfortunately fastBMA uses boost-MPI as a wrapper around the MPI routines for c++. Although this is available through apt-get - the packaged version DOES NOT work with mpich. The entire set of boost libraries must be compiled from source.

First we download and extract the source

wget https://sourceforge.net/projects/boost/files/boost/1.57.0/boost_1_57_0.tar.bz2
tar -xjvf boost_1_57_0.tar.bz2
rm boost_1_57_0.tar.bz2 -rf

The we set up the compile

cd boost_1_57_0
./bootstrap.sh

We then need to add a line to the configuraton file to tell it that we are using MPI

echo 'using mpi ;' >> project-config.jam

Now we compile and install. The example uses -j8 to compile with 8 threads. Change this to the number of threads that are available to you. On Windows and MacOS, the VM running Docker is usually set to 1 thread. This can take a few minutes - time to get a coffee...

./b2 -j8 -target=shared,static
./b2 install

Let the system know where to find these libraries

echo 'export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

###Adjust fastBMA Makefile#

cd /local/fastBMAdemo/src
make clean; make USEMPI=1;

This will give rise to an error because the openblas libraries are not in the same place as the machine where fastBMA was written. This is why we use Docker...

In this container the OpenBLAS libraries are at /opt/OpenBLAS/lib - so we need to add this to the Makefile. You can open the Makefile using "nano Makefile" and add "-L /opt/OpenBLAS/lib" to the end of the or use the following sed command

sed -i 's/LFLAGS=-L/LFLAGS=-L \/opt\/OpenBLAS\/lib -L /' Makefile

Now the file should link

make USEMPI=1;

Move the binary to a place where bash knows to find it

cp fastBMA /usr/local/bin/

Test it with the provided script

./run_fastBMAMPI

Your output should be an edge list like this:

###Save the history file and exit#

history > myHistoryFile
exit

###Make your DockerFile using the history file#

After you exit the container, the changes to the container are not kept. However, because the new Makefile and fastBMA file were in the shared directory, those files are on your local machine. The final Dockerfile that resulted is at https://github.com/lhhunghimself/fastBMA/Dockerfile and is very similar to the history file that you saved.

###Clean up Docker images# Don't forget to clean up the docker images by doing

Docker rmi fastbmampi
Docker rmi docker.io/ogrisel/openblas:latest