qiime-deploy is a tool for building, configuring, and deploying many of QIIME's dependencies on Linux systems.
Note: qiime-deploy does not install every dependency necessary for a
complete QIIME installation. Python, R, and their respective requisite packages
are not installed because we expect users to install these dependencies using
standard package managers that are available for the language (e.g., using
pip to install Python packages). Also, certain dependencies
(e.g., usearch) are not installed due to licensing restrictions.
To run qiime-deploy, you will first need to grab a configuration file. The configuration file will tell qiime-deploy what packages it should install. You can find many QIIME deployment configuration files in the qiime-deploy-conf project here as well as a description of all possible options and the configuration file format. There are also various other configuration files included for installing QIIME-related projects/tools.
Also, while qiime-deploy downloads, builds, and installs many of QIIME's dependencies, it does expect common packages to already be installed on your system. See below for a list of these prerequisites, which will differ depending on your Linux distribution.
After the deploy completes it will generate an
activate.sh file in the
base deploy directory. It is necessary to
source that file in order to
set the environment created by qiime-deploy. For example, if the deploy
/opt/qiime/ you will need to run the following command:
Note: By default, qiime-deploy will append the above line to your
~/.bashrc. This is done so that you do not need to run the command
each time you open a new terminal. To disable this behavior, you will need to
modify the qiime-deploy conf file (see the qiime-deploy-conf project
here for a description of the
options you can tweak).
Setting up qiime-deploy on Ubuntu
While qiime-deploy may work on various Linux distributions, it has been most heavily tested on Ubuntu 11.10 and 12.04 LTS (64-bit) systems. Thus, we provide a short guide to setting up your Ubuntu system so that it can run qiime-deploy.
In order to ensure that you have the required prerequisite packages installed on your Ubuntu system, perform the following steps. These steps require admin (sudo) access. If you do not have sudo access, you must ask your system administrator to grant you sudo access, or to run the following steps for you.
/etc/apt/sources.listwith a text editor. You can use your favorite text editor but we suggest pico for simplicity. Note that at the bottom of the screen you will have the commands to save, exit, etc.:
sudo pico /etc/apt/sources.list
Uncomment the universe and multiverse repositories in this file.
Add the following line to the bottom of this file in order to install a newer version of R than is available by default on the system.
For Ubuntu 11.10:
deb http://cran.rstudio.com/bin/linux/ubuntu oneiric/
For Ubuntu 12.04:
deb http://cran.rstudio.com/bin/linux/ubuntu precise/
For Ubuntu 14.04:
deb http://cran.rstudio.com/bin/linux/ubuntu trusty/
Save and close this file.
Update your system to retrieve the latest list of packages (this step is especially important because we added new package repositories in the previous step):
sudo apt-get -y update
Install the qiime-deploy prerequisite packages on your machine. In general, most of this software should already be installed but some packages may not be. It is therefore best to run this command before continuing.
For Ubuntu 11.10:
sudo apt-get --force-yes -y install python-dev libncurses5-dev libssl-dev libzmq-dev libgsl0-dev openjdk-6-jdk libxml2 libxslt1.1 libxslt1-dev ant git subversion build-essential zlib1g-dev libpng12-dev libfreetype6-dev mpich2 libreadline-dev gfortran unzip libmysqlclient16 libmysqlclient-dev ghc sqlite3 libsqlite3-dev libbz2-dev tcl-dev tk-dev r-base r-base-dev libatlas-dev libatlas-base-dev liblapack-dev swig libhdf5-serial-dev
For Ubuntu 12.04 and Ubuntu 14.04:
sudo apt-get --force-yes -y install python-dev libncurses5-dev libssl-dev libzmq-dev libgsl0-dev openjdk-6-jdk libxml2 libxslt1.1 libxslt1-dev ant git subversion build-essential zlib1g-dev libpng12-dev libfreetype6-dev mpich2 libreadline-dev gfortran unzip libmysqlclient18 libmysqlclient-dev ghc sqlite3 libsqlite3-dev libc6-i386 libbz2-dev tcl-dev tk-dev r-base r-base-dev libatlas-dev libatlas-base-dev liblapack-dev swig libhdf5-serial-dev
Setting up qiime-deploy on CentOS and RedHat
Below is the summary of steps required for running qiime-deploy on CentOS and RedHat (tested on CentOS 6.4):
Add the following repositories to your yum configuration directory (required to install ZeroMQ):
For RHEL/CentOS 6:
This can be done by creating a file in /etc/yum.repos.d/, e.g., named zeromq.repo (requires admin (sudo) access). You can use your favorite text editor but we suggest pico for simplicity. Note that at the bottom of the screen you will have the commands to save, exit, etc.
sudo pico /etc/yum.repos.d/zeromq.repo
Paste the following into that file:
For RHEL/CentOS 6:
[home_fengshuo_zeromq] name=The latest stable of zeromq builds (CentOS_CentOS-6) type=rpm-md baseurl=http://download.opensuse.org/repositories/home:/fengshuo:/zeromq/CentOS_CentOS-6/ gpgcheck=1 gpgkey=http://download.opensuse.org/repositories/home:/fengshuo:/zeromq/CentOS_CentOS-6/repodata/repomd.xml.key enabled=1
Save and exit that file. Added repository will be used in the next step.
Install the qiime-deploy dependencies on your machine. These steps require admin (sudo) access. If you do not have sudo access, you must ask your system administrator to grant you sudo access, or to run these commands for you.
sudo yum groupinstall -y "development tools" sudo yum install -y ant compat-gcc-34-g77 java-1.6.0-openjdk java-1.6.0-openjdk-devel freetype freetype-devel zlib-devel mpich2 readline-devel zeromq zeromq-devel gsl gsl-devel libxslt libpng libpng-devel libgfortran mysql mysql-devel libXt libXt-devel libX11-devel mpich2 mpich2-devel libxml2 xorg-x11-server-Xorg dejavu* python-devel sqlite-devel tcl-devel tk-devel R R-devel ghc
Installing requisite Python and R packages
Installing the latest version of QIIME (currently 1.9.1) using qiime-deploy does not install Python, R, or their respective requisite packages (this behavior differs from how qiime-deploy installed previous versions of QIIME). qiime-deploy now assumes that a minimal (base) QIIME install exists. Follow the instructions here to obtain a minimal (base) QIIME install. Next, follow the instructions here to install QIIME's requisite R packages. You're now ready to use qiime-deploy to install QIIME's remaining dependencies.
Common usage examples
The following subsections include examples of common qiime-deploy use cases.
Note: At the time of this writing, QIIME 1.9.1 is the latest public release, and QIIME 1.9.1-dev is the development version of QIIME. As newer versions of QIIME are released we will include conf files for each new version in the qiime-deploy-conf project. The following usage examples will work for any version of QIIME (unless otherwise noted), but you will need to supply the correct conf file as input to qiime-deploy.
To get started please download the repositories listed below. These commands assume you have already set up your system following the directions above and that you are in your home directory.
git clone git://github.com/qiime/qiime-deploy.git git clone git://github.com/qiime/qiime-deploy-conf.git cd qiime-deploy/
View qiime-deploy options
To see the available options provided by qiime-deploy, run the following command:
python qiime-deploy.py -h
Installing QIIME 1.9.1's dependencies
To install QIIME 1.9.1's dependencies under
$HOME/qiime_software/, run the
following commands. These commands assume you have already set up your system
following the directions above and that you are in your home directory. You can
change these paths as you like (e.g. to install QIIME's dependencies under a
different directory), but you will need to modify the commands we provide to use
the new paths.
python qiime-deploy.py $HOME/qiime_software/ -f $HOME/qiime-deploy-conf/qiime-1.9.1/qiime.conf --force-remove-failed-dirs source $HOME/.bashrc
To test that you have a functioning QIIME install, run the following command:
qiime-deploy will create a QIIME config file under
$HOME/qiime_software/qiime_config as part of the deployment process. If
you would like to edit this file to further customize your QIIME install, feel
free to do so. If you rerun qiime-deploy using the same deploy directory (in
$HOME/qiime_software/), your old QIIME config will be
qiime_config.bak and the new one will be named
Installing multiple versions of QIIME
You may install more than one version of QIIME's dependencies on your system. To
do so, you will need to install each version in its own deploy directory. For
example, if you would like to have both the dependencies for QIIME 1.9.1 and
QIIME 1.9.1-dev, you could install QIIME 1.9.1's dependencies under
$HOME/qiime-1.9.1/ and QIIME 1.9.1-dev's dependencies under
$HOME/qiime-1.9.1-dev/. To activate the QIIME version that you would like
source the appropriate
activate.sh file. For example, to
activate QIIME 1.9.1-dev, you would run the following command:
If you are unsure of what version of QIIME you currently have activated, run the following command:
Changing QIIME versions
If you want to change the version of QIIME's dependencies in an existing qiime-deploy install, you can simply run qiime-deploy with the conf file corresponding to the version that you'd like to upgrade/downgrade to. Make sure to specify the existing deploy directory in order to upgrade/downgrade your existing install. If you specify a new directory, you will end up with multiple versions of QIIME's dependencies installed on your system (which is okay; see the section above for more details).
Frequently Asked Questions
When I run
print_qiime_config.py -tf, I get a test failure for
usearch. How can I fix this?
qiime-deploy cannot install usearch due to licensing restrictions. You can obtain the usearch binary here. Please be sure to download the currently supported version for your version of QIIME. If you are using the latest stable release of QIIME, you can find the required usearch version here.
Rename the executable to
usearch and make sure it is somewhere that is in
PATH environment variable. You also need to ensure that execute
permissions are set. For example:
mkdir $HOME/bin mv <path to usearch executable> $HOME/bin/usearch chmod +x $HOME/bin/usearch echo 'export PATH=$HOME/bin:$PATH' >> $HOME/.bashrc (open a new terminal) usearch --version print_qiime_config.py -tf
Does qiime-deploy work on 32-bit operating systems?
No, qiime-deploy only supports 64-bit operating systems. If you run
qiime-deploy on a 32-bit system, it may correctly install some dependencies,
but when running
print_qiime_config.py -tf to check that your QIIME
install is functioning, you may receive strange errors similar to the
ELF : not found Syntax error: word unexpected (expecting ")") Syntax error: Unterminated quoted string
Upgrading your operating system to 64-bit and rerunning qiime-deploy will solve this issue.
New applications with custom build types should be added to
lib/custom.py. Both a function and a call to that function from
custom.custom_deployshould be added.
Any custom finalization code should be called from
If additional options need to be added to any of the config file sections, they will likely need to be added to the init of the
New generic build processes will need to be added to the
lib/application.py. Additionally, the generic utility functions required should be added to