Skip to content

Commit

Permalink
update anaconda section
Browse files Browse the repository at this point in the history
  • Loading branch information
cehbrecht committed Mar 19, 2015
1 parent 72ca4ce commit acd9b75
Show file tree
Hide file tree
Showing 2 changed files with 56 additions and 8 deletions.
48 changes: 41 additions & 7 deletions docs/source/dev_guide.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,25 +9,59 @@ Developer Guide

.. _anaconda:

Birdhouse Anaconda Packages
Using Anaconda in Birdhouse
---------------------------

The installation of the Birdhouse components and especially the processes involve many software dependencies. The core dependencies are of course the WPS related packages like `PyWPS` and `OWSLib` from the GeoPython project. But most dependencies come from the processes itself served by the WPS like `numpy`, `R`, `NetCDF`, `CDO`, `matplotlib`, ... and many more.
The installation of the Birdhouse components and especially the processes involve many software dependencies. The core dependencies are of course the WPS related packages like :term:`PyWPS` and :term:`OWSLib` from the :term:`GeoPython` project. But most dependencies come from the processes itself served by the WPS like `numpy`, `R`, `NetCDF`, `CDO`, `matplotlib`, `ncl`, `cdat`, ... and many more.

The aim of Birdhouse is to take care of all these dependencies so that the user does not need to install them manually. If these dependencies would only be *pure* Python packages then using the :term:`Buildout` build tool together with the Python package index :term:`PyPi` would be sufficient. But many Python packages have `C` extensions and there are also non-Python packages we need to install like `R` and `netcdflib`.
The aim of Birdhouse is to take care of all these dependencies so that the user does not need to install them manually. If these dependencies would only be *pure* Python packages then using the :term:`Buildout` build tool together with the Python package index :term:`PyPi` would be sufficient. But many Python packages have `C` extensions and there are also non-Python packages we need to install like `R` and :term:`NetCDF`.

In this situation the :term:`Anaconda Python distribution` comes helpful. Anaconda has already a lot of Python related packages available for different platforms (Linux, MacOSX, Windows) and there is no compilation needed on the installation host. Anaconda makes it easy to build own packages (*conda recipes*) and to upload them to the free Anaconda server on :term:`Binstar`.

Birdhouse is using Anaconda and it is integrated into the Buildout build tool. The additional *conda recipes* used by Birdhouse are available on `GitHub <https://github.com/bird-house/conda-recipes>`_. The build packages can be installed from the `Birdhouse organisation on Binstar <https://binstar.org/birdhouse>`_. For example if you are already using Anaconda, you can install `CDO` with the following command::
Conda recipes by Birdhouse
~~~~~~~~~~~~~~~~~~~~~~~~~~

$ conda install -c birdhouse cdo
Birdhouse is using :term:`Anaconda` to maintain package dependencies.
Anaconda gives you the possibility to write your own `conda recipes <http://conda.pydata.org/docs/build.html>`_.
In Birdhouse we have written several conda recipes for the packages that were not available on Anaconda.
These `additional conda recipes by Birdhouse <https://github.com/bird-house/conda-recipes>`_ are available on :term:`GitHub`.
Some of the missing packages are: PyWPS, OWSLib, CDO, ocgis/icclim, cfchecker, nginx, ...

Anaconda provides a free Anaconda Server on :term:`Binstar`. Here you can upload your builded conda packages for different platforms (Linux, MacOX, Windows). These packages are then available for installation with the :term:`conda` installer.

`Birdhouse has a Binstar organisation <https://binstar.org/birdhouse>`_ where all conda packages are collected which are
builded from the conda recipes on GitHub. These packages can be installed with the :term:`conda` installer using the `birdhouse` channel.
For example if you are already using Anaconda, you can install `CDO` with the following command::

$ conda install --channel birdhouse cdo

Building conda packages
~~~~~~~~~~~~~~~~~~~~~~~

There are several ways to build conda packages and upload them to Binstar.

* You can `build packages locally <http://docs.binstar.org/conda.html>`_ and upload them with the Binstar command line tool.
* You can also `build packages remotely on Binstar <http://docs.binstar.org/draft/examples.html#SubmitYourFirstBuild>`_. Additionally you can set a GitHub Webhook so that on each commit of your recipe a build will be run on Binstar.
* The remote build on Binstar are done using Docker images. The `Binstar docker image for Linux-64 <https://registry.hub.docker.com/u/binstar/linux-64/>`_ is available on :term:`Docker Hub`.

In Birdhouse we usually use the remote build on Binstar which is triggered by commits to GitHub.
But sometimes the docker image for Linux-64 provided by Binstar fails for some packages.
That is why `Birdhouse has in addition its own Linux-64 build image <https://registry.hub.docker.com/u/birdhouse/binstar-linux-64/>`_ which is based on the Binstar image.
The `Dockerfile for this image <https://github.com/bird-house/birdhouse-build/tree/master/docker/binstar-linux-64>`_ is on GitHub.


Anaconda Alternatives
~~~~~~~~~~~~~~~~~~~~~

If Anaconda would not be available one could also provide these packages from source and compile them on each installation host. Buildout does provide ways to do so. But an initial installation with most of the software used in climate science could *easily take hours*.

Alternative package manager to Anaconda are for example :term:`Homebrew` (MacOSX only) and :term:`Linuxbrew` (a fork of Homebrew for Linux).

Buildout Recipes provided by Birdhouse
--------------------------------------
Using Buildout in Birdhouse
---------------------------

Buildout Recipes by Birdhouse
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

:term:`Buildout` has a plugin mechanism to extend the build tool functionality with `recipes <http://www.buildout.org/en/latest/docs/recipe.html>`_. Birdhouse provides a Buildout recipe to install Anaconda packages. There is also a set of recipes to set up Web Processing Service with PyWPS, Nginx, Gunicorn and Supervisor. All these `Buildout recipes are on GitHub <https://github.com/bird-house?query=birdhousebuilder.recipe>`_ and can be `found on PyPi <https://pypi.python.org/pypi?%3Aaction=search&term=birdhousebuilder.recipe&submit=search>`_.

Expand Down
16 changes: 15 additions & 1 deletion docs/source/glossary.rst
Original file line number Diff line number Diff line change
Expand Up @@ -48,14 +48,23 @@ Glossary
Python distribution for large-scale data processing, predictive analytics, and scientific computing.
http://www.continuum.io/

Conda
The *conda* command is the primary interface for managing Anaconda installations.
http://conda.pydata.org/docs/index.html

Binstar
Binstar is a service that allows you to create and manage public and private :term:`Anaconda` package repositories.
https://binstar.org/
http://docs.binstar.org/

Docker
*Docker* - An open platform for distributed applications for developers and sysadmins.
https://www.docker.com/

Docker Hub
Docker Hub manages the lifecycle of distributed apps with cloud services for building and sharing containers and automating workflows.
https://hub.docker.com/

GeoPython
GitHub organisation of Python projects related to geospatial.
https://geopython.github.io/
Expand Down Expand Up @@ -118,7 +127,12 @@ Glossary

Linuxbrew
Linuxbrew is a fork of Homebrew, the Mac OS package manager, for Linux.
http://brew.sh/linuxbrew/`
http://brew.sh/linuxbrew/

GitHub
GitHub is a web-based Git repository hosting service.
https://github.com/
https://en.wikipedia.org/wiki/GitHub



0 comments on commit acd9b75

Please sign in to comment.