Browse files

Simplify customization: allow flavors to specify a directory containi…

…ng overrides for standard YAML/fabricrc configuration files. Update documentation and examples to match new behavior
  • Loading branch information...
1 parent bc89032 commit 5e045f9702db698a172def151feae3613256ac8e @chapmanb committed Sep 21, 2012
View
228 README.md
@@ -1,47 +1,22 @@
-CloudBioLinux is a build and deployment system which installs a large
-selection of Bioinformatics and machine learning libraries on a bare
-virtual machine (VM) image, freshly installed PC, or in the Cloud.
-All that is required is a login with ssh (secure shell) into a fresh
-base install of Linux. The CloudBiolinux scripts will login and
-install software remotely - followed by system configuration.
-
-By default CloudBioLinux includes a large suite of tools and
-libraries, largely pulled from the package management system provided
-by the image. In addition to the default configuration, other software
-configurations are supported through, so called, 'editions' and
-'flavors'. An edition is a named base install, e.g. CloudBioLinux for
-Ubuntu. Such an Edition reflects shared installation properties
-between different projects. Flavors support overriding base packages
-for specific installations. With a Flavor a derivative of a base
-Edition can be designed. For example a single web server cound be
-implemented as a Flavor on top of the 'Minimal' Debian edition.
-CloudBioLinux installs packages through multiple mechanisms, including
-the default distribution installer, native installers, and libraries
-for Perl, R, Python, JAVA and Ruby, as well as installers for special
-data resources.
-
-CloudBioLinux is designed as a single install route for both VMs on
-the desktop, such as [VirtualBox][v2], and cloud providers, such as
-[Amazon EC2][0], [Openstack][openstack], where you start with a bare
-bones system and bootstrap a running instance. You can even create a
-minimalistic private cloud on top of [XEN][XEN] or Linux [KVM][KVM],
-just plain, or with [Eucalyptus][eucalyptus]. CloudBioLinux included
-software packages are fully customizable, and different flavours of
-CloudBioLinux can be configured.
-
-CloudBioLinux provides a [Fabric build file][3], written in Python.
-There is little need to understand Python, though, as most configuration
-is done through configuration files. Other scripting languages can be
-added too. Package selection is through YAML files in the ./config
-directory.
+CloudBioLinux is a build and deployment system which installs a large selection
+of Bioinformatics and machine learning libraries on a bare virtual machine (VM)
+image, freshly installed PC, or in the Cloud. By default CloudBioLinux includes
+a large suite of tools installed through the default distribution installer,
+native installers, and libraries for Perl, R, Python, Java and Ruby.
+
+CloudBioLinux included software packages are fully customizable. In addition to
+the default configuration, we support custom configuration builds through
+flavors. Flavors support overriding default package installations, making it
+simple to create derived installs for specific purposes.
+
+CloudBioLinux is a single install route both for desktop VMs such as
+[VirtualBox][v2], cloud providers such as [Amazon EC2][0] or desktop machines.
+This works equally well for other virtual machines and private cloud
+environments, including [XEN][XEN], Linux [KVM][KVM], [Eucalyptus][eucalyptus]
+and [Openstack][openstack].
# Using an instance
-CloudBioLinux has ready-made images for Amazon EC2 and VirtualBox.
-Also a repository of biological data is available.
-
-It should be noted, at least 13GB free space is required on the disk.
-
## Amazon
See the 'Getting Started with CloudBioLinux' guide on the
@@ -51,27 +26,87 @@ version for users familiar with Amazon is:
* Login to the [Amazon EC2 console][2].
* Click Launch Instance, and choose the latest CloudBioLinux AMI from
the [website][1] in the community AMI section (search for
- 'biolinux').
+ 'CloudBioLinux').
* After launching the instance, find the host details of
your running instance from the Instances section.
* Connect to your machine via ssh or VNC (using the Amazon PEM keys)
-To use bioligical data provided on Amazon, you need to use
-CloudBioLinux tools and fabric:
+# Installing CloudBioLinux on a local machine
-# Using CloudBioLinux tools and fabric
+The install process for CloudBioLinux is fully automated through a
+[Fabric build file][3] written in Python. The Fabric build files are useful for
+automating installation of scientific software on local systems as well as
+Amazon cloud servers. Everything is fully configurable through
+plain text YAML configuration files, and custom build targets allow installation
+of a subset of the total available packages.
## Installation
-1. On your (local) machine, install [Fabric][3]:
-
- sudo apt-get install python-setuptools python-dev
- sudo easy_install fabric pyyaml
-
-2. Retrieve the cloudbiolinux code base:
+Retrieve the CloudBioLinux code base and install libraries and dependencies:
git clone git://github.com/chapmanb/cloudbiolinux.git
cd cloudbiolinux
+ python setup.py build
+ sudo python setup.py install
+
+## Usage
+
+The basic usage specifies the hostname of a machine accessible via ssh:
+
+ fab -f fabfile.py -H localhost install_biolinux
+
+Fabric contains some other useful commandline arguments for customizing this to
+your environments:
+
+- `-c your_fabricrc.txt` -- Specify the path to a fabricrc configuration files.
+ This allows customization of install directories and other server specific details.
+ See the default `config/fabricrc.txt` for a full list of options.
+
+- `-u username` -- The username on the remote machine, overriding the default of
+ your current username.
+
+## Using flavors
+
+CloudBioLinux normally creates a full system for bioinformatics, but can be
+easily configured to install only a subset of tools through flavors:
+
+ fab -f fabfile.py -H localhost install_biolinux:flavor=my_flavor
+
+`my_flavor` can be the name of an existing flavor in `contrib/flavor` or the
+path to a directory with customization information. The files in your flavor
+directory replace those in the standard `config` directory, allowing replacement
+of any of the configuration files like `main.yaml` with customized copies.
+
+If you desire even more control, flavors allow custom python hooks. See
+`doc/hacking.md` for more details.
+
+## Customized targets
+
+You can substitute `install_biolinux` with more specific targets:
+
+* `install_biolinux:packages` -- Install all of the defined system
+ packages.
+* `install_biolinux:libraries` -- Install all libraries for various
+ programming languages.
+* `install_libraries:language` -- Install libraries for a specific
+ language.
+* `install_biolinux:custom` -- Install all custom programs.
+* `install_custom:a_package_name` -- Install a specific custom
+ program.
+
+## Custom package installs
+
+The custom directory contains installation instructions for programs that are
+not available from standard package repositories. These instructions are written
+in Python using the [Fabric][3] remote deployment tool and can also be used for
+installing individual packages locally on your machine. To do this, run:
+
+ fab -f fabfile.py -H localhost install_custom:your_package_name
+
+To build and install `your_package_name` on the local machine. We welcome
+additional custom bioinformatics package definitions for inclusion in
+CloudBioLinux. `custom/shared.py` contains a number of higher level functions
+which make it easier to write installation instructions.
## Biological data
@@ -105,7 +140,9 @@ to a local or [cloud-based][bd3] Galaxy server.
[bd2]: https://github.com/chapmanb/cloudbiolinux/blob/master/config/biodata.yaml
[bd3]: https://bitbucket.org/galaxy/galaxy-central/wiki/cloud
-## VirtualBox with vagrant
+# Supported virtual environments
+
+## Vagrant VirtualBox
Vagrant allows easy deploying and connecting to VirtualBox images. The
setup is ideal for runnig CloudBioLinux on a desktop computer.
@@ -139,11 +176,6 @@ virtualbox itself. For more information, see the BioLinux
[Vagrant documentation][doc], as well as the
documentation on the [Vagrant website][v1].
-# Building an image from scratch using CloudBioLinux
-
-CloudBioLinux can be built for any virtualized platform, as long as
-the target VM is Linux, with an ssh command line.
-
## Amazon
A bare Linux image launched in Amazon EC2 is configured from another
@@ -178,12 +210,7 @@ local virtual machine from scratch with CloudBioLinux.
[vb1]: https://github.com/chapmanb/cloudbiolinux/blob/master/doc/virtualbox.md
-## OpenStack
-
-[OpenStack][openstack] is a promising platform for Cloud computing.
-CloudBioLinux support for OpenStack is planned.
-
-## XEN/KVM/Eucalyptus private Cloud
+## OpenStack/XEN/KVM/Eucalyptus private Cloud
As long as there is an 'ssh' entry to an running VM, CloudBioLinux can
install itself.
@@ -203,81 +230,12 @@ For more on private Cloud and CloudBioLinux see ./doc/private\_cloud.md.
[eucalyptus]: http://open.eucalyptus.com/
[openstack]: http://www.openstack.org/
-# Technical details for using build scripts
-
-## Targets for local builds
-
-The Fabric build files are useful for automating installation of
-scientific software on local systems as well as Amazon cloud
-servers. There are a number of build targets that can be used to
-install a subset of the total available packages.
-
-The main build command is used as described above, but the host
-can point to a local machine or any server on your network:
-
- fab -f fabfile.py -H localhost -c config/fabricrc.txt install_biolinux
-
-The `config/fabricrc.txt` configuration file specifies install
-directories and other server specific details.
-
-With this basic command, you can substitute `install_biolinux` with
-serveral more specific targets:
-
-* `install_biolinux:packages` -- Install all of the defined system
- packages.
-* `install_biolinux:libraries` -- Install all libraries for various
- programming languages.
-* `install_libraries:language` -- Install libraries for a specific
- language.
-* `install_biolinux:custom` -- Install all custom programs.
-* `install_custom:your_package_name` -- Install a specific custom
- program.
-
-### Custom package installs
-
-The custom directory contains installation instructions for programs that are
-not available from standard package repositories. These instructions are written
-in Python using the [Fabric][3] remote deployment tool and can also be used for
-installing individual packages locally on your machine. To do this, run:
-
- fab -f fabfile.py -H localhost install_custom:your_package_name
-
-To build and install `your_package_name` on the local machine. We welcome
-additional custom bioinformatics package definitions for inclusion in
-CloudBioLinux. `custom/shared.py` contains a number of higher level functions
-which make it easier to write installation instructions.
-
-### Using flavors
-
-A Flavor is a specialization of an installation edition. A flavor can filter on
-packages and add packages and other software. To use a flavor, pass it
-in on the command line, e.g.
-
- fab -f fabfile.py -H localhost install_biolinux:flavor=my_flavor
-
-And, like with the other options, a flavor can also be defined in the
-config file. A command line parameter, however, is preferred.
-
-### Rolling your own
-
-BioLinux normally creates a full system for Bioinformatics. The tools
-provided here to create such an environment are rather generic. So, if you want to
-install packages on any system, be it desktop, server or VM, the
-BioLinux tool set can be used to role your own. The first step it to us
-install_biolinux with a named package list, e.g.
-
- fab -f fabfile.py -H localhost install_biolinux:packagelist=./contrib/minimal/main.yaml
-
-where install_biolinux is the fab entry point to a non-specific install,
-starting from the package list in the passed in packagelist. For more
-information see the section 'Rolling your own' in the BioLinux vagrant documentation in [./doc/vagrant.md][doc].
-
-## EC2 quickstart
+# EC2 quickstart
This provides a quick cheat sheet of commands for getting up and running on EC2 using
Amazon's command line tools.
-### Initial set up
+## Initial set up
The first time using EC2, you'll need to install the toolkit and credentials
for connecting on your local machine, following the [getting started guide][qs1].
@@ -318,7 +276,7 @@ Allow ssh and web access to your instances:
[qs2]: http://aws.amazon.com/account/
[qs3]: http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351&categoryID=88
-### Starting an instance
+## Starting an instance
Each time you'd like to use EC2, you need to create a remote instance to work
with; the [AWS console][4] is useful for managing this process.
View
5 cloudbio/flavor/__init__.py
@@ -8,8 +8,6 @@
Other (main) flavors can be found in this directory and in ./contrib/flavors
"""
-from fabric.api import *
-
class Flavor:
"""Base class. Every flavor derives from this
"""
@@ -27,5 +25,4 @@ def rewrite_config_items(self, name, items):
def post_install(self):
"""Post installation hook"""
-
-env.flavor = Flavor(env)
+ pass
View
25 cloudbio/flavor/config.py
@@ -0,0 +1,25 @@
+"""Handle alternative configuration file locations for flavor customizations.
+"""
+import os
+import collections
+
+def _find_fname(env, fname):
+ for dirname in [env.get("flavor_dir", None), env.config_dir]:
+ print dirname, fname
+ if dirname:
+ full_path = os.path.join(dirname, fname)
+ if os.path.exists(full_path):
+ return full_path
+ return None
+
+def get_config_file(env, fname):
+ """Retrieve YAML configuration file from the default config directory or flavor directory.
+
+ This combines all options for getting distribution or flavor specific customizations.
+ """
+ base, ext = os.path.splitext(fname)
+ distribution_fname = "{0}-{1}{2}".format(base, env.get("distribution", "notspecified"), ext)
+ Config = collections.namedtuple("Config", "base dist")
+ out = Config(base=_find_fname(env, fname), dist=_find_fname(env, distribution_fname))
+ return out
+
View
28 cloudbio/package/deb.py
@@ -4,8 +4,9 @@
from fabric.contrib.files import *
from cloudbio.package.shared import _yaml_to_packages
+from cloudbio.flavor.config import get_config_file
-def _apt_packages(to_install=None, pkg_list=None, pkg_config_file_path=None):
+def _apt_packages(to_install=None, pkg_list=None):
"""
Install packages available via apt-get.
Note that ``to_install`` and ``pkg_list`` arguments cannot be used simultaneously.
@@ -18,34 +19,15 @@ def _apt_packages(to_install=None, pkg_list=None, pkg_config_file_path=None):
:type pkg_list: list
:param pkg_list: An explicit list of packages to install. No other files,
flavors, or editions are considered.
-
- :type pkg_config_file_path: string
- :param pkg_config_file_path: Allows a custom path to be specified where
- ``packages.yaml`` and ``packages-[dist].yaml``
- are stored. Note that the file names cannot
- be customized, only the path.
"""
env.logger.info("Update the system")
sudo("apt-get update") # Always update
if to_install is not None:
- env.logger.info("Will install all packages (as listed in packages*.yaml "
- "files in {0})".format(pkg_config_file_path))
- if pkg_config_file_path is None:
- pkg_config_file_path = env.config_dir
- pkg_config_file = os.path.join(pkg_config_file_path, "packages.yaml")
- subs_pkg_config_file = os.path.join(pkg_config_file_path, "packages-%s.yaml" %
- env.distribution)
- if not os.path.exists(subs_pkg_config_file): subs_pkg_config_file = None
+ config_file = get_config_file(env, "packages.yaml")
env.edition.apt_upgrade_system()
- # Retrieve final package names
- (packages, _) = _yaml_to_packages(pkg_config_file, to_install,
- subs_pkg_config_file)
- # At this point allow the Edition to rewrite the package list -
- # this is shared within and between editions.
- # Ref: https://github.com/chapmanb/cloudbiolinux/pull/10#issuecomment-1616423
+ (packages, _) = _yaml_to_packages(config_file.base, to_install, config_file.dist)
+ # Allow editions and flavors to modify the package list
packages = env.edition.rewrite_config_items("packages", packages)
-
- # At this point allow the Flavor to rewrite the package list
packages = env.flavor.rewrite_config_items("packages", packages)
elif pkg_list is not None:
env.logger.info("Will install specific packages: {0}".format(pkg_list))
View
3 cloudbio/package/nix.py
@@ -4,6 +4,7 @@
from fabric.contrib.files import *
from cloudbio.package.shared import _yaml_to_packages
+from cloudbio.flavor.config import get_config_file
def _setup_nix_sources():
if env.nixpkgs:
@@ -41,7 +42,7 @@ def _nix_packages(to_install):
"""
if env.nixpkgs:
env.logger.info("Update and install NixPkgs packages")
- pkg_config_file = os.path.join(env.config_dir, "packages-nix.yaml")
+ pkg_config_file = get_config_file(env, "packages-nix.yaml").base
sudo("nix-channel --update")
# Retrieve final package names
(packages, _) = _yaml_to_packages(pkg_config_file, to_install)
View
3 cloudbio/package/rpm.py
@@ -4,6 +4,7 @@
from fabric.contrib.files import *
from cloudbio.package.shared import _yaml_to_packages
+from cloudbio.flavor.config import get_config_file
def _yum_packages(to_install):
"""Install rpm packages available via yum.
@@ -12,7 +13,7 @@ def _yum_packages(to_install):
package_file = "packages-scientificlinux.yaml"
else:
package_file = "packages-yum.yaml"
- pkg_config = os.path.join(env.config_dir, package_file)
+ pkg_config = get_config_file(env, package_file).base
with settings(warn_only=True):
sudo("yum check-update")
sudo("yum -y upgrade")
View
3 cloudbio/utils.py
@@ -2,7 +2,6 @@
"""
import logging
-from fabric.api import sudo
from fabric.colors import yellow, red, green, magenta
class ColorFormatter(logging.Formatter):
@@ -69,4 +68,4 @@ def _update_biolinux_log(env, target, flavor):
logfn = "/var/log/biolinux.log"
info = "Target="+target+"; Edition="+env.edition.name+"; Flavor="+flavor
env.logger.info(info)
- sudo("date +\"%D %T - Updated "+info+"\" >> "+logfn)
+ env.safe_sudo("date +\"%D %T - Updated "+info+"\" >> "+logfn)
View
0 contrib/flavor/ngs_pipeline/fabricrc-rpm.txt → contrib/flavor/ngs_pipeline/fabricrc.txt
File renamed without changes.
View
1 contrib/flavor/variantviz/fabricrc.txt
@@ -25,7 +25,6 @@ is_ec2_image = false
# Global installation directory for packages and standard programs
system_install = /usr
-config_dir=contrib/flavor/variantviz
# Local install directory for versioned software that will not
# be included in the path by default
View
12 doc/hacking.md
@@ -14,7 +14,7 @@ The Minimal edition is the smallest common denominator of all Editions, as it
installs the minimum of packages to bootstrap a full install. Once the
vagrant box is up and running, Minimal is invoked from the desktop by
- fab -f $source/fabfile.py -H target_hostname -c $source/contrib/minimal/fabricrc_debian.txt install_biolinux:packagelist=$source/contrib/minimal/main.yaml
+ fab -f $source/fabfile.py -H target_hostname -c $source/contrib/minimal/fabricrc_debian.txt install_biolinux:flavor=$source/contrib/minimal
where $source points to your biolinux source tree (replace 'target_hostname'
with 'vagrant' when using that). In fact, the testing script in
@@ -40,9 +40,9 @@ To expand on the package list you can define your own main.yaml, and pass that
in. In your main.yaml file add the meta-packages listed in
config/packages.yaml. Invoke your new package list with
- fab -f $source/fabfile.py -H target_hostname -c $source/contrib/minimal/fabricrc_debian.txt install_biolinux:packagelist=myproject/main.yaml
+ fab -f $source/fabfile.py -H target_hostname -c $source/contrib/minimal/fabricrc_debian.txt install_biolinux:flavor=/path/to/myproject
-It is that simple!
+where the `myproject` directory contains your main.yaml. It is that simple!
If packages.yaml is not complete, you may suggest changing its contents in the
main repository. The alternative is to create your own flavor, which we will do
@@ -71,7 +71,7 @@ statement. For example:
The flavor is itself is found through a fabricrc.txt file. The main package
list may be in a new main.yaml file. Kicking it into submission:
- fab -f $source/fabfile.py -H target_hostname -c $source/contrib/flavor/pjotrp/biotest/fabricrc_debian.txt install_biolinux:packagelist=$source/contrib/flavor/pjotrp/biotest/main.yaml
+ fab -f $source/fabfile.py -H target_hostname -c $source/contrib/flavor/pjotrp/biotest/fabricrc_debian.txt install_biolinux:flavor=$source/contrib/flavor/pjotrp/biotest
The flavor module itsefl sets env.flavor on loading the module (this can only
happen once). For more examples see the files in ./contrib/flavor.
@@ -161,7 +161,7 @@ a script by adding a post_install method to your flavor. E.g.
You can run post_install on its own (convenient for testing!) using the finalize
target, e.g.
- fab -H hostname -f $source/fabfile.py -c $flavor/fabricrc_debian.txt install_biolinux:packagelist=$flavor/main.yaml,target=finalize
+ fab -H hostname -f $source/fabfile.py -c $flavor/fabricrc_debian.txt install_biolinux:flavor=$flavor,target=finalize
(Note: finalize may become post-install in the future)
@@ -178,7 +178,7 @@ could add a parameter to the fabricrc file. Even better, add a command
line parameter named 'environment' to the install_biolinux parameter
list. E.g.
- fab -H hostname -f $source/fabfile.py -c $flavor/fabricrc_debian.txt install_biolinux:packagelist=$flavor/main.yaml,environment=special
+ fab -H hostname -f $source/fabfile.py -c $flavor/fabricrc_debian.txt install_biolinux:flavor=$flavor,environment=special
which automatically becomes part of the Flavor environment state as
'env.environment'. Use this parameter to distinguish between targets.
View
112 fabfile.py
@@ -42,23 +42,18 @@
from cloudbio.package.rpm import (_yum_packages, _setup_yum_bashrc,
_setup_yum_sources)
from cloudbio.package.nix import _setup_nix_sources, _nix_packages
+from cloudbio.flavor import Flavor
+from cloudbio.flavor.config import get_config_file
# ## Utility functions for establishing our build environment
def _parse_fabricrc():
"""Defaults from fabricrc.txt file; loaded if not specified at commandline.
"""
- # ## General setup
- if not env.has_key("config_dir"):
- env.config_dir = os.path.join(os.path.dirname(__file__), "config")
-
+ env.config_dir = os.path.join(os.path.dirname(__file__), "config")
if not env.has_key("distribution") and not env.has_key("system_install"):
env.logger.info("Reading default fabricrc.txt")
- config_file = os.path.join(env.config_dir, "fabricrc.txt")
- if os.path.exists(config_file):
- env.update(load_settings(config_file))
- else:
- env.logger.warn("Skipping fabricrc.txt as distribution is already defined")
+ env.update(load_settings(get_config_file(env, "fabricrc.txt").base))
def _create_local_paths():
"""Expand any paths defined in terms of shell shortcuts (like ~).
@@ -84,85 +79,68 @@ def _create_local_paths():
result = run("pwd")
env.local_install = result
-def _setup_flavor(flavor, environment=None):
- """Setup flavor
+def _setup_flavor(flavor):
+ """Setup a flavor, providing customization hooks to modify CloudBioLinux installs.
+
+ Specify flavor as a name, in which case we look it up in the standard flavor
+ directory (contrib/flavor/your_flavor), or as a path to a flavor directory outside
+ of cloudbiolinux.
"""
- if not flavor:
- flavor = env.get("flavor", None)
- if not environment:
- environment = env.get("environment", None)
- if environment:
- env.environment = environment
- env.logger.info("Environment %s" % env.environment)
+ env.flavor = Flavor(env)
+ env.flavor_dir = None
if flavor:
- # import a flavor defined through parameters flavor and flavor_path
- flavor_path = env.get("flavor_path", None)
- if flavor_path == None:
- raise ImportError("You need to define the flavor_path for %s!" % flavor)
- # Add path for flavors
- sys.path.append(os.path.join(os.path.dirname(__file__), "contrib", "flavor"))
- env.logger.info("Flavor %s loaded from %s" % (flavor, flavor_path))
- try:
- mod = __import__(flavor_path, fromlist=[flavor])
- except ImportError:
- raise ImportError("Failed to import %s" % flavor)
- else:
- # import default Flavor
- from cloudbio.flavor import Flavor
+ # setup the directory for flavor customizations
+ if os.path.isabs(flavor):
+ flavor_dir = flavor
+ else:
+ flavor_dir = os.path.join(os.path.dirname(__file__), "contrib", "flavor", flavor)
+ assert os.path.exists(flavor_dir), \
+ "Did not find directory {0} for flavor {1}".format(flavor_dir, flavor)
+ env.flavor_dir = flavor_dir
+ # Load python customizations to base configuration if present
+ py_flavor = "{0}flavor".format(os.path.split(os.path.realpath(flavor_dir)))
+ flavor_custom_py = os.path.join(flavor_dir, "{0}.py".format(py_flavor))
+ if os.path.exists(flavor_custom_py):
+ sys.path.append(flavor_dir)
+ mod = __import__(flavor_dir, fromlist=[py_flavor])
env.logger.info("This is a %s" % env.flavor.name)
# ### Shared installation targets for all platforms
-def install_biolinux(target=None, packagelist=None, flavor=None, environment=None,
- pkg_config_file_path=None):
- """
- Main entry point for installing BioLinux on a remote server.
-
- ``packagelist`` should point to a top level file (eg, ``main.yaml``) listing
- all the package categories that should be installed. This allows a different
- package list and/or use of Flavor. So you can say::
-
- install_biolinux:packagelist=contrib/mylist/main.yaml,flavor=specialflavor
+def install_biolinux(target=None, flavor=None):
+ """Main entry point for installing BioLinux on a remote server.
- ``pkg_config_file_path`` can be used to specify a path where a custom
- ``packages.yaml`` and ``packages-[dist].yaml`` are located, allowing fine-
- grained (i.e., individual package) customization. Otherwise, default
- to ``./contrib`` where the CBL files are defined.
-
- Both ``packagelist`` and ``flavor``, as well as the Edition, can also be
- passed in through the ``fabricrc`` file.
+ `flavor` allows customization of CloudBioLinux behavior. It can either
+ be a flavor name that maps to a corresponding directory in contrib/flavor
+ or the path to a custom directory. This can contain:
+
+ - alternative package lists (main.yaml, packages.yaml, custom.yaml)
+ - custom python code (nameflavor.py) that hooks into the build machinery
- target can also be supplied on the fab CLI. Special targets are:
+ `target` allows running only particular parts of the build process. Valid choices are:
- packages Install distro packages
- custom Install custom packages
- libraries Install programming language libraries
- post_install Setup CloudMan, FreeNX and other system services
- cleanup Remove downloaded files and prepare images for AMI builds
-
- ``environment`` allows adding additional information on the command line -
- usually for defining environments, for example ``environment=testing``, or
- ``environment=production``, will set the deployment environment and tune
- post-installation settings.
"""
_setup_logging(env)
time_start = _print_time_stats("Config", "start")
_check_fabric_version()
+ _setup_flavor(flavor)
_parse_fabricrc()
_setup_edition(env)
- _setup_flavor(flavor, environment)
_setup_distribution_environment() # get parameters for distro, packages etc.
_create_local_paths()
- env.logger.debug("Meta-package list is '%s'" % packagelist)
- env.logger.debug("File path for explicit packages is '%s'" % pkg_config_file_path)
env.logger.debug("Target is '%s'" % target)
- pkg_install, lib_install, custom_ignore = _read_main_config(packagelist) # read main yaml
+ pkg_install, lib_install, custom_ignore = _read_main_config()
if target is None or target == "packages":
if env.distribution in ["debian", "ubuntu"]:
_setup_apt_sources()
_setup_apt_automation()
_add_apt_gpg_keys()
- _apt_packages(pkg_install, pkg_config_file_path=pkg_config_file_path)
+ _apt_packages(pkg_install)
elif env.distribution in ["centos", "scientificlinux"]:
_setup_yum_sources()
_yum_packages(pkg_install)
@@ -219,7 +197,7 @@ def _check_fabric_version():
def _custom_installs(to_install, ignore=None):
if not exists(env.local_install):
run("mkdir -p %s" % env.local_install)
- pkg_config = os.path.join(env.config_dir, "custom.yaml")
+ pkg_config = get_config_file(env, "custom.yaml").base
packages, pkg_to_group = _yaml_to_packages(pkg_config, to_install)
packages = [p for p in packages if ignore is None or p not in ignore]
for p in env.flavor.rewrite_config_items("custom", packages):
@@ -247,11 +225,12 @@ def install_custom(p, automated=False, pkg_to_group=None):
p = p.lower() # All packages are listed in custom.yaml are in lower case
time_start = _print_time_stats("Custom install for '{0}'".format(p), "start")
if not automated:
+ _setup_flavor(None)
_parse_fabricrc()
_setup_edition(env)
_setup_distribution_environment()
_create_local_paths()
- pkg_config = os.path.join(env.config_dir, "custom.yaml")
+ pkg_config = get_config_file(env, "custom.yaml").base
packages, pkg_to_group = _yaml_to_packages(pkg_config, None)
try:
@@ -276,13 +255,12 @@ def install_custom(p, automated=False, pkg_to_group=None):
fn(env)
_print_time_stats("Custom install for '%s'" % p, "end", time_start)
-def _read_main_config(yaml_file=None):
+def _read_main_config():
"""Pull a list of groups to install based on our main configuration YAML.
Reads 'main.yaml' and returns packages and libraries
"""
- if yaml_file is None:
- yaml_file = os.path.join(env.config_dir, "main.yaml")
+ yaml_file = get_config_file(env, "main.yaml").base
with open(yaml_file) as in_handle:
full_data = yaml.load(in_handle)
packages = full_data['packages']
@@ -417,16 +395,16 @@ def install_libraries(language):
"""
_setup_logging(env)
_check_fabric_version()
+ _setup_flavor(None)
_parse_fabricrc()
_setup_edition(env)
- _setup_flavor(None)
_setup_distribution_environment()
_create_local_paths()
_do_library_installs(["%s-libs" % language])
def _do_library_installs(to_install):
for iname in to_install:
- yaml_file = os.path.join(env.config_dir, "%s.yaml" % iname)
+ yaml_file = get_config_file(env, "%s.yaml" % iname).base
with open(yaml_file) as in_handle:
config = yaml.load(in_handle)
lib_installers[iname](config)
View
2 setup.py
@@ -7,7 +7,7 @@
from setuptools import setup, find_packages
setup(name = "cloudbiolinux",
- version = "0.2a",
+ version = "0.3a",
author = "Brad Chapman",
author_email = "chapmanb@50mail.com",
description = "configure virtual (or real) machines with tools for biological analyses",
View
10 test/test_biolinux
@@ -36,19 +36,19 @@ CONFIG = <<EOCONFIG
---
Minimal:
:fabricrc: contrib/minimal/fabricrc_debian.txt
- :packagelist: contrib/minimal/main.yaml
+ :packagelist: contrib/minimal
Boinc:
# Boinc is a Minimal flavor for running Boinc clients
:fabricrc: contrib/flavor/boinc/fabricrc_debian.txt
- :packagelist: contrib/flavor/boinc/main.yaml
+ :packagelist: contrib/flavor/boinc
Phylogeny:
# Phylogeny is a Minimal flavor for running Phylogenetic software in the Cloud
:fabricrc: contrib/flavor/phylogeny/fabricrc_debian.txt
- :packagelist: contrib/flavor/phylogeny/main.yaml
+ :packagelist: contrib/flavor/phylogeny
:call_test: test_phylogeny_flavor
BioNode:
:fabricrc: contrib/bionode/fabricrc_bionode.txt
- :packagelist: contrib/bionode/main.yaml
+ :packagelist: contrib/bionode
EOCONFIG
config = YAML::load(CONFIG)
@@ -151,7 +151,7 @@ fab_opts += " -i #{$sshkey}" if $sshkey
fab_opts += " -H #{$hostname}"
fab_opts += ":#{$port}" if $port
-run "fab #{fab_opts} -f #{srcpath}/fabfile.py -c #{fabricrc} install_biolinux:packagelist=#{packagelist},environment=biolinux-test"
+run "fab #{fab_opts} -f #{srcpath}/fabfile.py -c #{fabricrc} install_biolinux:flavor=#{packagelist},environment=biolinux-test"
# Net::SSH.start($hostname, $user, :password => $pwd, :keys => [$sshkey], :port => $port) do |ssh|
Net::SSH.start($hostname, $user, :password => $pwd, :port => $port) do |ssh|
View
10 test/test_vagrant
@@ -24,26 +24,26 @@ BOX = {
{ :url => 'http://bio4.dnsalias.net/download/biolinux/biolinux-minimal-0.91.box',
:md5 => '64cdea1f76028971e53e2cd2f19a490a',
:fabricrc => 'contrib/minimal/fabricrc_debian.txt',
- :packagelist => 'contrib/minimal/main.yaml'
+ :packagelist => 'contrib/minimal'
},
'Boinc' => # Boinc is a Minimal flavor for running Boinc clients
{ :url => 'http://bio4.dnsalias.net/download/biolinux/biolinux-minimal-0.91.box',
:md5 => '64cdea1f76028971e53e2cd2f19a490a',
:fabricrc => 'contrib/flavor/boinc/fabricrc_debian.txt',
- :packagelist => 'contrib/flavor/boinc/main.yaml'
+ :packagelist => 'contrib/flavor/boinc'
},
'Phylogeny' => # Phylogeny is a Minimal flavor for running Phylogenetic software in the Cloud
{ :url => 'http://bio4.dnsalias.net/download/biolinux/biolinux-phylogeny-debian-0.92-i386.box',
:md5 => 'e7481104d24a7896ea1151a14e05d497',
:fabricrc => 'contrib/flavor/phylogeny/fabricrc_debian.txt',
- :packagelist => 'contrib/flavor/phylogeny/main.yaml',
+ :packagelist => 'contrib/flavor/phylogeny',
:call_test => 'test_phylogeny_flavor'
},
'BioNode' =>
{ :url => 'http://bio4.dnsalias.net/download/biolinux/biolinux-minimal-0.91.box',
:md5 => '64cdea1f76028971e53e2cd2f19a490a',
:fabricrc => 'contrib/bionode/fabricrc_bionode.txt',
- :packagelist => 'contrib/bionode/main.yaml'
+ :packagelist => 'contrib/bionode'
},
}
@@ -242,7 +242,7 @@ Dir.chdir(testname) do
print "We can use #{testname}!\n"
# Fire up the BioLinux fab file - this installs the VM
- run "fab -f #{srcpath}/fabfile.py -H vagrant -c #{fabricrc} install_biolinux:packagelist=#{packagelist},environment=biolinux-test"
+ run "fab -f #{srcpath}/fabfile.py -H vagrant -c #{fabricrc} install_biolinux:flavor=#{packagelist},environment=biolinux-test"
# Now set up Ruby's vagrant access for testing
vagrant = Vagrant::Environment.new

0 comments on commit 5e045f9

Please sign in to comment.