No description, website, or topics provided.
Clone or download
Pull request Compare This branch is 177 commits ahead, 58 commits behind jdswinbank:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.



Build scripts for the Uni. Southampton Astronomy Dept. Lofar cluster (Hence, S.A.L. build scripts)

A collection of scripts for building various radio astronomy software packages, including casacore/casarest/pyrap, the LOFAR imaging pipeline, and the LOFAR-TKP transients detection pipeline.

Linux build system engineers might also find some useful snippets here, since it employs some slightly circuitous bash scripts to effect continuous deployment practices, despite the packages being built from a rag-tag collection of C++ and python. While slightly hacky, this is a very lightweight and dependency-free alternative to say, creating a virtual machine for each build.


The most generally applicable routines are contained in, including snippets for forcing fresh checkouts of git repositories, and grabbing their id hashes.

Many of the scripts are simply convenience routines for downloading and building packages with a single command (the grab_install_X routines). These make use the master variables set in CONFIG to determine the install target.

Scripts named in CAPS are generally more complex, doing things like updating repositories and then building to target folders named using the svn/git hashes.

Using versioned builds in turn allows for 'continuous deployment' practices, wherein the latest of each the package builds is collected together to create a 'buildset' - this is done by collecting symlinks pointing to particular builds into one master set of bin/lib/pythonpath directories, archived via the date and time at which the set was collected. These master directories can then be added to the environment variables to make use of a given buildset. To make this a non-intrusive process for end-users, we manually update a symlink to the current desired buildset, and end-users simply run an environment setup script sourced via that symlink. This also allows multiple buildsets to be used concurrently, for purposes of testing, stability and backwards compatibility - simply set the environment variable (e.g. PREF_SOFT_BUILD) pointing to the desired buildset before sourcing the init script that adds the relevant directories to the environment path variables.

The symlinking process is a bit arcane, but has been refined to resolve intermediate symlinks to ensure that once a buildset has been defined, it will stay fixed and working unless the original build folders are deleted. A pleasant side effect is that the symlinked programs in a buildset are never too many layers of symlinks from their final path entry; one can determine the exact build of a program in use via the command:

readlink $(which <someprogram>)

You can find the code that collects and resolves symlinks in

Note that python packages that prefer installation via a .pth file (e.g., pyrap) don't fit well in this system, as a .pth file cannot simply be added to the PYTHONPATH. Fortunately, it's pretty simple to script the unpacking of a .pth file to a regular python package folder, at least in this case. See for the relevant bash snippet.

See CONFIG for the master variables defining where repositories / tarballs are downloaded to, and where builds are installed to.

See for a typical command sequence when installing from scratch on a fresh Ubuntu 12.04 system.

To Do

  • needs updating.