Administration

Cian Wilson edited this page Apr 11, 2017 · 7 revisions

TerraFERMA Home | Wiki Home

This page contains instructions on commonly undertaken administrative tasks related to maintaining TerraFERMA on our (primarily Ubuntu) systems. Currently these mostly pertain to adding new tests to buildbot and managing commonly used environment modules as well as a few useful git commands.

Git

We primarily use git for our version control. The git home page provides links to some excellent documentation for using it. Here we just cover some basic commands that come in useful occasionally.

Bitbucket to Github

In July 2015 we transitioned the public TerraFERMA repository from bitbucket to github (though some of our dependencies repositories remain on bitbucket as they themselves are forks and are therefore easier to maintain there). If you cloned TerraFERMA before this you will want to switch over to github as we don't plan on maintaining the TerraFERMA bitbucket mirror. To do this you can simply reclone from the new repository location:

git clone git@github.com:TerraFERMA/TerraFERMA.git

or, if you have local modifications you wish to preserve you can adjust the remote on your current clone using the command:

git remote set-url origin git@github.com:TerraFERMA/TerraFERMA.git

This assumes that you cloned directly from bitbucket and hence have the standard origin remote. Modify the command as necessary if this isn't true. After adjusting the remote url fetch or pull as normal:

git fetch

or

git pull

to get the latest changes from the new repository.

Buildbot

The buildbot consists of a buildmaster and multiple buildslaves.

For TerraFERMA the buildmaster is run on a desktop computer called sreabhan, which also provides the buildbot web interface at http://sreabhan.ldeo.columbia.edu:8010. The buildmaster is run from the buildbot account on sreabhan. All configuration files for this account are backuped in the buildbot-home repository. In addition we maintain accounts for the buildbot on github and bitbucket under the username ldeobuildbot.

Any machine (including virtual machines) can be configured to be a buildslave. These are generally also configured with a buildbot account and are added to the buildbot following the instructions below. The buildslaves are where the builds are actually run before the results are communicated back to the buildmaster. Hence our supported platforms are all buildslaves.

Adding a new builder or buildslave to the TerraFERMA buildbot can be accomplished using the instructions below. These can require sudo permissions and shouldn't be attempted if you don't know what you're doing or without backing up the buildbot configuration first (see above).

New Virtual Machines

Acknowledgement: The following instructions were originally written by Tim Greaves for the AMCG wiki.

Some of the TerraFERMA buildbot queues are run on a series of KVM (Kernel Virual Machine) instances. These run a variety of operating systems (see supported platforms). The following instructions describe setting up one of these virtual machines.

Initial Setup

Most machines so far are run on a desktop machine, sreabhan, where the initial setup has already been performed. If running on another machine it will be necessary to ensure that the following (Ubuntu) packages are installed:

sudo apt-get install kvm libvirt-bin ubuntu-vm-builder bridge-utils virt-viewer virtinst

Creating a Virtual Machine

Partitioning

On sreabhan the virtual machines are installed on individual partitions of the disk. To create a new partition:

sudo lvcreate -L 32G -n vm_<vm name> sys

where the virtual machine name needs updating.

Set up a filesystem on the new partition:

sudo mkfs -t xfs /dev/mapper/sys-vm_<vm name>

and make a directory to mount it:

sudo mkdir /kvm/<vm name>

Edit /etc/fstab by adding a line:

/dev/mapper/sys-vm_<vm name> /kvm/<vm name> xfs defaults 0 2

so that it is automatically mounted and then mount the filesystem:

sudo mount /kvm/<vm name>

Installation

Download a disk image (ISO) of the operating system you wish to install and use:

sudo virt-install --connect qemu:///system -n <vm name> -r 2048 \
                  -f <vm name>.qcow2 -s 32 -c <iso file> \
                  --vnc --noautoconsole --os-type <type> \
                  --os-variant <variant> --accelerate \
                  --hvm --network=network:default --force

to install it. The options used above correspond to:

--connect qemu:///system      # add to the system KVM instance
-n <vm name>                  # name of the new VM 
-r 2048                       # RAM
-f <vm name>.qcow2            # name of disk image
-s 32                         # 32 GB disk
-c <iso file>                 # ISO file
--vnc                         # use VNC to connect graphically
--noautoconsole               # don't automatically connect a console
--os-type <type>              # OS type (probably linux)
--os-variant <variant>        # OS variant 
--accelerate                  # use acceleration
--hvm                         # OS is virtualization-aware
--network=network:default     # default network from KVM
--force                       # 'yes' to any virt-install question

See man virt-install for a list of valid os-type and os-variant options.

Connect to the console using:

sudo virt-viewer <vm name>

which will interactively give you access to the install process. Typically we use the username buildbot. After installation is complete reboot the virtual machine.

You can list the status of the system virtual machines by connecting:

sudo virsh --connect qemu:///system

and listing them:

virsh list --all

which will produce output like:

 Id    Name                           State
----------------------------------------------------
 29    centos5.10_64bit               running
 -     centos6.2_64bit                shut off
 -     centos6.5_64bit                shut off
 -     ubuntu12.04_32bit              shut off
 -     ubuntu12.04_64bit              shut off
 -     ubuntu13.10_64bit              shut off
 -     ubuntu14.04_64bit              shut off

Start the new virtual machine:

virsh start <vm name>

and reconnect to it:

virt-viewer <vm name>

Install open ssh server:

sudo apt-get install openssh-server

where we have assumed an Ubuntu machine. Set up the virtual machine with a static IP address on the 192.168.122.* network (netmask: 255.255.255.0, gateway: 192.168.122.1, DNS server: 192.168.122.1) that isn't 192.168.122.1 or any of the others already defined in the host machine's (most likely sreabhan) /etc/hosts file.

Once this is set up reboot the virtual machine and add it to the host machine's /etc/hosts file as:

192.168.122.X

It should now be possible to ssh into the new virtual machine from the host machine using:

ssh buildbot@<vm name>

New Buildslaves

On a new buildslave machine, either a new machine or a new virtual machine, install the buildslave package:

sudo apt-get install buildbot-slave

where we have assumed an Ubuntu machine. The buildmaster (sreabhan) is currently running buildbot 0.8.8 with twisted 13.0.0. Make sure the buildslave is compatible with this.

On the buildslave we typically create a new directory:

cd ~
mkdir slaves/<slave name> -p

where the slave name should be entered.

Then create the buildslave in that directory:

cd slaves/<slave name>
buildslave create-slave . sreabhan.ldeo.columbia.edu:9989 <slave name> <password>

This will create a buildslave that communicates with the buildmaster (sreabhan) on port 9989 with slave name <slave name> and password <password>.

It's useful (and necessary on virtual machines) to configure the buildslave to start at startup. On linux machines this is done by editting `/etc/default/buildslave':

SLAVE_RUNNER=<path to buildslave>                     # most likely /usr/bin/buildslave

# NOTE: SLAVE_ENABLED has changed its behaviour in version 0.8.4. Use
# 'true|yes|1' to enable instance and 'false|no|0' to disable. Other
# values will be considered as syntax error.

SLAVE_ENABLED[1]=1                                    # 1-enabled, 0-disabled
SLAVE_NAME[1]="buildslave #1"                         # short name printed on start/stop
SLAVE_USER[1]="buildbot"                              # user to run slave as
SLAVE_BASEDIR[1]="/home/buildbot/slaves/<slave name>" # basedir to slave (absolute path)
SLAVE_OPTIONS[1]=""                                   # buildbot options
SLAVE_PREFIXCMD[1]=""                                 # prefix command, i.e. nice, linux32, dchroot

and making sure that the <path to buildslave> matches that provided for the SLAVE_RUNNER in /etc/init.d/buildslave.

It is also necessary to have appropriate ssh keys so that the buildslave can access repositories on github and/or bitbucket. These can either be generated using:

ssh-keygen -t rsa -N <password>

and then registered on the github and/or bitbucket ldeobuildbot accounts or, more simply, copied from the buildbot account on the buildmaster (sreabhan):

scp buildbot@sreabhan.ldeo.columbia.edu:~buildbot/.ssh/id_rsa* ~buildbot/.ssh/.

On the buildmaster (sreabhan), you must edit the ~buildbot/masters/tferma/master.cfg to tell it about the new buildslave. Find the list of slaves below c['slaves'] = [] and append:

c['slaves'].append(LibVirtSlave('<slave name>', '<password>', \
                                Connection( 'qemu:///system' ), \
                                '/kvm/<slave name>/<slave name>.qcow2', \
                                max_builds=2, build_wait_timeout=2*60, \
                                properties={'os':'<os type>','arch':'<architecture>'}))

for a buildslave on a virtual machine on the buildmaster with slave name and password matching those used to the create the buildslave on the virtual machine above. <os type> should be set to a string describing the os type (e.g. ubuntu12.04 or centos6.5). This is used in master.cfg so needs to be in quite a specific (but poorly defined!) format. If the os type has been used before copy that. Otherwise master.cfg may need modifying anyway but start by trying <os name><major version>.<minor version> if available. <architecture> should be 32bit or 64bit, most likely the latter. max_builds controls the number of builds that can run on the buildslave simultaneously and build_wait_timeout defines how long a virtual machine will remain powered on after a build completes in case new build requests are expected.

If adding a buildslave on another physical machine append instead:

c['slaves'].append(BuildSlave('<slave name>', '<password>', \
                              max_builds=2, \
                              properties={'os':'<os type>','arch':'<architecture>'}))

with similar instructions as for a virtual machine buildslave.

Once added, reconfigure the buildmaster (on sreabhan) using the command:

buildbot reconfig ~buildbot/masters/tferma

If adding a buildslave on a physical machine then you can now start the buildslave on that machine:

buildslave start ~buildbot/slaves/<slave name>

You can now edit ~buildbot/masters/tferma/master.cfg on the buildmaster (sreabhan) to add builders.

New Builders

The ~buildbot/masters/tferma/master.cfg script provides functions for setting up builders for all TerraFERMA dependencies and related projects. These are most easily understood by examples in the script but below we illustrate typical usage for them.
Additional arguments are available for many of the functions (e.g. for triggering follow-up builds or testing merges). All functions return (at least) a buildbot builder object that should be appended to the builders list. After modifying this list the buildbot will need to be reconfigured using:

buildbot reconfig ~buildbot/masters/tferma

All builders are automatically forceable through the buildbot web interface (provided sufficient privileges are available) but if new builders need to be automatically triggered then they should also be added to the schedulers list with an appropriate buildbot scheduler type (e.g. Nightly, Triggerable etc.).

New builder names must be unique and are used to cross reference builders that depend on each other. Typically they following the naming convection <camel case project name>-<repository owner>_<repository name>_<branch name>, where hyphens are normally dropped from the various names (e.g. TerraFERMA-tferma_tferma_dolfinmaster). The base FEniCS builder name is automatically suffixed with the subproject name (e.g. FEniCS-tferma_fenics_tfermamaster-dolfin).

Dorsal

dorsalrepo = ['<dorsal repository url>', '<dorsal branch>']
builders.append(mk_dorsal_builder(dorsalrepo, '<slave name>', '<dorsal builder name>'))

SPuD

builders.append(mk_spud_builder('<spud repository url>', '<slave name>', '<spud builder name>', test=True))

PETSc

petscrepo = ['<petsc repository url>', '<petsc branch>']
builders.append(mk_petsc_builder(petscrepo, '<slave name>', '<petsc builder name>', debug=False))

FEniCS

repos = { 
          'ufl'     : ['<ufl repository url>', '<ufl branch>'],
          'instant' : ['instant repository url', 'instant branch'],
          'fiat'    : ['fiat repository url', 'fiat branch'],
          'ffc'     : ['ffc repository url', 'ffc branch'],
          'dolfin'  : ['dolfin repository url', 'dolfin branch'],
        }
dolfinbuilder, tmpbuilders, tmpschedulers = mk_fenics_builders(repos, '<slave name>', '<fenics base builder name>', 
                                                               petscbuildername='<petsc builder name>', dolfintest=True,
                                                               debug=True, custom_swig=False)
builders = builders + tmpbuilders
schedulers = schedulers + tmpschedulers
builders.append(dolfinbuilder)

TerraFERMA

tfermarepo = ['<terraferma repository url>', '<terraferma branch>']
builders.append(mk_tferma_builder(tfermarepo, '<slave name>', '<terraferma builder name>',
                      '<fenics base builder name>', petscbuildername='<petsc builder name>', spudbuildername='<spud builder name>',
                      test=True, mediumtest=False, debug=True))
schedulers.append(Triggerable(name=builders[-1].name, builderNames=[builders[-1].name]))

TerraFERMA Utilities

tfermautilsrepo = ['<terraferma utilities repository url>', '<terraferma utilities branch>']
builders.append(mk_tferma_utils_builder(tfermautilsrepo, '<slave name>', '<terraferma utilities builder name>'))
schedulers.append(Triggerable(name=builders[-1].name, builderNames=[builders[-1].name]))

Benchmarks

benchmarksrepo = ['<benchmarks repository url>', '<benchmarks branch>']
builders.append(mk_tferma_benchmarks_builder(benchmarksrepo, '<slave name>', '<benchmarks builder name>',
                '<terraferma utilities builder name>', tfermabuildername='<terraferma builder name>', 
                fenicsbuildername='<fenics base builder name>', petscbuildername='<petsc builder name>',
                spudbuildername='<spud builder name>', debug=False, buildtest=False, validationtest=True, 
                testfolders=["<benchmark folder>"]))
schedulers.append(Nightly(name=builders[-1].name, branch=None, builderNames=[builders[-1].name], dayOfWeek=0, hour=6))

Environment Modules

Environment modules are very useful for running TerraFERMA as it depends on numerous environment variables. To ease this we have a environment-modules repository that contains examples of commonly used environment modules on our systems. Below we provide our typical usage instructions of these modules on Ubuntu machines.

Install the environment-modules package:

sudo apt-get install environment-modules

Add the following lines to /etc/bash.bashrc:

#!bash

# enable environment modules
if [ -f /usr/share/modules/init/bash ]; then
    . /usr/share/modules/init/bash
elif [ -f /usr/share/Modules/init/bash ]; then
    . /usr/share/Modules/init/bash
fi
if [ "x$MODULESHOME" != "x" ]; then
    module use --append /usr/local/share/environment-modules
fi

to /etc/bash.bashrc, (optionally) using the command:

echo -e '\n# enable environment modules\nif [ -f /usr/share/modules/init/bash ]; then\n    . /usr/share/modules/init/bash\nelif [ -f /usr/share/Modules/init/bash ]; then\n    . /usr/share/Modules/init/bash\nfi\nif [ "x$MODULESHOME" != "x" ]; then\n    module use --append /usr/local/share/environment-modules\nfi\n' | sudo tee -a /etc/bash.bashrc > /dev/null

which will append those lines to the end of the file but note that it is often better to insert this by hand before the activation of bash completion (so that module commands can be auto completed too).

Clone this git repository into /usr/local/share/environment-modules:

sudo mkdir -p /usr/local/share/environment-modules
cd /usr/local/share
sudo chgrp adm environment-modules
sudo chmod g+w environment-modules
git clone git@github.com:TerraFERMA/environment-modules.git

Modules should now be available on next login.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.