Buildbot plans
The plan is for the buildbot master hardware to be installed in the week beginning 23/04/2012. I have little info about the exact configuration of this machine - more details may follow next week.
- The buildbot master can also function as a build slave. Most of the shorter tests will probably be run on this machine.
- CX1 - it is possible to set up a slave on CX1 that submits jobs using qsub. This should be used for all the short tests (to make sure they compile/work properly on CX1) and also for longer and performance-sensitive tests.
At present a buildbot that has been used for testing is triggered by Github - Github posts a JSON payload describing each commit to a public-facing webpage that then records the payload and posts it to the buildbot web interface - this is done since it is not possible to expose the Buildbot's web admin port externally.
This setup is probably undesirable for the new buildbot because it creates security and potentially reliability issues. The Fluidity buildbot presently polls an email mailbox for messages the describe each commit - it is similarly possible to set Github to send emails to an email box that the new buildbot may poll in order to trigger builds
Ideally it is possible to run a significant subset of tests on the local machine, outside the Github/buildbot infrastructure. This can be done by incorporating the tests into scripts that are invoked by the buildbot. One can also invoke these scripts locally, so you could get an idea if you've broken anything before you even commit. It is a good idea to run the local tests before committing.
Tests will generally be run each time someone pushes commits to the master branch of any OP2 repository. If this causes a test failure, it will be the responsibility of the person who pushed the offending commits to fix it immediately.
It should also be possible to set up temporary queues on the buildbot for testing branches - this will facilitate people being able to test their branches before they are merged into master. It will be sensible to mandate that branches cannot be merged into master unless they have already passed all the tests - this will minimise test failures on the master branch.
A record of all the output of every test and the outcome of the test is maintained automatically by the buildbot. This is cross-referenced with the commit that the tests were run for. One can access this history through a web interface that displays the current state of all the tests as well as all their past states.
Non-Imperial users will need to SSH tunnel into Imperial to be able to view this (or VPN). However, we can also have the buildbot announce failures in the IRC channel or by email, so you wouldn't need to keep checking the web interface to find out if something you'd done had caused a problem.
In order to actually set these up as tests, we need to establish a list of commands that can be run to invoke each test, and to determine whether the test has passed or failed. The buildbot will determine if a test has passed or failed by examining the return code of each of the steps in the build process - if any of these are non-zero, the test will be considered to have failed. For example, in order to test the building of the most current revision of the OP2 sequential library and the C apps, we might expect to have a list of steps like:
git clone git@github.com:OP2/OP2-Common.git
cd OP2-Common/op2/c
./cmake.local
cd ../../apps/c
./cmake.local
There are a few environment variables that need to be set for most of the OP2 build processes - this is not a problem since we can specify exactly the values of the environment variables on a test-by-test basis.
This is a vague outline of the tests that could be run - please feel free to suggest more:
- Successful generation of the core library by the Matlab/octave scripts.
- Successful building of the OP2-Common sequential library in C and the fortran interface.
- Successful building of the apps.
- Successful building of the ROSE translator.
- Successful building of the code generated by the ROSE translator.
- Successful building of the code generated by the MATLAB/Octave translator.
- Successful execution of the generated code.
- Check that there is no trailing whitespace in the source files, and that there are no tabs in source files except for in Makefiles.
Mike has also expressed an interest in testing for performance regressions of the translated GPU code. These tests should be run on CX1 in order to have exclusive access to a node to obtain reliable performance results.
The following tests will be run to check for memory leaks:
- Running of the airfoil sequentially:
valgrind --log-file=valgrindout --leak-check=full ./airfoil_seq
- Running the airfoil with CUDA (details to follow)
- Running the airfoil with OpenMP
- Running the airfoil with MPI:
export LD_PRELOAD=/home/gihan/valgrind/lib/valgrind/libmpiwrap-amd64-linux.so
mpirun -np 12 valgrind --log-file=valgrindout --leak-check=full ./airfoil_mpi
There will need to be some checking of the log file with an additional buildbot step that runs a script to check the output.
If you know the set of steps for a test and would like it added to the Buildbot, please open an issue with a title like "Add test for xxx", describing in the issue what the test is testing, and the steps for executing the test.