Skip to content

Continuous Integration

Jump to bottom
Rob Norris edited this page Oct 4, 2018 · 4 revisions

Continuous Integration

OCS3 was formerly built with Travis CI but the project quickly became too big for the small build machines Travis CI provides. Our new build setup uses BuildKite which is set up as described below.

Build Machine

We use our own build hardware, namely sbfocsdev-lv1.cl.gemini.edu , which is a VM.

  • 16G of RAM
  • 4 Xeon Processors
  • Not hooked up to OpenDirectory. If you need an account ask ISG.
  • Very limited local storage, don't use it.
  • Essentially infinite mounted storage on /gemstorage. Use that instead.

The build machine is running the following software:

  • Docker, for running isolated processes. The build uses Docker to create isolated Postgres instances required for testing. The docker user owns this process, and anyone who wishes to use it must be part of the docker group.
  • Buildkite Agent, for running builds. Builds run as the buildkite-agent user. If you sudo su buildkite-agent and then cd you will find the builds directory where everything happens.

BuildKite

Builds are coordinated by BuildKite. If you wish to see build details you will need to sign up and join the gemini-hlsw user group (Rob can send you an invitation). We have complimentary usage due to our education/open-source status.

The workflow is as follows:

  • There is a Web Hook in the gemini-hlsw/ocs3 repository that notifies BuildKite whenever anyone pushes a branch or opens a PR. In response BuildKite will assign a build to the next available agent. Right now we have only one build machine and thus one agent, but in the future we could have many. This would allow us to do many builds in parallel.
  • The agent running on the build server periodically polls for work, so we don't need to open a port. When it receives a job it checks out the repo and executes the "pipeline" we have defined on BuildKite.
  • When the build terminates, the agent reports status back to BuildKite, which then reports it to GitHub.

Now, one issue is that since we're running on our own hardware, inside the firewall, it's a potential security risk because a PR runs arbitrary code (via the build script and in tests). For this reason we have added an Agent Hook in /etc/buildkite-agent/hooks/environment that immediately fails the build if the PR comes from anyone other than us.

Build Pipeline

The BuildKite Pipeline for OCS3 is trivial. It simply runs build/buildkite-build.sh, which you can run locally if you wish, if you have Docker, node, yarn, and sbt.

Miscellania

If buildkite-agent gets hung for some reason you can restart it with

sudo systemctl restart buildkite-agent

Images are still not cleaned up in all cases for some reason. If you run out of disk space run

docker image prune -af

It will take a while to run.

Clone this wiki locally