Compare: Buildbot administration

Salt

We use SaltStack to configure our build master and the slaves. The sources are at: https://github.com/servo/saltfs/

Homu (previously Bors)

Homu is the service that watches our PRs for approvals and shepherds them into the buildbot queue. Its sources are at: https://github.com/barosl/homu

It runs.

The queue of jobs can be seen at: http://build.servo.org/homu/queue/all

Buildbot Builders

Their status can be viewed at: http://build.servo.org/buildslaves

Buildbot log files can be found on the individual machines at: /home/servo/buildbot/(master|slave)/twistd.log

If you need access, create a PR against https://github.com/servo/saltfs/, including your account in the common/map.jinja file and SSH pubkey in the common/ssh folder. To access the machines, log in as root on Linux or Administrator on OSX; there are no individual accounts on slaves. If you need to test something (e.g., a reftest failure) make sure to su - servo to simulate the space.

IP addresses

servo-mac1: 208.52.161.130
servo-mac2: 208.52.161.128
servo-mac3 63.135.170.19
servo-master: 96.126.125.232
servo-linux1: 104.237.128.153
servo-linux-android1: 72.14.176.110
servo-head: 96.126.114.185

Linux builders are Linode 8GB machines, with 8GB of RAM and 6 cores. The OSX builders are MacStadium quad core mac minis with 8GB of RAM. The Linux "head" builder is a Linode 4GB machine with 4GB of RAM and 4 cores.

There also are two EC2 machines (linux1, linux2) that are used for the Linux builds, but they do not have stable IPs. The IPs can be found in the buildbot log, via grep Broker /home/servo/buildbot/master/twistd.log | tail -n 10 (last two unique IPs, usually).

Setting up a new build slave

Linux

⚠️ Setting up a master requires additional steps, these instructions only set up a minion.

Install the Salt minion:

# curl https://raw.githubusercontent.com/servo/saltfs/master/.travis/install_salt | sh -s linux

Configure and start the Salt minion:

# echo '96.126.125.232 salt' >> /etc/hosts
# printf 'servo-linux1' > /etc/salt/minion_id # replace servo-linux1 with the hostname of the machine
# service salt-minion restart

OSX

⚠️ These instructions for installing Salt are out of date. (They need to be synced up with recent updates in the saltfs repo, and in particular currently install a version of Salt on OS X that is broken.)

install brew: 
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
install latest Xcode command line tools from https://developer.apple.com/downloads
hdiutil attach command_line_tools_for_osx_mountain_lion_april_2014.dmg 
sudo installer -pkg /Volumes/Command\ Line\ Tools\ \(Mountain\ Lion\)/Command\ Line\ Tools\ \(Mountain\ Lion\).mpkg -target /
brew install git
 echo export PATH='/usr/local/bin:$PATH' >> ~/.bash_profile
logout and back in
brew doctor (should say you are good to go)
sudo easy_install -s /usr/bin pip
brew install saltstack
curl -O https://raw.githubusercontent.com/saltstack/salt/develop/pkg/darwin/com.saltstack.salt.minion.plist
add     <key>EnvironmentVariables</key>
    <dict>
        <key>PATH</key>
        <string>/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
        <key>HOME</key>
        <string>/Users/administrator</string>
    </dict>
right after the Label section
sudo salt-run launchd.write_launchd_plist salt-minion
sudo cp com.saltstack.salt.minion.plist /Library/LaunchDaemons/
sudo bash -c "echo '96.126.125.232 salt' >> /etc/hosts"
edit /etc/salt/minion and set `id: servo-mac$N` where $N is the number of the slave
sudo launchctl load -w /Library/LaunchDaemons/com.saltstack.salt.minion.plist
install xcode 5.1.1 from developer.apple.com
sudo xcodebuild -license (and agree)
set screen resolution to 1280x960 or else reftests will fail randomly
Create a standard user named 'Tom Servo', give him the same password as admin below, and log in. 
Set up to autologin on reboot. LEAVE LOGGED IN (else reftests will fail)

OSX OpenSSL

We currently dynamically link against OpenSSL, which is amazingly old on OSX. To update it, do:

brew update
brew install openssl
brew link --force openssl

If openssl version still reports a very old version (i.e., one that isn't the same as homebrew just installed), you will need to override a final symlink:

sudo rm /usr/bin/openssl
sudo ln -s /usr/local/Cellar/openssl/VERSION/bin/openssl /usr/bin/openssl

OSX maxfiles

The default OSX maxfiles setting is too small for our web platform tests runs and needs to be increased, as below. This may need to be re-performed after OS upgrades (even dot-releases), as Apple tends to reset all the things.

echo 'kern.maxfiles=60000\nkern.maxfilesperproc=20000\nkern.ipc.somaxconn=2048' | sudo tee -a /etc/sysctl.conf
echo -e 'limit maxfiles 8192 20480\nlimit maxproc 1000 2000' | sudo tee -a /etc/launchd.conf
echo 'ulimit -n 10240' | sudo tee -a /etc/profile

Enabling a build slave for the first time

On master use salt-key -L to list pending minon keys. Then use salt-key -a KEY to accept new minion keys.

Verify by logging in to the master and doing salt '*' test.ping. Should see results from all minons.

After adding it to common.sls, do salt servo-NEWMACHINE state.highstate to install of the dependencies.

Finally, add it as appropriate to the buildbot master.cfg file and then run salt servo-master state.highstate to restart buildbot. Check the /home/servo/buildbot logs to ensure it started property and can see the new machine.

File configuration notes on servo-master

file_roots:
  base:
    - /srv/salt
    
pillar_roots:
  base:
    - /srv/pillar

credentials need to go into /srv/pillar/

Adding another build flavor

bors/bors.cfg needs to know about the build flavor (its string name from master.cfg below)homu???
buildbot/master/master.cfg should add another type of build, hooked up in all the same ways as e.g., "mac1"
salt servo-master state.highstate

Making changes

Make changes as needed, and test them using Vagrant. A Vagrantfile is included in the servo/saltfs repo. Each node type has a corresponding Vagrant target, and Vagrant will run the Salt states when provisioning. Run vagrant up (or you can specify just one target, e.g. vagrant up servo-master), and wait for the highstate to run. You can login to the VM with e.g. vagrant ssh servo-master if you need to debug.

💨 vagrant-cachier is recommended to make running highstates faster by caching downloaded packages. The Vagrantfile will automatically take advantage of it: run vagrant plugin install vagrant-cachier to install it.

⚠️ Vagrant's Salt provisioner is currently broken, awaiting two pull requests: #1, #2. You can apply these patches to your local Vagrant installation if you wish until they get accepted and released upstream.

Once you're happy, git commit your changes, git push them up to a fork of the saltfs repo on Github, and open a pull request.

Discouraged

It's sometimes easiest to iterate on changes directly on servo-master.servo.org. If you make local edits to the state tree (/srv/salt/), make sure you push them upstream:

# git commit -am 'Integrated new cowsay feature'
# git push origin master

Applying changes

Log into the machine and do:

# cd /srv/salt
# git fetch origin
# git merge --ff-only origin/master

Always verify the changes to be applied with

# salt servo-master state.highstate test=True

Then, if homu is not running anything, update all the machines with:

# salt servo-master state.highstate

Afterwards, always verify that the services are working by tailing the homu and buildbot logs:

# less +F /var/log/upstart/homu.log
# less +F /home/servo/buildbot/master/twistd.log

Seekrits

Secret passwords and account information are stored at the following, secured location: https://docs.google.com/document/d/1bJfq47eGfipX0R-S6rwe8InVNM7TspwlBZiPD7pRvDo/edit

Kicking buildbot

For a variety of ailments:

# service buildbot-master restart

If a build is aborted due to a lost client machine, first go to the waterfall view (http://build.servo.org/waterfall), then click on the build that was aborted. Finally, click on the Rebuild button.

servo / servo