Use yaml for buildbot steps #325

In b7fc797, we disabled blocking on gonk; since we're not using the results, removing the builder will use less resources.

We should be running pkill (or the platform-specific equivalent) at the start and end of each build, but this is really a CI detail, not something that is integral to the list of steps, so remove it for now.

The steps gzipping build logs did not have logfiles listed, so they were not available for download from the web interface. The only way to get the logs is via ssh/scp, which works just as well for non-gzipped logs. This will cut down on build times.

The existing mechanism for chaining environment dictionaries is a hack that Guido has noted as abuse of the '**' mechanism: https://mail.python.org/pipermail/python-dev/2010-April/099459.html Create a subclass that allows 'adding' in a single expression, as copy() + update() takes two lines. Use str.join for listing PATHs to keep lines short. This also exposed an error in the Windows PATH, with a missing semicolon between r'C:\msys64\usr\bin' and r'C:\Windows\system32'. Move the environments into a separate file to keep them all together. Rename the various environment variables for clarity and consistency. The environments.py file now passes flake8; this should be added to our test suite.

In order to move the build steps to the main servo repository, they must be converted from python code that requires the Buildbot library to be installed (to manually build steps) into more declarative steps. This commit starts this conversion by using heuristics to convert a list of shell commands (akin to those in a .travis.yml or .appveyor.yml file) into the appropriate Buildbot Steps. The doc factory and windows factory still use hardcoded steps; see the comments in the code for more details. The android nightly factory now uses a relative instead of absolute path for the .apk file to make the Jinja templating unnecessary.

This automatically runs pkill for each build. It is cleaner to put this in the main configuration instead of the lists of steps because this is a CI-related cleanup task, not a necessary step to run the build + test suite. Also, this ensures that the pkill step cannot be forgotten when adding new builders. Note: this doesn't work on Windows yet.

This was used a long time ago for bors, but Buildbot now reports status indirectly via Homu.

Also rename LINUX_RESERVED_SLAVES to LINUX_SLAVES, since all of our builders are now on reserved instances.

As a stepping-stone to having builder steps live in the main Servo repo, switch to a simple data format to express builder configuration (lists of strings accessed via yaml.safe_load) instead of needing to run or import arbitrary untrusted Python code. Currently, the YAML file is still part of the master configuration, but it is now ready to be moved to the main Servo repo.

Use linux-dev to run all lints because it is the fastest builder; these are platform-independent. Ensure that the lockfile_changed.sh and manifest_changed.sh scripts are run for every builder, since these are platform-dependent. Note: these should also be added for the Windows builder. This is also meant as an example of how much easier a YAML file makes it to edit our build scripts; note the lack of needing to know about `logfiles` or other parameters which have been forgotten in the past.

This test requires the flake8 python module; to ensure it is installed, add a requirements.txt file. Travis-CI will automatically install the dependencies specified in that file.

Currently, the only other python files are the test suite.

servo / saltfs

Use yaml for buildbot steps #325

Use yaml for buildbot steps #325

Commits on Apr 20, 2016