Upgrade to Buildbot Nine

[aneeshusa]

Buildbot Nine is currently in beta (beta8 is due to be released in a week or so) and is getting close to a full release; we're currently running Buildbot Eight (0.8.12) and while not urgent, I'd like to upgrade when Nine is released. There are a significant number of changes in Nine, so it's not a drop-in replacement and we'll need to update our configuration.

I'd like to add more extensive (functional) test coverage to ensure a smooth upgrade; anything that has broken in the past would be a good candidate to add a regression test for. (I'm not sure exactly how Buildbot breaks when it does.)

This would also be a good opportunity to discuss switching to a different CI system since Buildbot is fragile, although I think Buildbot is fine and our biggest wins will come from moving configuration into the main Servo repo as much as possible.

[metajack]

I'm not opposed to a discussion of alternative CI, but what options exist besides Jenkins? Buildbot and Jenkins seem the best things out there for self-hosted stuff, and having used both I'm not sure I would switch from one to the other in either direction.

[larsbergstrom]

I'd love to see a world where we can rely completely on externally-hosted services (e.g., Travis and AppVeyor), but there are two things preventing it:

1. They need to support much larger instance types. We use c4.4xlarge on EC2, which is way bigger than the GCE instances either of them have
2. homu needs better status reporting/integration of multiple services

So, I'm tempted to stick with the buildbot we know for now, doing the upgrade that you've suggested, and limping along until some external services reach a larger scale point. We are not planning to scale Servo to a point internally where we'd have a team of multiple full-time engineers dedicated to keeping the system running, which is what I think we will end up needing (certainly, every other project does!) if we continue to scale on internally-hosted systems like buildbot, jenkins, etc.

This is pretty consistent with what @edunham and I have been talking about :-) She's responsible for systems across all of research, and Servo would quickly take up 100% of her time if we don't get some of it happening outside of our (virtual) datacenters.

I also agree that the more of it we move into the tree the better! Landing changes that span the source code and CI system are really awkward and require having an administrator push buttons/run scripts at exactly the right time.

[aneeshusa]

Status update: Buildbot 9 was released on October 6, 2016: https://docs.buildbot.net/0.9.0.post1/relnotes/index.html

We've made progress and now read steps from the servo/servo tree, but have had to write custom code for this which has caused some bumps along the way: #515, #524, etc. Buildbot 9 has built in support for dynamic steps and a rewritten (nicer) web UI, which both sound enticing to me. However, I'm not confident that attempting an upgrade wouldn't end up causing more problems than it's worth right now without a better testing strategy. Many of the problems we had would have essentially required standing up a separate Buildbot instance to discover (as opposed to static analysis), e.g. Windows behavior differences or log file naming.

[aneeshusa]

Also from the release notes, it seems like the Buildbot APIs will still be in flux until 1.0.0: https://docs.buildbot.net/0.9.0.post1/relnotes/index.html#master

[aneeshusa]

Closing in favor of #559.