Servo needs nightly builds

[larsbergstrom]

We need nightly builds for Servo. Initial work is going on here: https://github.com/servo/servo-nightly/

[aneeshusa]

Can we just add a cron job to the Salt states to kick off a build every night, or will this be more involved? Is there a reason to do it on Travis instead?

[larsbergstrom]

I was thinking of having a cron job from Salt that creates a tag and would cause a build to kick off every night.

We could also do the builds on our buildbot buildslaves, but we would need to have special buildbot steps to package up the outputs and S3 or GitHub release API tokens to push the builds out to the shares.

The main reason I like doing some of this stuff on Travis is that other people can make edits to add more resources, platforms, etc. Changes to Buildbot are super fragile, both to get working (and often involve a bit of round-and-back-again on the live servers since it's so hard to set up a full Vagrant farm to test them) and for admins to push out changes. Because if we push out a buildbot update while anything is currently happening on homu, the events get lost and the build has to be manually retried.

[aneeshusa]

Creating a new tag every night doesn't sound scalable, but I understand wanting to use Travis; can we just force-push to a certain branch instead like homu does with auto?

I'm +1 on a cron job to kick off these builds. I still think it would be nice to converge the saltfs repo with the nightly builds, so the nightly builds on Travis use a subset of the Salt configurations to get all the pre-requisites installed, and then also run the Servo builds.

[larsbergstrom]

Oooh, I like the force-push to a nightly branch idea way better. And definitely doing that as a cron job, managed by saltfs, makes me happy. As much as I like all of the random external services we use, our nightly builds of Servo should be something we have a little more directly control over.

[Ms2ger]

Does that mean we would not archive nightly builds at all? I'm not at all convinced I'll be able to build today's nightly this time next year, and having the nightlies archived would be very useful for bisecting regressions.

[larsbergstrom]

@Ms2ger No, we'd definitely still archive nightlies (unlike what we do for Android, which is just a hack I threw together). I think the big question is:

1. Do we put them in an S3 bucket
2. Do we create a 'git release' and put them up on GitHub every night
3. {Something else}

It's something we are scheduled to solve in Q2, but we could certainly do some of it now :-)

CC @edunham

[edunham]

Why do we need to keep every nightly from this stage of development? If we expect to only need a given nightly for weeks or months and nobody else's systems are gating on their availability, I'd be comfortable with publishing them as releases. If we expect to need them years from now, S3 will give us better options for moving old artifacts to cheaper but slower-to-retrieve storage. Also, if we know of plans to use the nightlies in anyone else's CI systems, s3's availability is significantly more reliable than GitHub's.

[aneeshusa]

Somewhat off-topic: how important is being able to checkout any commit and reliably build that version/nightly? Nix is a great build tool that I'm a huge fan of which makes this possible. Its Windows support is a bit spotty due to lack of developers, and it isn't going to supplant Mach/Cargo any time soon for Servo, but if reliable historical builds are important to you it may be valuable to open an issue to discuss it.

[Ms2ger]

If Servo still exists in ten years from now, I definitely expect us to want to bisect years into the past; I believe @bzbarsky ends up doing that somewhat regularly with Gecko.

I believe it's important that we can keep building old versions, but it doesn't take much to break that, so I also want the binaries backed up.

[larsbergstrom]

I agree with @Ms2ger that the binaries and source code are the more critical bits. There are certain pieces of the dependency chain (e.g., Apple does not provide XCode downloads or versions of their OS for all time) that will make it impossible to exactly reproduce the binary. That said, certainly better logging of the installed versions of libraries (particularly static ones!) that were built against in the nightly build might be helpful. Particularly since much of that is not available in our default build logs, due to Cargo's suppression of native build script outputs.

[larsbergstrom]

Related: #9918

[paulrouget]

This is something we will need if we ever want to release a developer preview of servo+bhtml.
Any update on this?

[aneeshusa]

If we want to use Travis for this, one option is using a service like https://nightli.es/. I'm not sure about how to securely provide S3 upload credentials, though.

Buildbot also has a builtin Nightly scheduler that we could use for this, similar to the existing (but currently busted) android-nightly builder; it should be straightforward to define a few additional nightly builders. This also means a cron job should not be needed. I prefer this option.

Not sure if this is affected by integrating with Gecko CI.

[larsbergstrom]

We've now launched nightly builds!