We're gating on Android

[Ms2ger]

Somehow #539 got deployed before its time.

On servo-master1, I cloned 70fe510 into /tmp/salt-testing-root/saltfs and ran salt 'servo-master1' state.highstate 2>&1. Waiting to see if that helped.

[Ms2ger]

CC @aneeshusa @larsbergstrom @metajack @Manishearth

[Ms2ger]

CC @edunham

[Ms2ger]

That seems to have helped as a temporary band-aid.

Some notes for the post-mortem:

• I didn't find docs on how saltfs code is deployed now; if that had existed I could probably have reverted in the repo and followed the docs
• https://github.com/servo/servo/wiki/Buildbot-administration still mentions /srv/salt/, which no longer exists
• We landed #539 before we were able to deploy it, putting saltfs master in a known-bad state
• There is no log of deploys (that I could find), which would have made it easier to verify that a bad deploy had happened
• There is no obvious / documented way to revert a deployment
• The changes were deployed, apparently without someone watching that everything still worked after
• No issue was filed when the problem was first discovered

[Manishearth]

Bad deploy was mine; I was testing out the intermittent tracker stuff. It's much harder to deploy from branch now, though.

[larsbergstrom]

It's my fault for landing the Android gating assuming it would be "ready to go shortly" and then taking forever to get mozjs building on Android.

We do try to log (see https://github.com/servo/servo/wiki/Buildbot-administration#applying-changes), but there's no default verbose / logged mode for salt highstates, at least that I know how to turn on. It's something we've definitely wanted!

[aneeshusa]

#549 covers documentation, and #437 tracks logging, which is addressed by #547. We're still gating on Android but recent builds are all green, so I'm closing this.