[SEV] 41 minutes of downtime due to bad AMI rollout

[raxod502]

See incident history on Statuspage.io:

• https://riju.statuspage.io/incidents/3szq1k00bhsf
• https://riju.statuspage.io/incidents/vpq61g78nyx2

The root cause appears to have been #86. I've isolated an affected node so that we can investigate the behavior later, but in the meantime I rolled back the autoscaling group to the old AMI and reverted the code change in #88.

Things that could have gone better generally:

• Mitigation would have been an order of magnitude faster if I had remembered that I could just scp a new supervisor binary to the running instance instead of spinning up a new one (twice!). We may want to prepare a runbook for future availability incidents.
• There were a lot of unnecessarily manual operations here and I did screw up some of the configuration for the new node I added to the ASG at first. This screw-up led me to mistakenly think that my manual operations might have been responsible for the outage, rather than the recent AMI change. Add tooling to roll ASG with new AMI #87 would solve this problem.

[raxod502]

Manual operations are no longer needed to roll the autoscaling group due to my alternate resolution of #78 (place all ASG instances in a single AZ).