tracking issue: instance restart behavior during upgrade

[davepacheco]

I'll start with the action plan and add a comment later with some discussion.

Short-term (R17):

• reduce cooldown period from 1 hour to 5 minutes ([nexus] decrease auto-restart cooldown to 5 minutes #9097)

These are longer-term goals that aren't specific enough to make tasks yet:

• Instances should not be restarted at all for system upgrades. We have long planned to use live migration to avoid this.
• Whether we do live migration or use instance restarts, we could make allocation choices more intelligently to minimize the number of instance movements required. (e.g., prefer to move instances to sleds that have already been updated). This is much harder than it sounds. See RFD 564.
• Even when we have to restart instances to move them, we could do so in the same coordinated way that we plan to use live migration for. (Roughly: we've discussed having the update system mark a sled as needing evacuation, avoid putting new instances there, and then waiting for evacuation to happen. The plan is to do that evacuation with live migration, but all of this could also be done with ordinary VM restarts, too.) This would leverage the same work and also make sure that we don't cooldown instances when they fail because of the upgrade.
• Instances should not be cooled down for "start" failures that can't be its fault (e.g., failure to start on a sled due to the sled not having sync'd time, or not having U2 devices, etc.). @jgallagher is filing a separate issue on this shortly. This isn't really upgrade-related but we hit it during upgrade testing and it contributed to instance unavailability.

[davepacheco]

Several of us discussed this in a recorded meeting this morning.

We started by going over a case during our upgrade testing on dublin where an instance was bounced as part of the update and then immediately was cooling down, not restarted. This is described in #9095 and #9096.

The long-term goals are in the description above. Here I'll summarize the discussion of the short-term ideas.

---

Change the default cooldown period from 1 hour to 5 minutes. The 1-hour cooldown is really aimed at an unlikely (but important) case where an instance is somehow taking out the whole sled that it's on (e.g., inducing an OS panic). This would of course be a severe bug, but such bugs can exist and the system should limit the blast radius. By waiting an hour to restart these instances, there's only so much damage they can do. However, we've not seen a bug like this and always expect it to be very rare. The upside of this intervention (tuning it down) is that it's very cheap to implement and makes the upgrade impact on instance availability pretty much match what we expected in R17. None of the other options was easy or lower risk. (And if we're to compare risks: the risk of significant instance downtime in R17 without this is virtually certain and pretty high impact.) As we do the long-term work mentioned above, we can tune the cooldown back up to mitigate this risk again.

Have the update system somehow clear the cooldown period for instances that were running when a sled was rebooted for upgrade. Or relatedly: mark sled as "updating" while we update it and if an instance fails in that state, don't cooldown. It's not as easy as it sounds to keep track of this state. To give an example: suppose reconfigurator did update a sled property or rendezvous table with this information. (First, that'd probably be an extra planning step, since we'd want to know it succeeded before proceeding with the update. We'd also want to keep some state to reflect that if we previously did this, then we should update that sled in the next planning step.) When could it clear it? The correct answer would have to be "after it knows that all instance_watcher background tasks have finished processing the affected instances". How would it know that? We could instead try to save timestamps before and after the update so that the instance watcher could figure out if an update was ongoing. But suppose a sled was rebooted between T1 and T2: that doesn't mean a failure detected at T3 > T2 wasn't due to the update -- in fact, we probably wouldn't notice the instance wasn't running until after the sled update was finished. I just outline these to give a flavor for why it's tricky, not to say it can't be done.

@jgallagher made the important observation that this latter approach wouldn't fix the update problem unless we also fixed #9096, whereas the proposed change to the cooldown by itself is enough for R17.