Disaster detection and disabled actions

[ssalinas]

In a critical situation it is helpful to limit the amount of task churn in Singularity. This PR adds the ability for an admin to globally disable certain actions. So far it is implemented for BOUNCE, DEPLOY, SCALE, REMOVE, and DECOMMISSION but it's easy to add more if needed.

a POST to /disasters/disabled-actions/{action} adds an action to the list of ones that are disabled with an optional message
a DELETE to /disasters/disabled-actions/{action} removes it from that list

Singularity will respond with a 423 (locked) and the message given when disabling (or a default message)

In this PR I am also adding an automated way of disabling actions based on things such as task lag and the frequency of lost slaves or lost tasks.

TODO for this PR:

• Add a test endpoint to explicitly trigger task reconciliation
• Add the ability to globally disable actions of certain types
• Admin ui page to add/remove disabled actions
• Add disaster detection around things like number of overdue tasks, slaves lost, etc, that will run on the leader only
• Allow the disaster detection to disable certain actions when one of its thresholds is crossed
• Email admins when a disaster is detected
• Tune metrics appropriately to decide when something bad is happening
• Add an api endpoint to manually enter/exit disaster mode
• Admin ui page to manually enter/exit disaster mode and show current stats
• Docs

/cc @tpetr

[ssalinas]

@tpetr @wolfd @Calvinp feel free to tear apart my react if I'm doing anything silly here. Modeled this page off of the webhooks one since it's fairly similar and pretty simple.

[Calvinp]

It would be nice to exclude already-disabled actions from this list.
(It would also be nice to be able to select multiple. But I don't think we have a multiselect component in FormModal yet, and I don't know if the backend even supports that.)

[ssalinas]

Yeah, I didn't write the backend to support multiple yet, might be a good idea at some point. However, it does support overriding the message for one that is already present (i.e if you issue a new request for disabling bounces when they are alreayd disabled, it will take the message form the newer one). So I might leave it be for the moment

[ssalinas]

@tpetr Updated the PR a bit to take the time range over which events are occurring into account better. - task lag takes into account how long the calculated value has been over the specified threshold. A disaster will trigger if it has been over for a certain amount of time (45s default)

• For lost tasks/slaves, the poller will take all lost items from the last X seconds into account (30s by default) instead of just checking an instantaneous value. Now the interval on which the poller runs does not matter as much (though things get more accurate the more often it does)
• Poller now keeps more data points (10 by default)
• Eliminated the call to get all pending tasks for efficiency (just getting the IDs) so now the poller can run more often as well.

[ssalinas]

@tpetr added a more comprehensive SingularityAction enum, each action in the list has a property to determine if it can be disabled or not (i.e. if I actually put the validator code in place to make it disableable)

[tpetr]

can we word this better? canDisable maybe?

[tpetr]

this snuck in from merge conflicts