metrics: alert based on SLIs

[cirocosta]

Hey,

At the moment, we have our SLIs (see Datadog SLI Dashboard) being displayed in a monitor in the office, but this does not necessarily mean that the operators (Test Pilot Pair) end up acting on problems that happen there, as the pair would need to keep looking at it.

By tying those SLIs with PagerDuty, we're then able to treat degradations in those numbers as triggers for action.

To better reflect the fact that an alert is generated, we should also update the way that the colors are set there (such that it's immediate for someone to see that there's an active incident).

In terms of "which thresholds to use", let's start with an SLO of daily 95 (which means 1h12m of downtime), and adjust that accordingly.

Acceptance criteria

• With a daily SLI being hurt, a page is generated to PagerDuty
• Have Datadog colors representing our objectives.

Thanks!

[scottietremendous]

We should have follow up issues to continually raise the SLOs.

[cirocosta]

@pivotal-bin-ju 🙌

[cirocosta]

Whoops, it turns out that we were performing the evaluations in a 1m interval (which is pretty much a 99.93% daily).

Now we're also including error budgets that tell us how much time we have left to explore aiming at that 99% monthly: