Add icinga checks for email-alert-api content changes

[edwardkerry]

Two new Icinga checks, warning and critical, for more refined reporting on unprocessed content changes in email-alert-api. Alerts will trigger if either value exceeds 0.

This is part of work to slim down the email-alert-api-healthcheck-not-ok alert, making individual alerts which are not machine-specific. Data lives in stats/gauges/govuk/email-alert-api in Graphite.

Trello

[issyl0]

Is this value so large because this is always a warning alert and the critical should never trigger from here? Is there a better way?

[edwardkerry]

That is correct, there are existing comments in the file to this effect

  # We are only interested in the `warning` state but `critical` is also required
  # for valid Icinga check configuration. Setting it to a high value that won't be
  # reached allows us to get round this issue

I don't think we are currently aware of a better way, but are very open to ideas!

[issyl0]

I have no ideas. It was an observation!

[deborahchua]

@cbaines suggested we could make the critical and warning arguments optional in the Icinga check template configuration(I think!?) which means we wouldn't need to do this. Having seen the other two healthchecks we need to set up in a similar way maybe it's worth doing this?

[cbaines]

I think that's still a possibility, but not quite worth doing yet. I think it would be good to see the first iteration of the extracted healthchecks in action for a bit, and have some data to work with, and then we'll be able to make more informed decisions.