Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add icinga checks for email-alert-api content changes #10025

Merged
merged 2 commits into from Jan 9, 2020

Conversation

@edwardkerry
Copy link
Contributor

edwardkerry commented Jan 8, 2020

Two new Icinga checks, warning and critical, for more refined reporting on unprocessed content changes in email-alert-api. Alerts will trigger if either value exceeds 0.

This is part of work to slim down the email-alert-api-healthcheck-not-ok alert, making individual alerts which are not machine-specific. Data lives in stats/gauges/govuk/email-alert-api in Graphite.

Trello

host_name => $::fqdn,
target => 'transformNull(stats.gauges.govuk.email-alert-api.content_changes.warning_total)',
warning => '0',
critical => '100000000',

This comment has been minimized.

Copy link
@issyl0

issyl0 Jan 8, 2020

Member

Is this value so large because this is always a warning alert and the critical should never trigger from here? Is there a better way?

This comment has been minimized.

Copy link
@edwardkerry

edwardkerry Jan 8, 2020

Author Contributor

That is correct, there are existing comments in the file to this effect

  # We are only interested in the `warning` state but `critical` is also required
  # for valid Icinga check configuration. Setting it to a high value that won't be
  # reached allows us to get round this issue

I don't think we are currently aware of a better way, but are very open to ideas!

This comment has been minimized.

Copy link
@issyl0

issyl0 Jan 8, 2020

Member

I have no ideas. It was an observation!

This comment has been minimized.

Copy link
@deborahchua

deborahchua Jan 8, 2020

Contributor

@cbaines suggested we could make the critical and warning arguments optional in the Icinga check template configuration(I think!?) which means we wouldn't need to do this. Having seen the other two healthchecks we need to set up in a similar way maybe it's worth doing this?

This comment has been minimized.

Copy link
@cbaines

cbaines Jan 8, 2020

Contributor

I think that's still a possibility, but not quite worth doing yet. I think it would be good to see the first iteration of the extracted healthchecks in action for a bit, and have some data to work with, and then we'll be able to make more informed decisions.

@edwardkerry edwardkerry merged commit f3c80f4 into master Jan 9, 2020
1 check passed
1 check passed
continuous-integration/jenkins/branch This commit looks good
Details
@edwardkerry edwardkerry deleted the add-content-changes-icinga-check branch Jan 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.