Monitoring: add alert rules and alert handling (deduplicate, group, route, silence, inhibit) #59
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Basic AlertManager config is provided with extensions via env.local env var. Prometheus is configured to talk to Alertmanager. Alerting rules in Prometheus will be for another commit.
…ntly of prometheus
… and notifications sent
Found a bunch of ready made alert rules, disabled those that did not seems to parse properly. The rest did not assess if they work correctly.
…e to our scrape_interval of 60s
…oint filter was wrong anyways, should have been mountpoint="/"
…t was wrong anyways, should have been mountpoint="/"
…tency rate range, probably due to our scrape_interval of 60s
… and node-exporter are local since same hostname
…kernel 4.10.15 and newer
…bleErrorsDetected, HostNetworkReceiveErrors, HostNetworkTransmitErrors
Yml syntax error, need to espace double-quote in {{ printf \"%.0f\" $value }}.
… added by copy/paste
…ts added by copy/paste
…r so will never exceed threshold so will never fire
…not a container (missing "name" label)
When memory limit not set (added container_spec_memory_limit_bytes > 0) and when not a container (missing "name" label).
…mporary containers spawned for the auto deployment
… new threshold from 1 day observation on prod, might need tweaking again later
tlogan2000
approved these changes
Jul 9, 2020
Switching from .md format to .rst format for automated table of content generation.
To have all components documentation at the same place.
|
Tagged |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a follow up to the previous PR #56 that added the monitoring itself.
Ping @dbyrns @matprov you might be interested in this.
Added cAdvisor and Node-exporter collection of alert rules found here https://awesome-prometheus-alerts.grep.to/rules with a few fixing because of errors in the rules and tweaking to reduce false positive alarms (see list of commits). Great collection of sample of ready-made rules to hit the ground running and learn PromML query language on the way.
Added Alertmanager to handle the alerts (deduplicate, group, route, silence, inhibit). Currently the only notification route configured is email but Alertmanager is able to route alerts to Slack and any generic services accepting webhooks.
This is an initial attempt at alerting. There are several ways to tweak the system without changing the code:
ALERTMANAGER_EXTRA_INHIBITIONviaenv.localfile.env.local:ALERTMANAGER_EXTRA_GLOBAL, ALERTMANAGER_EXTRA_ROUTES, ALERTMANAGER_EXTRA_RECEIVERS.What more could be done after this initial attempt:
Possibly add more graphs to Grafana dashboard since we have more alerts on metrics that we do not have matching Grafana graph. Graphs are useful for historical trends and correlation with other metrics, so not required if we do not need trends and correlation.
Only basic metrics are being collected currently. We could collect more useful metrics like SMART status and alert when a disk is failing.
The autodeploy mechanism can hook into this monitoring system to report pass/fail status and execution duration, with alerting for problems. Then we can also correlate any CPU, memory, disk I/O spike, when the autodeploy runs and have a trace of previous autodeploy executions.
I had to test these alerts directly in prod to tweak for less false positive alert and to debug not working rules to ensure they work on prod so these changes are already in prod ! This also test the SMTP server on the network.
See rules on Prometheus side: http://pavics.ouranos.ca:9090/rules, http://medus.ouranos.ca:9090/rules
Manage alerts on Alertmanager side: http://pavics.ouranos.ca:9093/#/alerts, http://medus.ouranos.ca:9093/#/alerts
Part of issue #12