Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Configurable Alerting threshold to avoid false positive alerts #66

Closed
tlvu opened this issue Aug 28, 2020 · 0 comments · Fixed by #187
Closed

Configurable Alerting threshold to avoid false positive alerts #66

tlvu opened this issue Aug 28, 2020 · 0 comments · Fixed by #187
Assignees
@tlvu
Labels
enhancement New feature or request

Comments

@tlvu
Copy link
Collaborator

tlvu commented Aug 28, 2020

Too much false positive alerts will decrease the importance and usefulness of each alert. Alerts should not feel like spams.

Not all deployments are equal so can not have same threshold.

Have to basically use variables instead of hardcoding threshold values in this file https://github.com/bird-house/birdhouse-deploy/blob/cf055ccf7c06439de36aba5686758ff37db2d864/birdhouse/components/monitoring/prometheus.rules.template

Now that each component can have their own localized defaults (PR #64) we can implement this. This was left out during the initial implementation since this would literally spam the global default file.
@fmigneault fmigneault added the enhancement New feature or request label Jan 22, 2021
tlvu added a commit that referenced this issue Aug 3, 2021
@tlvu
monitoring: make some prometheus alert threshold configurable via env…
6efefd3 
….local

Default values are previous hardcoded values.

Different organizations with different policies and hardware can now
adapt the alert threshold to their specific needs, decreasing false
positive alerts.

Too much false positive alerts will decrease the importance and
usefulness of each alert. Alerts should not feel like spams.

Fixes #66.
@tlvu tlvu mentioned this issue Aug 4, 2021
Merged
@tlvu tlvu closed this as completed in #187 Aug 31, 2021
tlvu added a commit that referenced this issue Aug 31, 2021
@tlvu
Merge pull request #187 from bird-house/configurable-alerting-threshold
b9d4ce2 
monitoring: make some prometheus alert threshold configurable via env.local

Default values are previous hardcoded values so this is fully backward compatible

Different organizations with different policies and hardware can now
adapt the alert threshold to their specific needs, decreasing false
positive alerts.

Too much false positive alerts will decrease the importance and
usefulness of each alert. Alerts should not feel like spams.

Not all alerts are changed to make configurable.  I've only changed those that I think are most likely to need customization or that logically should be configurable.

Fixes #66.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tlvu tlvu

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
2 participants
@tlvu @fmigneault