Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resolution notifications dependency on group_interval #1660

Open
kapodes opened this issue Dec 12, 2018 · 3 comments
Open

Resolution notifications dependency on group_interval #1660

kapodes opened this issue Dec 12, 2018 · 3 comments

Comments

@kapodes
Copy link

kapodes commented Dec 12, 2018

What did you do?
Sent an alert and waited for it to hit resolve_timeout and be resolved

What did you expect to see?
Notification about alert being resolved right away after it was resolved according to Alertmanager

What did you see instead? Under which circumstances?
Nothing until group_interval timer hit.

alertmanager, version 0.15.3 (branch: HEAD, revision: d4a7697cc90f8bce62efe7c44b63b542578ec0a1)
  build user:       root@4ecc17c53d26
  build date:       20181109-15:40:48
  go version:       go1.11.2
  • Alertmanager configuration file:
global:
  resolve_timeout: 1m
route:
  group_wait: 5s
  group_interval: 5m
  receiver: slack-notifications

receivers:
  - name: "slack-notifications"
    slack_configs:
      - channel: "#alerts"
        send_resolved: true
        ..........
  • Logs:
level=debug ts=2018-12-12T15:36:34.910107275Z caller=dispatch.go:445 component=dispatcher aggrGroup={}:{} msg=Flushing alerts=[TestAlert[c508c62][active]]
level=debug ts=2018-12-12T15:41:34.910626655Z caller=dispatch.go:445 component=dispatcher aggrGroup={}:{} msg=Flushing alerts=[TestAlert[c508c62][resolved]]

If I decrease group_interval to something like 5s I will see notification almost instantly after alert is resolved.

I think send_resolved logic shouldn't be dependent on group_interval.
It's logical not to send new alert notifications from same group right away but resolution messages are not new alerts...

@brian-brazil
Copy link
Contributor

The idea of group_interval is to throttle all notifications from the group to avoid spamming the oncall, whether they be firing or resolved notifications.

@kapodes
Copy link
Author

kapodes commented Dec 12, 2018

Consider updating docs to mention resolution notifications are also affected.
It took me some time to understand what is happening.

# When the first notification was sent, wait 'group_interval' to send a batch
# of new alerts that started firing for that group.
group_interval: 5m

@mxinden
Copy link
Member

mxinden commented Dec 13, 2018

@kapodes would you like to prepare a patch to improve the docs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants