Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to make 2 events from one body #2

Closed
Alexvianet opened this issue Nov 17, 2018 · 1 comment

Comments

@Alexvianet
Copy link

commented Nov 17, 2018

I got Body form Prometheus

{"receiver":"xmatters-receiver","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"BOSHJobEphemeralDiskFull","bosh_deployment":"concourse","bosh_job_index":"0","bosh_job_name":"worker-pcf-backup","bosh_name":"gp2-dev-infraservices","env":"dev-gp2","environment":"dev-gp2","service":"bosh-job","severity":"critical"},"annotations":{"description":"BOSH Job `dev-gp2/gp2-dev-infraservices/concourse/worker-pcf-backup/0` has used more than 80% of its ephemeral disk for 30ms: 90%","summary":"BOSH Job `dev-gp2/gp2-dev-infraservices/concourse/worker-pcf-backup/0` is running out of ephemeral disk"},"startsAt":"2018-11-16T09:02:16.538555134Z","endsAt":"2018-11-17T09:31:16.538555134Z","generatorURL":"https://prometheus-dev.gp2.axadmin.net/graph?g0.expr=avg+by%28environment%2C+bosh_name%2C+bosh_deployment%2C+bosh_job_name%2C+bosh_job_index%29+%28bosh_job_ephemeral_disk_percent%7Bbosh_deployment%21%3D%22bosh-health-check%22%2Cbosh_job_name%21~%22%5Ecompilation.%2A%22%7D%29+%3E+80\u0026g0.tab=1"},{"status":"firing","labels":{"alertname":"BOSHJobEphemeralDiskFull","bosh_deployment":"shield-v8","bosh_job_index":"0","bosh_job_name":"dedicated_backupnode","bosh_name":"gp2-dev-infraservices","env":"dev-gp2","environment":"dev-gp2","service":"bosh-job","severity":"critical"},"annotations":{"description":"BOSH Job `dev-gp2/gp2-dev-infraservices/shield-v8/dedicated_backupnode/0` has used more than 80% of its ephemeral disk for 30ms: 96%","summary":"BOSH Job `dev-gp2/gp2-dev-infraservices/shield-v8/dedicated_backupnode/0` is running out of ephemeral disk"},"startsAt":"2018-11-15T23:02:16.538555134Z","endsAt":"2018-11-17T09:31:16.538555134Z","generatorURL":"https://prometheus-dev.gp2.axadmin.net/graph?g0.expr=avg+by%28environment%2C+bosh_name%2C+bosh_deployment%2C+bosh_job_name%2C+bosh_job_index%29+%28bosh_job_ephemeral_disk_percent%7Bbosh_deployment%21%3D%22bosh-health-check%22%2Cbosh_job_name%21~%22%5Ecompilation.%2A%22%7D%29+%3E+80\u0026g0.tab=1"}],"groupLabels":{},"commonLabels":{"alertname":"BOSHJobEphemeralDiskFull","bosh_job_index":"0","bosh_name":"gp2-dev-infraservices","env":"dev-gp2","environment":"dev-gp2","service":"bosh-job","severity":"critical"},"commonAnnotations":{},"externalURL":"https://alertmanager-dev.gp2.axadmin.net","version":"4","groupKey":"{}/{alertname=~\"^(?:BOSHJobEphemeralDiskFull)$\"}:{}"}

and Xmatters create only one Event for it with:

Prometheus Alert: 2 firing for
[FIRING:2] BOSHJobEphemeralDiskFull bosh-job(alertname = BOSHJobEphemeralDiskFull bosh_deployment = concourse bosh_job_index = 0 bosh_job_name = worker-pcf-backup bosh_name = gp2-dev-infraservices env = dev-gp2 environment = dev-gp2 service = bosh-job severity = critical )

How to make it a separate events for each Alert?

@xMTinkerer

This comment has been minimized.

Copy link
Contributor

commented Nov 19, 2018

Hey @Alexvianet,
It looks like Alert Manager will group the alerts to help the over notification problem. So I think you have two options. 1. Unset grouping of your alert manager alerts, or 2. Update the script in the inbound integration on the xMatters side to iterate over the data.alerts array and build a new event for each one. This probably isn't a good idea if the alerts array contains more than say 50-100 entries though as the script execution is time boxed to 60s. And then you're also re-opening the over notification can of worms. We can deal with that using the event flood control option in xMatters, so I guess it just depends on where you want to stem the flow.

I did poke around the code on that inbound integration and it does only seem to include details about the first alert in the resulting notifications. So I suppose you could also expand that to include the first 5-10.

I'm not well versed in the prometheus side, which do you think is the most "prometheus-ic" option?

@Alexvianet Alexvianet closed this Aug 21, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.