PagerDuty Notifier Broken #127
Description
Overview
The current PagerDuty Notifier is broken, _get_alerts is using an undocumented api to get alerts for a service. When this API is used even when a limit param is passed it just hangs. The longest time I had this hanging in a request was 2 hours, after that I stoped the request.
This has been an issue for months and our implementation does not page us, instead the travis build does not notify PD due to the hanging request.
Requirements
- Move away from the undocumented api
- Fix the gathering of alerts so auditree can page.
Approach
Instead of using the undocumented https://api.pagerduty.com/alerts api and instead use a combination of the https://api.pagerduty.com/incidents and https://api.pagerduty.com/incidents/{id}/alerts apis to get the alerts.
We would first get all incidents for a service and loop over each one to get all alerts for the returned incidents.
Documentation for these APIS:
- https://developer.pagerduty.com/api-reference/reference/REST/openapiv3.json/paths/~1incidents/get
- https://developer.pagerduty.com/api-reference/reference/REST/openapiv3.json/paths/~1incidents~1%7Bid%7D~1alerts/get
Security and Privacy
N/A
Test Plan
Will be tested localy with our compliance checks both before and after the change to confirm that the change now pages us.