Make PagerDuty output idempotent; adds support for responder requests

[Ryxias]

to: @ryandeivert or @chunyong-lin
cc: @airbnb/streamalert-maintainers

Background

The pagerduty-incident code used to have a serious bug where, due to the way it would sequence multiple write methods non-atomically, it could result in a situation where a Pagerduty alert is sent but the output fails. This results in the alert retrying, creating another Pagerduty alert, then failing again... repeating ad infinitum.

Changes

This PR introduces two concepts:

Reduced number of write calls

PagerDuty's code used to do the following:

• POST /incidents (Create an incident container)
• POST /events/enqueue (Create an alert)
• GET /incidents?dedup_key=??? (Find the alert-incident)
• PUT /incidents/#/merge?incident_id=??? (Merge the alert-incident into the incident container)

I found that this was redundant; the workflow is optimized now:

• POST /events/enqueue (Create an alert)
• GET /incidents?dedup_key=??? (Find the alert's incident)
• PUT /incidents (Modify the alert's incident)

Fewer API calls means the code is cleaner and easier to understand. Also it only has POST method, which reduces the number of non-idempotent write API calls.

Idempotency

PagerDuty's pagerduty-incident output is now idempotent. The POST /events/enqueue API call
now leverages a dedup_key that is equal to the Alert ID.

The usage of a dedup_key ensures that any unique StreamAlert alert will only ever create a single PagerDuty alert. Subsequent retries will cause PagerDuty to return the previously created alert.

Responder Requests

I added a new PagerDuty feature that allows the Alert to leverage Responder Requests. This is a neat feature that allows you to invite people other than the assignee to join in the PagerDuty.

It shows up like this on the PD UI.

All members of the response team get all notification (push notification/SMS/etc) related to the Alert except escalation notices. These stay with the assignee.

The awesome benefit here is that we can have PagerDuty alerts stick to a single escalation policy, but attach more watchers.

You can add responder requests to an alert using context:

context={
  responders=['derek1@email.com', 'derek2@email.com' ...],
  responder_message='This message shows up on their request',
}

Testing

CI, Plus I tried
it on Stage A lot. I think
it bothered people

[coveralls]

Coverage increased (+0.2%) to 96.835% when pulling 55c0712 on dw--pd-idempotency into 4baf713 on master.

[chunyong-lin]

where do we add this note to?

[Ryxias]

This note is a small string of text that is added to the PagerDuty incident. It shows up on their UI

[ryandeivert]

are responders optional? meaning should we return False if not responder or is that okay?

[Ryxias]

Responders are optional; it's a new feature so I need to maintain reverse compatibility.

[ryandeivert]

great work on this and thanks a ton for the thorough documentation on the changes