Add custom alerting mechanism

[phenixblue]

Expected Behavior

Add a custom alerting mechanism that can send alerts on policy failures to Slack (and maybe other systems). My specific thought is around Kubernetes, but this may be applicable across the board.

Along with #1368 this could allow for passive alerting on "would be" policy failures in addition to normal policy failures.

A default alert target can be defined for the admission controller as well as user defined targets that can be read from annotations on a Kubernetes object (something like the namespace).

Actual Behavior

NA

Steps to Reproduce the Problem

NA

Additional Info

NA

[tsandall]

This feels like it could be built on-top of the existing Decision Log API. For example:

• Decision Log events would be sent to the alerting service
• Alerting service would look for the alert document in the decision
• Based on the the environment/resource/alert the service would send notifications

Definitely need to think about this further.

This is related to #1368.

[stale]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days.