Opsgenie - Setting priorities across all Sentry alert types & general integration improvements

[dgbailey]

Problem Statement

Priorities are important for customer routing workflows in Opsgenie. For example:

We have teams that own tier 1, 2, or 3 services. On top of them, our platform metric alerts like crash-free users rate will be the most important alerts. So we can consider P1 for our crash-free session/user alerts. P2 for high number of crashes assigned to tier 1 teams, P3, for high number of crashes assigned to tier 2 teams, and P4 for tier 3 teams. With this flexibility we can define different escalation policies for different priorities on Opsgenie.

Priorities are crucial if we want to use Sentry as an important part of our day to day operations. This is vital to the way we manage on calls, incident response, escalations. Without priorities it is just another integration similar to Slack.

Improvement: OG Priority configurable across all alert types

1. Having Opsgenie priority configurable in the Sentry UI would support the case above and other workflows. The default priority of 3 is not sufficient for effective routing on the OG side.
2. Currently priority is only configurable in Issue Alerts using the integration. Not in "number of users experiencing errors", "number of errors", or any other metric alert.

Improvement: Ease of use

Process for adding new alerts and teams/keys in opsgenie & sentry is inconvenient. Currently, Api key needs to be created for each team in ops genie. Then manually add keys to Sentry teams. Entities with multiple keys exist in both platforms.

Some initial feedback: We would ideally just specify the OG team name on the platform (sentry side). This would be routed as metadata to OG. If the team doesn't exist, some default route can handle this.

Improvement: Primary descriptors for Sentry issue alerts in OG

Triggering rule details (e.g. New issue with more than X events detected) are not used as alert titles when using OG integration with issue alerts. Instead issue title (e.g. [Sentry]: This is an example Flask exception) is the top level descriptor in OG. Making it more difficult to initially understand alert. The alert rule's title would be helpful as main descriptor in this case.

Solution Brainstorm

No response

Product Area

Settings - Integrations

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[getsantry]

Routing to @getsentry/product-owners-settings-integrations for triage ⏲️

[Dhrumil-Sentry]

This is a valid ask and we'll add it to our backlog but we don't have any concrete plans to do this in the next couple of quarters

[dgbailey]

An additional consideration for this work around the "test notification" functionality of some alert types (e.g. issue alerts): With added priorities P1, etc there is the potential to accidentally create a false positive P1 alert based on current defaults in the UI. It would be helpful to have safeguards around this. Either no default channel selected, or some warning message.

[leedongwei]

Prior art: #56140. Need to do the same for Metric Alerts too.

[Dhrumil-Sentry]

We've shipped this change, Please see the changelog : https://sentry.io/changelog/change-the-severity-for-your-pagerduty-and-opsgenie-alerts/