Add minimum threshold on Failure rate alerts before triggering the alert.

[lucas-zimerman]

Problem Statement

Failure Rate doesn't necessarily mean what it's suppose to do, It can give many false positives if a given project has a low amount of transactions received, see example below:

This is an alert that got triggered by the Failure Rate alert, notice that the failure rate is over the defined value (92%)

But, if looked on discover, the same timestamp only had 4 events, where 3 of them failed.

The alert is doing what it was defined for, it generated the alert, but a critical alert for such low amount of samples is not ideal on projects , it can trigger many noise due to the low amount of sampling (In the given picture, the issue is the project itslef isnt used a lot )

Solution Brainstorm

A simple solution would be to add a field that defines the minimum amount of samples before triggering the alert.
Lets say the user defines the minimum amount of samples to 15, so if all transactions failed but are below the value 15, the alerts will not be triggered, but, if there's more than 15 transactions, then the alert will check for the failure rate defined by the end user.

This will avoid cases where a project has low amount of samples or they have a low usage rate compared to other projects and gets triggered by 2 failed events on a range of 15 - 30 minutes.

[getsentry-release]

Routing to @getsentry/workflow for triage. ⏲️

[wedamija]

Thanks for the suggestion @lucas-zimerman. We've added this to our backlog for consideration in the future.