Filter Alerts by [time of day] and [day of week]

[mark-langer]

Problem Statement

We'd like to get informed on slow average query times, but only in certain time ranges.

We know from our previous APM tool that many alerts came in unnecessarily because the average response time of our servers was very high because of very few requests at night.
E.g. a single customer generating two documents at 3 am in the morning drove up the average time altough it wasn't overburdening our servers; it just takes 10-15 seconds to generate each one.

Solution Brainstorm

We'd like to filter alerts so that they are only triggered if...

• ...the average response time is > 2000 ms and
• ...this occurs between 7am and 8pm on a weekday

Product Area

Alerts

[getsantry]

Assigning to @getsentry/support for routing ⏲️

[getsantry]

Routing to @getsentry/product-owners-alerts for triage ⏲️

[Dhrumil-Sentry]

@mark-langer thanks a lot for this feedback. It seems like you basically want to get rid of daily/weekly seasonality based false positives from these alerts i.e. Sentry should be smart enough to see a drop and know this drop happens every evening.

Is that accurate?

[mark-langer]

Yes!

…

On Tue, May 7, 2024 at 22:59 Dhrumil ***@***.***> wrote: @mark-langer <https://github.com/mark-langer> thanks a lot for this feedback. It seems like you basically want to get rid of daily/weekly seasonality based false positives from these alerts i.e. Sentry should be smart enough to see a drop and know this drop happens every evening. Is that accurate? — Reply to this email directly, view it on GitHub <#70310 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEKNHIVULBNZH6DTSMEHFZLZBE6BXAVCNFSM6AAAAABHIT3V76VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJZGMYDAMZVHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Dhrumil-Sentry]

We're considering building anomaly detection-based alerts in Sentry. Could you please share what type of metrics you get alerted on beyond response times?

Also do you setup alerting for specific endpoints or response times for all your APIs/pages?