PagerDuty alerts email using Dagster Cloud Alerts not triggering

[marcilj]

We've configured PagerDuty alerts using Dagster Cloud Deployment Alerts feature as describe here.

We have a pretty simple configuration,

We've seen a couple of alerts that were never created on the Pager Duty side even if we have the steps telling us that Dagster did send an email to Pager Duty in the job run.

On the pager duty side, we've look and it's configure to receive all without any filters and generate an alerts for each emails, so I don't know how this configuration can be faulty.

It's been like 3 or 4 events like this that we found in the last month, so it doesn't seems like a isolated events.

Since we are using the Dagster Cloud alerts options there isn't any other option than email, so unless we configure it another way we need to stick with SMTP.

Is there anyway we can debug this further or is there any recommendation?

[shalabhc]

Hi @marcilj - since the logs show it finished sending email, you may want to reach out to pagerduty to see why email alerts may not be sending notifications. We'll take a look at our logs as well just to verify there is no other failure at our end.

[marcilj]

After verification it seems like it was a bad configuration on our side.
It seems like PagerDuty won't generate alerts even if it receive some if no one is on call at the moment of the event. This seems very weird, but that's what we've found.
Thank you @shalabhc.