Skip to content

Commit

Permalink
[tick-alerting] Add docs for tick alerting (#11417)
Browse files Browse the repository at this point in the history
Updates the alerting docs to include tick-based alerting. Uses the UI
added here:
https://app.graphite.dev/github/pr/dagster-io/internal/4510/tick-alerting-Add-tick-alerting-UI?panel=stack
  • Loading branch information
dpeng817 committed Jan 12, 2023
1 parent b9d4c67 commit 9b1f2f5
Showing 1 changed file with 33 additions and 21 deletions.
54 changes: 33 additions & 21 deletions docs/content/dagster-cloud/account/setting-up-alerts.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,12 @@ In this guide, we'll walk you through configuring job alerts in Dagster Cloud.

## Understanding alert policies

Alert policies define which jobs will trigger an alert, the conditions under which an alert will be sent, and how the alert will be sent.
Alert policies define which events will trigger an alert, the conditions under which an alert will be sent, and how the alert will be sent.

An alert policy includes a set of configured tags. If an alert policy has no configured tags, all jobs will be eligible for that alert. Otherwise, only jobs that contain all the tags for a given alert policy are eligible for that alert.
Currently, alerts can be triggered by job run failure/success, or schedule/sensor tick failure.

- **Job run based alert policies** include a set of configured tags. If an alert policy has no configured tags, all jobs will be eligible for that alert. Otherwise, only jobs that contain all the tags for a given alert policy are eligible for that alert.
- **Alert policies created for schedule/sensor tick failure will apply to all schedules/sensors**. However, you will only receive alerts when the schedule/sensor changes from a state of succeeding to failing, so subsequent failures will not trigger new alerts.

Alert policies are configured on a per deployment basis. For example, alerts configured in a `prod` deployment are only applicable to jobs in that deployment.

Expand Down Expand Up @@ -44,33 +47,38 @@ Currently, Slack and email notifications are supported.

4. Click **+ Create alert policy**.

5. In the **Create alert policy** window, fill in the following:
5. From the **Alert Policy Type** drop-down, select **Run Alert** or **Schedule/Sensor Alert**.

6. In the **Create alert policy** window, fill in the following:

- **Alert policy name** - Enter a name for the alert policy. For example, `slack_urgent_failure`
- **Description** - Enter a description for the alert policy
- **Tags** - Add tag(s) for the alert policy. Jobs with these tags will trigger the alert. For example: `level:critical` or `team:sales`
- **Events** - Select whether the alert should trigger on job success, failure, or both
- **Notification service** - Select the service for the alert policy:

- **Slack** - If you haven't connected Slack, click **Connect** to add the Dagster Cloud Slack app to your workspace. After the installation completes, invite the `@Dagster Cloud` bot user to the desired channel.
For run-based alerts, fill out these additional options:

- **Tags** - Add tag(s) for the alert policy. Jobs with these tags will trigger the alert. For example: `level:critical` or `team:sales`
- **Events** - Select whether the alert should trigger on job success, failure, or both
- **Notification service** - Select the service for the alert policy:

You can then configure the alert policy to message this channel. **Note**: Only messaging one channel per alert policy is currently supported:
- **Slack** - If you haven't connected Slack, click **Connect** to add the Dagster Cloud Slack app to your workspace. After the installation completes, invite the `@Dagster Cloud` bot user to the desired channel.

<Image
alt="Slack alert configured to alert the sales-notifications channel"
src="/images/dagster-cloud/alerts/cloud-slack-alert-creation.png"
width={1580}
height={511}
/>
You can then configure the alert policy to message this channel. **Note**: Only messaging one channel per alert policy is currently supported:

- **Email** - Email alerts can be sent to one or more recipients. For example:
<Image
alt="Slack alert configured to alert the sales-notifications channel"
src="/images/dagster-cloud/alerts/cloud-slack-alert-creation.png"
width={1580}
height={511}
/>

<Image
alt="Email alert configured to alert two recipients"
src="/images/dagster-cloud/alerts/cloud-email-alert-creation.png"
width={1571}
height={514}
/>
- **Email** - Email alerts can be sent to one or more recipients. For example:

<Image
alt="Email alert configured to alert two recipients"
src="/images/dagster-cloud/alerts/cloud-email-alert-creation.png"
width={1571}
height={514}
/>

6. When finished, click **Save policy**.

Expand Down Expand Up @@ -221,3 +229,7 @@ def important_computation():
def important_job():
important_computation()
```

### Compatible event types

When creating an alert policy using the CLI, only certain `event_types` can be specified together. You can specify multiple job run-based event types together (`JOB_SUCCESS`, `JOB_FAILURE`), or a tick-based event type (`TICK_FAILURE`), but attempting to mix these will result in an error.

1 comment on commit 9b1f2f5

@vercel
Copy link

@vercel vercel bot commented on 9b1f2f5 Jan 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please sign in to comment.