diff --git a/docs/use-cases/observability/clickstack/alerts.md b/docs/use-cases/observability/clickstack/alerts.md index e42bce80fb1..c215137f0d4 100644 --- a/docs/use-cases/observability/clickstack/alerts.md +++ b/docs/use-cases/observability/clickstack/alerts.md @@ -11,33 +11,195 @@ keywords: ['ClickStack', 'observability', 'alerts', 'search-alerts', 'notificati import Image from '@theme/IdealImage'; import search_alert from '@site/static/images/use-cases/observability/search_alert.png'; +import edit_chart_alert from '@site/static/images/use-cases/observability/edit_chart_alert.png'; +import add_chart_alert from '@site/static/images/use-cases/observability/add_chart_alert.png'; +import create_chart_alert from '@site/static/images/use-cases/observability/create_chart_alert.png'; +import alerts_search_view from '@site/static/images/use-cases/observability/alerts_search_view.png'; +import add_new_webhook from '@site/static/images/use-cases/observability/add_new_webhook.png'; +import add_webhook_dialog from '@site/static/images/use-cases/observability/add_webhook_dialog.png'; +import manage_alerts from '@site/static/images/use-cases/observability/manage_alerts.png'; +import alerts_view from '@site/static/images/use-cases/observability/alerts_view.png'; +import multiple_search_alerts from '@site/static/images/use-cases/observability/multiple_search_alerts.png'; +import remove_chart_alert from '@site/static/images/use-cases/observability/remove_chart_alert.png'; -## Search alerts {#search-alerts} +## Alerting in ClickStack {#alerting-in-clickstack} -After entering a [search](/use-cases/observability/clickstack/search), you can create an alert to be -notified when the number of events (logs or spans) matching the search exceeds or falls below a threshold. +ClickStack includes built-in support for alerting, enabling teams to detect and respond to issues in real time across logs, metrics, and traces. -### Creating an alert {#creating-an-alert} +Alerts can be created directly in the HyperDX interface and integrate with popular notification systems like Slack and PagerDuty. -You can create an alert by clicking the `Alerts` button on the top right of the `Search` page. +Alerting works seamlessly across your ClickStack data, helping you track system health, catch performance regressions, and monitor key business events. -From here, you can name the alert, as well as set the threshold, duration, and notification method for the alert (Slack, Email, PagerDuty or Slack webhook). +## Types of alerts {#types-of-alerts} -The `grouped by` value allows the search to be subject to an aggregation e.g. `ServiceName`, thus allowing potential multiple alerts to be triggered off the same search. +ClickStack supports two complementary ways to create alerts: **Search alerts** and **Dashboard chart alerts**. Once the alert is created, it is attached to either the search or the chart. + +### 1. Search alerts {#search-alerts} + +Search alerts allow you to trigger notifications based on the results of a saved search. They help you detect when specific events or patterns occur more (or less) frequently than expected. + +An alert is triggered when the count of matching results within a defined time window either exceeds or falls below a specified threshold. + +To create a search alert: + + + +For an alert to be created for a search, the search must be saved. Users can either create the alert for an existing saved search or save the search during the alert creation process. In the example below, we assume the search is not saved. + +#### Open alert creation dialog {#open-dialog} + +Start by entering a [search](/use-cases/observability/clickstack/search) and clicking the `Alerts` button in the top-right corner of the `Search` page. + +Alerts search view + +#### Create the alert {#create-the-alert} + +From the alert creation panel, you can: + +- Assign a name to the saved search associated with the alert. +- Set a threshold and specify how many times it must be reached within a given period. Thresholds can also be used as upper or lower bounds. The period here will also dictate how often the alert is triggered. +- Specify a `grouped by` value. This allows the search to be subject to an aggregation, e.g., `ServiceName`, thus allowing multiple alerts to be triggered off the same search. +- Choose a webhook destination for notifications. You can add a new webhook directly from this view. See [Adding a webhook](#add-webhook) for details. + +Before saving, ClickStack visualizes the threshold condition so you can confirm it will behave as expected. Search alerts -### Common alert scenarios {#common-alert-scenarios} + + +Note that multiple alerts can be added to a search. If the above process is repeated, users will see the current alerts as tabs at the top of the edit alert dialog, with each alert assigned a number. + +Multiple alerts + +### 2. Dashboard chart alerts {#dashboard-alerts} + +Dashboard alerts extend alerting to charts. + +You can create a chart-based alert directly from a saved dashboard, powered by full SQL aggregations and ClickHouse functions for advanced calculations. + +When a metric crosses a defined threshold, an alert triggers automatically, allowing you to monitor KPIs, latencies, or other key metrics over time. + +:::note +For an alert to be created for a visualization on a dashboard, the dashboard must be saved. +::: + +To add a dashboard alert: + + + +Alerts can be created during the chart creation process, when adding a chart to a dashboard, or added to existing charts. In the example below, we assume the chart already exists on the dashboard. + +#### Open the chart edit dialog {#open-chart-dialog} + +Open the chart's configuration menu and select the alert button. This will show the chart edit dialog. + +Edit chart alert + +#### Add the alert {#add-chart-alert} + +Select **Add Alert**. + +Add alert to chart + +#### Define the alert conditions {#define-alert-conditions} + +Define the condition (`>=`, `<`), threshold, duration, and webhook. The duration here will also dictate how often the alert is triggered. + +Create alert for chart + +You can add a new webhook directly from this view. See [Adding a webhook](#add-webhook) for details. + + + +## Adding a webhook {#add-webhook} + +During alert creation, users can either use an existing webhook or create one. Once created, the webhook will be available for reuse across other alerts. + +A webhook can be created for different service types, including Slack and PagerDuty, as well as generic targets. + +For example, consider the alert creation for a chart below. Before specifying the webhook, the user can select `Add New Webhook`. + +Add new webhook + +This opens the webhook creation dialog, where users can create a new webhook: + +Webhook creation + +A webhook name is required, while descriptions are optional. Other settings that must be completed depend on the service type. + +Note that different service types are available between ClickStack Open Source and ClickStack Cloud. See [Service type integrations](#integrations). + +### Service type integrations {#integrations} + +ClickStack alerts integrate out of the box with the following service types: + +- **Slack**: send notifications directly to a channel via either a webhook or API. +- **PagerDuty**: route incidents for on-call teams via the PagerDuty API. +- **Webhook**: connect alerts to any custom system or workflow via a generic webhook. + +:::note ClickHouse Cloud only integrations +The Slack API and PagerDuty integrations are only supported in ClickHouse Cloud. +::: + +Depending on the service type, users will need to provide different details. Specifically: + +**Slack (Webhook URL)** + +- Webhook URL. For example: `https://hooks.slack.com/services/`. See the [Slack documentation](https://docs.slack.dev/messaging/sending-messages-using-incoming-webhooks/) for further details. + +**Slack (API)** + +- Slack bot token. See the [Slack documentation](https://docs.slack.dev/authentication/tokens/#bot/) for further details. + +**PagerDuty API** + +- PagerDuty integration key. See the [PagerDuty documentation](https://support.pagerduty.com/main/docs/api-access-keys) for further details. + +**Generic** + +- Webhook URL +- Webhook headers (optional) +- Webhook body (optional). The body currently supports the template variables `{{title}}`, `{{body}}`, and `{{link}}`. + +## Managing alerts {#managing-alerts} + +Alerts can be centrally managed through the alerts panel on the left-hand side of HyperDX. + +Manage alerts + +From this view, users can see all alerts that have been created and are currently running in ClickStack. + +Alerts view + +This view also displays the alert evaluation history. Alerts are evaluated on a recurring time interval (defined by the period/duration set during alert creation). During each evaluation, HyperDX queries your data to check whether the alert condition is met: + +- **Red bar**: The threshold condition was met during this evaluation and the alert fired (notification sent) +- **Green bar**: The alert was evaluated but the threshold condition was not met (no notification sent) + +Each evaluation is independent - the alert checks the data for that time window and fires only if the condition is true at that moment. + +In the example above, the first alert has fired on every evaluation, indicating a persistent issue. The second alert shows a resolved issue - it fired twice initially (red bars), then on subsequent evaluations the threshold condition was no longer met (green bars). + +Clicking an alert takes you to the chart or search the alert is attached to. + +### Deleting an alert {#deleting-alerts} + +To remove an alert, open the edit dialog for the associated search or chart, then select **Remove Alert**. +In the example below, the `Remove Alert` button will remove the alert from the chart. + +Remove chart alert + +## Common alert scenarios {#common-alert-scenarios} -Here are a few common alert scenarios that you can use HyperDX for: +Here are a few common alert scenarios you can use HyperDX for: -**Errors:** We first recommend setting up alerts for the default +**Errors:** We recommend setting up alerts for the default `All Error Events` and `HTTP Status >= 400` saved searches to be notified when -excess error occurs. +excess errors occur. -**Slow Operations:** You can set up a search for slow operations (ex. +**Slow Operations:** You can set up a search for slow operations (e.g., `duration:>5000`) and then alert when there are too many slow operations occurring. **User Events:** You can also set up alerts for customer-facing teams to be -notified when new users sign up, or a critical user action is performed. +notified when new users sign up or a critical user action is performed. diff --git a/static/images/use-cases/observability/add_chart_alert.png b/static/images/use-cases/observability/add_chart_alert.png new file mode 100644 index 00000000000..6e7e2dac00c Binary files /dev/null and b/static/images/use-cases/observability/add_chart_alert.png differ diff --git a/static/images/use-cases/observability/add_new_webhook.png b/static/images/use-cases/observability/add_new_webhook.png new file mode 100644 index 00000000000..4c84e90e9b1 Binary files /dev/null and b/static/images/use-cases/observability/add_new_webhook.png differ diff --git a/static/images/use-cases/observability/add_webhook_dialog.png b/static/images/use-cases/observability/add_webhook_dialog.png new file mode 100644 index 00000000000..3b07492344e Binary files /dev/null and b/static/images/use-cases/observability/add_webhook_dialog.png differ diff --git a/static/images/use-cases/observability/alerts_search_view.png b/static/images/use-cases/observability/alerts_search_view.png new file mode 100644 index 00000000000..c0472e73c8e Binary files /dev/null and b/static/images/use-cases/observability/alerts_search_view.png differ diff --git a/static/images/use-cases/observability/alerts_view.png b/static/images/use-cases/observability/alerts_view.png new file mode 100644 index 00000000000..05b65c86389 Binary files /dev/null and b/static/images/use-cases/observability/alerts_view.png differ diff --git a/static/images/use-cases/observability/create_chart_alert.png b/static/images/use-cases/observability/create_chart_alert.png new file mode 100644 index 00000000000..80ed5ca34de Binary files /dev/null and b/static/images/use-cases/observability/create_chart_alert.png differ diff --git a/static/images/use-cases/observability/edit_chart_alert.png b/static/images/use-cases/observability/edit_chart_alert.png new file mode 100644 index 00000000000..dd4bedd9436 Binary files /dev/null and b/static/images/use-cases/observability/edit_chart_alert.png differ diff --git a/static/images/use-cases/observability/manage_alerts.png b/static/images/use-cases/observability/manage_alerts.png new file mode 100644 index 00000000000..00d1644c67f Binary files /dev/null and b/static/images/use-cases/observability/manage_alerts.png differ diff --git a/static/images/use-cases/observability/multiple_search_alerts.png b/static/images/use-cases/observability/multiple_search_alerts.png new file mode 100644 index 00000000000..3d85d58d7d3 Binary files /dev/null and b/static/images/use-cases/observability/multiple_search_alerts.png differ diff --git a/static/images/use-cases/observability/remove_chart_alert.png b/static/images/use-cases/observability/remove_chart_alert.png new file mode 100644 index 00000000000..b245208949d Binary files /dev/null and b/static/images/use-cases/observability/remove_chart_alert.png differ