diff --git a/deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md b/deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md index 1f8fd8fa9d..b97b40e755 100644 --- a/deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md +++ b/deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md @@ -75,9 +75,9 @@ Watcher encryption Key Setup is not supported. Changing the default throttle period is not possible. You can specify a throttle period per watch, however. -Watcher comes preconfigured with a directly usable email account provided by Elastic. However, this account can’t be reconfigured and is subject to some limitations. For more information on the limits of the Elastic mail server, check the [cloud email service limits](../../../explore-analyze/alerts-cases/watcher.md#ec-cloud-email-service-limits) +Watcher comes preconfigured with a directly usable email account provided by Elastic. However, this account can’t be reconfigured and is subject to some limitations. For more information on the limits of the Elastic mail server, check the [cloud email service limits](../../../explore-analyze/alerts-cases/watcher/enable-watcher.md#cloud-email-service-limits) -Alternatively, a custom mail server can be configured as described in [Configuring a custom mail server](../../../explore-analyze/alerts-cases/watcher.md#ec-watcher-custom-mail-server) +Alternatively, a custom mail server can be configured as described in [Configuring a custom mail server](../../../explore-analyze/alerts-cases/watcher/enable-watcher.md#watcher-custom-mail-server) ## Private Link and SSO to Kibana URLs [ec-restrictions-traffic-filters-kibana-sso] diff --git a/explore-analyze/alerts-cases/watcher.md b/explore-analyze/alerts-cases/watcher.md index 3b4945a9e8..4baa4704ba 100644 --- a/explore-analyze/alerts-cases/watcher.md +++ b/explore-analyze/alerts-cases/watcher.md @@ -9,26 +9,40 @@ mapped_urls: # Watcher -% What needs to be done: Lift-and-shift +::::{tip} +{{kib}} Alerting provides a set of built-in actions and alerts that are integrated with applications such as APM, Metrics, Security, and Uptime. You can use {{kib}} Alerting to detect complex conditions within different {{kib}} apps and trigger actions when those conditions are met. For more information, refer to [Alerts and Cases](../alerts-cases.md). +:::: -% Use migrated content from existing pages that map to this page: +You can use Watcher to watch for changes or anomalies in your data and perform the necessary actions in response. For example, you might want to: -% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/xpack-alerting.md -% - [ ] ./raw-migrated-files/cloud/cloud/ec-watcher.md -% - [ ] ./raw-migrated-files/kibana/kibana/watcher-ui.md -% - [ ] ./raw-migrated-files/docs-content/serverless/elasticsearch-differences.md -% - [ ] ./raw-migrated-files/kibana/kibana/secure-reporting.md +* Monitor social media as another way to detect failures in user-facing automated systems like ATMs or ticketing systems. When the number of tweets and posts in an area exceeds a threshold of significance, notify a service technician. +* Monitor your infrastructure, tracking disk usage over time. Open a helpdesk ticket when any servers are likely to run out of free space in the next few days. +* Track network activity to detect malicious activity, and proactively change firewall configuration to reject the malicious user. +* Monitor Elasticsearch, and send immediate notification to the system administrator if nodes leave the cluster or query throughput exceeds an expected range. +* Track application response times and if page-load time exceeds SLAs for more than 5 minutes, open a helpdesk ticket. If SLAs are exceeded for an hour, page the administrator on duty. -% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): +All of these use-cases share a few key properties: -$$$watcher-create-advanced-watch$$$ +* The relevant data or changes in data can be identified with a periodic Elasticsearch query. +* The results of the query can be checked against a condition. +* One or more actions are taken if the condition is true — an email is sent, a 3rd party system is notified, or the query results are stored. -$$$ec-cloud-email-service-limits$$$ +## How watches work [_how_watches_work] -$$$ec-watcher-custom-mail-server$$$ +The {{alert-features}} provide an API for creating, managing and testing *watches*. A watch describes a single alert and can contain multiple notification actions. -$$$watcher-create-threshold-alert$$$ +A watch is constructed from four simple building blocks: -$$$watcher-deactivate$$$ +Schedule +: A schedule for running a query and checking the condition. -$$$watcher-getting-started$$$ \ No newline at end of file +Query +: The query to run as input to the condition. Watches support the full Elasticsearch query language, including aggregations. + +Condition +: A condition that determines whether or not to execute the actions. You can use simple conditions (always true), or use scripting for more sophisticated scenarios. + +Actions +: One or more actions, such as sending email, pushing data to 3rd party systems through a webhook, or indexing the results of the query. + +A full history of all watches is maintained in an Elasticsearch index. This history keeps track of each time a watch is triggered and records the results from the query, whether the condition was met, and what actions were taken. diff --git a/explore-analyze/alerts-cases/watcher/action-conditions.md b/explore-analyze/alerts-cases/watcher/action-conditions.md index 20b6b4b0e2..d9a27014e5 100644 --- a/explore-analyze/alerts-cases/watcher/action-conditions.md +++ b/explore-analyze/alerts-cases/watcher/action-conditions.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/action-conditions.html --- - - # Adding conditions to actions [action-conditions] - When a watch is triggered, its condition determines whether or not to execute the watch actions. Within each action, you can also add a condition per action. These additional conditions enable a single alert to execute different actions depending on a their respective conditions. The following watch would always send an email, when hits are found from the input search, but only trigger the `notify_pager` action when there are more than 5 hits in the search result. ```console @@ -64,5 +61,3 @@ PUT _watcher/watch/log_event_watch ``` 1. A `condition` that only applies to the `notify_pager` action, which restricts its execution to when the condition succeeds (at least 5 hits in this case). - - diff --git a/explore-analyze/alerts-cases/watcher/action-foreach.md b/explore-analyze/alerts-cases/watcher/action-foreach.md index 7d02c49d65..1e986ba220 100644 --- a/explore-analyze/alerts-cases/watcher/action-foreach.md +++ b/explore-analyze/alerts-cases/watcher/action-foreach.md @@ -41,5 +41,3 @@ PUT _watcher/watch/log_event_watch ``` 1. The logging statement will be executed for each of the returned search hits. - - diff --git a/explore-analyze/alerts-cases/watcher/actions-email.md b/explore-analyze/alerts-cases/watcher/actions-email.md index 1628d59b3c..5dd2fd2d77 100644 --- a/explore-analyze/alerts-cases/watcher/actions-email.md +++ b/explore-analyze/alerts-cases/watcher/actions-email.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/actions-email.html --- - - # Email action [actions-email] - Use the `email` action to send email notifications. To send email, you must [configure at least one email account](#configuring-email) in `elasticsearch.yml`. Email notifications can be plain text or styled using HTML. You can include information from the watch execution payload using [templates](how-watcher-works.md#templates) and attach the entire watch payload to the message. @@ -39,8 +36,6 @@ For example, the following email action uses a template to include data from the 4. The subject of the email can contain static text and Mustache [templates](how-watcher-works.md#templates). 5. The body of the email can contain static text and Mustache [templates](how-watcher-works.md#templates). Must be specified in the action definition or in the email account configuration. - - ## Configuring email attachments [configuring-email-attachments] You can attach the execution context payload or data from an any HTTP service to the email notification. There is no limit on the number of attachments you can configure. @@ -108,8 +103,6 @@ You can use the `reporting` attachment type in an `email` action to automaticall See [Automating report generation](../../report-and-share/automating-report-generation.md). - - ## Email action attributes [email-action-attributes] | Name | Required | Default | Description | @@ -139,7 +132,6 @@ $$$address-list$$$ Address List : A list of addresses can be specified as a an array: `[ 'Personal Name ', 'user2@host.domain' ]`. - ## Configuring email accounts [configuring-email] {{watcher}} can send email using any SMTP email service. Email messages can contain basic HTML tags. You can control which groups of tags are allowed by [Configuring HTML Sanitization Options](#email-html-sanitization). @@ -148,7 +140,8 @@ You configure the accounts {{watcher}} can use to send email in the `xpack.notif If your email account is configured to require two step verification, you need to generate and use a unique App Password to send email from {{watcher}}. Authentication will fail if you use your primary password. -$$$email-profile$$$ +### Email profiles [email-profile] + {{watcher}} provides three email profiles that control how MIME messages are structured: `standard` (default), `gmail`, and `outlook`. These profiles accommodate differences in how various email systems interpret the MIME standard. If you are using Gmail or Outlook, we recommend using the corresponding profile. Use the `standard` profile if you are using another email system. For more information about configuring {{watcher}} to work with different email systems, see: @@ -158,7 +151,7 @@ For more information about configuring {{watcher}} to work with different email * [Sending email from Microsoft Exchange](#exchange) * [Sending email from Amazon SES (Simple Email Service)](#amazon-ses) -If you configure multiple email accounts, you must either configure a default account or specify which account the email should be sent with in the [`email`]() action. +If you configure multiple email accounts, you must either configure a default account or specify which account the email should be sent with in the `email` action. ```yaml xpack.notification.email: @@ -170,8 +163,7 @@ xpack.notification.email: ... ``` - -#### Sending email from Gmail [gmail] +### Sending email from Gmail [gmail] Use the following email account settings to send email from the [Gmail](https://mail.google.com) SMTP service: @@ -197,8 +189,7 @@ If you get an authentication error that indicates that you need to continue the If two-step verification is enabled for your account, you must generate and use a unique App Password to send email from {{watcher}}. See [Sign in using App Passwords](https://support.google.com/accounts/answer/185833?hl=en) for more information. - -#### Sending email from Outlook.com [outlook] +#### Sending email from Outlook.com [outlook] Use the following email account settings to send email action from the [Outlook.com](https://www.outlook.com/) SMTP service: @@ -222,13 +213,11 @@ bin/elasticsearch-keystore add xpack.notification.email.account.outlook_account. When sending emails, you have to provide a from address, either a default one in your account configuration or as part of the email action in the watch. -::::{note} +::::{note} You need to use a unique App Password if two-step verification is enabled. See [App passwords and two-step verification](http://windows.microsoft.com/en-us/windows/app-passwords-two-step-verification) for more information. :::: - - -#### Sending email from Amazon SES (Simple Email Service) [amazon-ses] +#### Sending email from Amazon SES (Simple Email Service) [amazon-ses] Use the following email account settings to send email from the [Amazon Simple Email Service](http://aws.amazon.com/ses) (SES) SMTP service: @@ -249,20 +238,17 @@ xpack.notification.email.account: 1. In certain cases `email_defaults.from` is validated by Amazon SES to ensure that it is a valid local email account. 2. `smtp.host` varies depending on the region. - To store the account SMTP password, use the keystore command (see [secure settings](../../../deploy-manage/security/secure-settings.md)) ```yaml bin/elasticsearch-keystore add xpack.notification.email.account.ses_account.smtp.secure_password ``` -::::{note} +::::{note} You need to use your Amazon SES SMTP credentials to send email through Amazon SES. For more information, see [Obtaining Your Amazon SES SMTP Credentials](http://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-credentials.md). You might also need to verify [your email address](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/verify-email-addresses.md) or [your whole domain](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/verify-domains.md) at AWS. :::: - - -#### Sending email from Microsoft Exchange [exchange] +#### Sending email from Microsoft Exchange [exchange] Use the following email account settings to send email action from Microsoft Exchange: @@ -283,15 +269,13 @@ xpack.notification.email.account: 1. Some organizations configure Exchange to validate that the `from` field is a valid local email account. 2. Many organizations support use of your email address as your username, though it is a good idea to check with your system administrator if you receive authentication-related failures. - To store the account SMTP password, use the keystore command (see [secure settings](../../../deploy-manage/security/secure-settings.md)) ```yaml bin/elasticsearch-keystore add xpack.notification.email.account.exchange_account.smtp.secure_password ``` - -#### Configuring HTML sanitization options [email-html-sanitization] +#### Configuring HTML sanitization options [email-html-sanitization] The `email` action supports sending messages with an HTML body. However, for security reasons, {{watcher}} [sanitizes](https://en.wikipedia.org/wiki/HTML_sanitization) the HTML. @@ -310,5 +294,3 @@ To disable sanitization entirely, add the following setting to `elasticsearch.ym ```yaml xpack.notification.email.html.sanitization.enabled: false ``` - - diff --git a/explore-analyze/alerts-cases/watcher/actions-index.md b/explore-analyze/alerts-cases/watcher/actions-index.md index 60d1610d57..28317c18aa 100644 --- a/explore-analyze/alerts-cases/watcher/actions-index.md +++ b/explore-analyze/alerts-cases/watcher/actions-index.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/actions-index.html --- - - # Index action [actions-index] - Use the `index` action to index data into Elasticsearch. See [Index action attributes](#index-action-attributes) for the supported attributes. ## Configuring index actions [_configuring_index_actions] @@ -34,8 +31,6 @@ The following snippet shows a simple `index` action definition: 4. The index, alias, or data stream to which the data will be written 5. An optional `_id` for the document - - ## Index action attributes [index-action-attributes] | Name | Required | Default | Description | @@ -47,7 +42,6 @@ The following snippet shows a simple `index` action definition: | `timeout` | no | 60s | The timeout for waiting for the index api call to return. If no response is returned within this time, the index action times out and fails. This setting overrides the default timeouts. | | `refresh` | no | - | Optional setting of the [refresh policy](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-refresh.html) for the write request | - ## Multi-document support [anatomy-actions-index-multi-doc-support] Like with all other actions, you can use a [transform](transform.md) to replace the current execution context payload with another and by that change the document that will end up indexed. @@ -87,6 +81,3 @@ The following snippet shows a multi-document `index` action definition: 3. A new `severity` field derived from the original document 4. The payload `_doc` field which is an array of documents 5. Since the `_index` was informed per document this should be empty - - - diff --git a/explore-analyze/alerts-cases/watcher/actions-jira.md b/explore-analyze/alerts-cases/watcher/actions-jira.md index e63161fbb7..00281c0877 100644 --- a/explore-analyze/alerts-cases/watcher/actions-jira.md +++ b/explore-analyze/alerts-cases/watcher/actions-jira.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/actions-jira.html --- - - # Jira action [actions-jira] - Use the `jira` action to create issues in [Atlassian’s Jira Software](https://www.atlassian.com/software/jira). To create issues you need to [configure at least one Jira account](#configuring-jira) in `elasticsearch.yml`. ## Configuring Jira actions [configuring-jira-actions] @@ -51,17 +48,14 @@ The following snippet shows a simple jira action definition: 6. The labels to apply to the Jira issue. 7. The priority of the Jira issue. - - ## Jira action attributes [jira-action-attributes] Depending of how Jira projects are configured, the issues can have many different fields and values. Therefore the `jira` action can accept any type of sub fields within its `issue` field. These fields will be directly used when calling Jira’s [Create Issue API](https://docs.atlassian.com/jira/REST/cloud/#api/2/issue-createIssue), allowing any type of custom fields to be used. -::::{note} +::::{note} The `project.key` (or `project.id`), the `issuetype.name` (or `issuetype.id`) and `issue.summary` are always required to create an issue in Jira. :::: - | Name | Required | Description | | --- | --- | --- | | `account` | no | The Jira account to use to send the message. | @@ -78,7 +72,6 @@ The `project.key` (or `project.id`), the `issuetype.name` (or `issuetype.id`) an | `fields.environment` | no | Name of the environment related to the issue. | | `fields.customfield_XXX` | no | Custom field XXX of the issue (ex: "customfield_10000": "09/Jun/81") | - ## Configuring Jira accounts [configuring-jira] You configure the accounts {{watcher}} can use to communicate with Jira in the `xpack.notification.jira` namespace in `elasticsearch.yml`. @@ -90,20 +83,17 @@ bin/elasticsearch-keystore add xpack.notification.jira.account.monitoring.secure bin/elasticsearch-keystore add xpack.notification.jira.account.monitoring.secure_user bin/elasticsearch-keystore add xpack.notification.jira.account.monitoring.secure_password ``` - -::::{warning} +::::{warning} Storing sensitive data (`url`, `user` and `password`) in the configuration file or the cluster settings is insecure and has been deprecated. Please use {{es}}'s secure [keystore](../../../deploy-manage/security/secure-settings.md) method instead. :::: - To avoid credentials that transit in clear text over the network, {{watcher}} will reject `url` settings like `http://internal-jira.elastic.co` that are based on plain text HTTP protocol. This default behavior can be disabled with the explicit `allow_http` setting: -::::{note} +::::{note} The `url` field can also contain a path, that is used to create an issue. By default this is `/rest/api/2/issue`. If you set this as well, make sure that this path is the full path to the endpoint to create an issue. :::: - ```yaml xpack.notification.jira: account: @@ -111,11 +101,10 @@ xpack.notification.jira: allow_http: true ``` -::::{warning} +::::{warning} It is strongly advised to use Basic Authentication with secured HTTPS protocol only. :::: - You can also specify defaults for the [Jira issues](https://www.elastic.co/guide/en/elasticsearch/reference/current/notification-settings.html#jira-account-attributes): ```yaml @@ -131,7 +120,7 @@ xpack.notification.jira: labels: ["auto"] ``` -If you configure multiple Jira accounts, you either need to configure a default account or specify which account the notification should be sent with in the [`jira`]() action. +If you configure multiple Jira accounts, you either need to configure a default account or specify which account the notification should be sent with in the `jira` action. ```yaml xpack.notification.jira: @@ -142,5 +131,3 @@ xpack.notification.jira: team2: ... ``` - - diff --git a/explore-analyze/alerts-cases/watcher/actions-logging.md b/explore-analyze/alerts-cases/watcher/actions-logging.md index c5f209a2f0..b5c1ff537a 100644 --- a/explore-analyze/alerts-cases/watcher/actions-logging.md +++ b/explore-analyze/alerts-cases/watcher/actions-logging.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/actions-logging.html --- - - # Logging action [actions-logging] - Use the `logging` action to log text to the standard Elasticsearch logs. See [Logging action attributes](#logging-action-attributes) for the supported attributes. This action is primarily used during development and for debugging purposes. @@ -34,8 +31,6 @@ The following snippet shows a simple logging action definition: 2. An optional [transform](transform.md) to transform the payload before executing the `logging` action. 3. The text to be logged. - - ## Logging action attributes [logging-action-attributes] | Name | Required | Default | Description | @@ -43,5 +38,3 @@ The following snippet shows a simple logging action definition: | `text` | yes | - | The text that should be logged. Can be static text or include Mustache [templates](how-watcher-works.md#templates). | | `category` | no | xpack.watcher.actions.logging | The category under which the text will be logged. | | `level` | no | info | The logging level. Valid values are: `error`, `warn`, `info`, `debug` and `trace`. | - - diff --git a/explore-analyze/alerts-cases/watcher/actions-pagerduty.md b/explore-analyze/alerts-cases/watcher/actions-pagerduty.md index 4c270fe290..89d6c07afb 100644 --- a/explore-analyze/alerts-cases/watcher/actions-pagerduty.md +++ b/explore-analyze/alerts-cases/watcher/actions-pagerduty.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/actions-pagerduty.html --- - - # PagerDuty action [actions-pagerduty] - Use the PagerDuty action to create events in [ PagerDuty](https://pagerduty.com/). To create PagerDuty events, you must [configure at least one PagerDuty account](#configuring-pagerduty) in `elasticsearch.yml`. ## Configuring PagerDuty actions [configuring-pagerduty-actions] @@ -31,8 +28,6 @@ The following snippet shows a simple PagerDuty action definition: 1. Description of the message - - ## Adding meta information to a PagerDuty incident [adding-context-and-payloads-to-pagerduty-actions] To give the PagerDuty incident some more context, you can attach the payload as well as an array of contexts to the action. @@ -62,7 +57,6 @@ To give the PagerDuty incident some more context, you can attach the payload as } ``` - ## Pagerduty action attributes [pagerduty-action-attributes] | Name | Required | Description | @@ -73,7 +67,6 @@ To give the PagerDuty incident some more context, you can attach the payload as Although some of the attributes below have names which match the PagerDuty "Events API v1" parameter names, the "Events API v2" API is finally used by translating the attributes appropriately. :::: - $$$pagerduty-event-trigger-incident-attributes$$$ | Name | Required | Description | @@ -94,7 +87,6 @@ You can configure defaults for the above values for the whole service using the All of those objects have templating support, so you can use data from the context and the payload as part of all the fields. :::: - $$$pagerduty-event-trigger-context-attributes$$$ | Name | Required | Description | @@ -103,7 +95,6 @@ $$$pagerduty-event-trigger-context-attributes$$$ | `href` | yes/no | A link to include more information. Must be there if the type is `link`, optional if the type is `image` | | `src` | no | A src attribute for the `image` type. | - ## Configuring PagerDuty accounts [configuring-pagerduty] You configure the accounts {{watcher}} uses to communicate with PagerDuty in the `xpack.notification.pagerduty` namespace in `elasticsearch.yml`. @@ -127,7 +118,6 @@ bin/elasticsearch-keystore add xpack.notification.pagerduty.account.my_pagerduty Storing the service api key in the YAML file or via cluster update settings is still supported, but the keystore setting should be used. :::: - You can also specify defaults for the [PagerDuty event attributes](#pagerduty-event-trigger-incident-attributes): . ```yaml @@ -143,7 +133,7 @@ xpack.notification.pagerduty: attach_payload: true ``` -If you configure multiple PagerDuty accounts, you either need to set a default account or specify which account the event should be sent with in the [`pagerduty`]() action. +If you configure multiple PagerDuty accounts, you either need to set a default account or specify which account the event should be sent with in the `pagerduty` action. ```yaml xpack.notification.pagerduty: diff --git a/explore-analyze/alerts-cases/watcher/actions-slack.md b/explore-analyze/alerts-cases/watcher/actions-slack.md index a6583fab34..5589c9b226 100644 --- a/explore-analyze/alerts-cases/watcher/actions-slack.md +++ b/explore-analyze/alerts-cases/watcher/actions-slack.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/actions-slack.html --- - - # Slack action [actions-slack] - Use the `slack` action to send messages to a [Slack](https://slack.com/) team’s channels or users. To send Slack messages, you need to [configure at least one Slack account](#configuring-slack) in `elasticsearch.yml`. ## Configuring Slack actions [configuring-slack-actions] @@ -35,8 +32,6 @@ The following snippet shows a simple slack action definition: 1. The channels and users you want to send the message to. 2. The content of the message. - - ## Using attachments to format Slack messages [formatting-slack-messages] In addition to sending simple text-based messages, you can use the Slack [attachment](https://api.slack.com/docs/attachments) mechanism to send formatted messages. {{watcher}} leverages Slack attachments to enable you to dynamically populate templated messages from the execution context payload. @@ -66,7 +61,8 @@ The following snippet shows a standard message attachment: } ``` -$$$slack-dynamic-attachment$$$ +## Dynamic attachments [slack-dynamic-attachment] + To define an attachment template that is dynamically populated from the payload, you specify `dynamic_attachments` in the watch action. For example, a dynamic attachment could reference histogram buckets in the payload and build an attachment per bucket. In the following example, the watch input executes a search with a date histogram aggregation and the Slack action: @@ -124,8 +120,6 @@ In the following example, the watch input executes a search with a date histogra 1. The list generated by the action’s transform. 2. The parameter placeholders refer to attributes in each item of the list generated by the transform. - - ## Slack action attributes [slack-action-attributes] | Name | Required | Description | @@ -139,7 +133,6 @@ In the following example, the watch input executes a search with a date histogra | `proxy.host` | no | The proxy host to use (only in combination with `proxy.port`) | | `proxy.port` | no | The proxy port to use (only in combination with `proxy.host`) | - ## Configuring Slack Accounts [configuring-slack] You configure the accounts {{watcher}} can use to communicate with Slack in the `xpack.notification.slack` namespace in `elasticsearch.yml`. @@ -157,7 +150,6 @@ You can no longer configure Slack accounts using `elasticsearch.yml` settings. P :::: - You can specify defaults for the [Slack notification attributes](https://www.elastic.co/guide/en/elasticsearch/reference/current/notification-settings.html#slack-account-attributes): ```yaml @@ -190,5 +182,3 @@ xpack.notification.slack: team2: ... ``` - - diff --git a/explore-analyze/alerts-cases/watcher/actions-webhook.md b/explore-analyze/alerts-cases/watcher/actions-webhook.md index 8f433ceba0..48c841e642 100644 --- a/explore-analyze/alerts-cases/watcher/actions-webhook.md +++ b/explore-analyze/alerts-cases/watcher/actions-webhook.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/actions-webhook.html --- - - # Webhook action [actions-webhook] - Use the `webhook` action to send a request to any web service. The webhook action supports both HTTP and HTTPS connections. See [Webhook action attributes](#webhook-action-attributes) for the supported attributes. ## Configuring webhook actions [configuring-webook-actions] @@ -42,7 +39,6 @@ The following snippet shows a simple webhook action definition: 7. The path (URI) to use in the HTTP request 8. The body to send with the request - You can use basic authentication when sending a request to a secured webservice. For example, the following `webhook` action creates a new issue in GitHub: ```js @@ -68,16 +64,13 @@ You can use basic authentication when sending a request to a secured webservice. 1. The username and password for the user creating the issue - -::::{note} +::::{note} By default, both the username and the password are stored in the `.watches` index in plain text. When the {{es}} {security-features} are enabled, {{watcher}} can encrypt the password before storing it. :::: - You can also use PKI-based authentication when submitting requests to a cluster that has {{es}} {security-features} enabled. When you use PKI-based authentication instead of HTTP basic auth, you don’t need to store any authentication information in the watch itself. To use PKI-based authentication, you [configure the SSL key settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/notification-settings.html#ssl-notification-settings) for {{watcher}} in `elasticsearch.yml`. - -## Query Parameters [webhook-query-parameters] +## Query parameters [webhook-query-parameters] You can specify query parameters to send with the request with the `params` field. This field simply holds an object where the keys serve as the parameter names and the values serve as the parameter values: @@ -99,9 +92,7 @@ You can specify query parameters to send with the request with the `params` fiel 1. The parameter values can contain templated strings. - - -## Custom Request Headers [webhook-custom-request-headers] +## Custom request headers [webhook-custom-request-headers] You can specify request headers to send with the request with the `headers` field. This field simply holds an object where the keys serve as the header names and the values serve as the header values: @@ -124,8 +115,6 @@ You can specify request headers to send with the request with the `headers` fiel 1. The header values can contain templated strings. - - ## Webhook action attributes [_webhook_action_attributes] $$$webhook-action-attributes$$$ @@ -146,5 +135,3 @@ $$$webhook-action-attributes$$$ | `connection_timeout` | no | 10s | The timeout for setting up the http connection. If the connection could not be set up within this time, the action will timeout and fail. | | `read_timeout` | no | 10s | The timeout for reading data from http connection. If no response was received within this time, the action will timeout and fail. | | `url` | no | - | A shortcut for specifying the request scheme, host, port, and path as a single string. For example, `http://example.org/foo/my-service`. | - - diff --git a/explore-analyze/alerts-cases/watcher/actions.md b/explore-analyze/alerts-cases/watcher/actions.md index ec697521b9..a3aa96ba91 100644 --- a/explore-analyze/alerts-cases/watcher/actions.md +++ b/explore-analyze/alerts-cases/watcher/actions.md @@ -4,18 +4,14 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/actions.html --- - - # Actions [actions] - When a watch’s condition is met, its actions are executed unless it is being [throttled](#actions-ack-throttle). A watch can perform multiple actions. The actions are executed one at a time and each action executes independently. Any failures encountered while executing an action are recorded in the action result and in the watch history. ::::{note} If no actions are defined for a watch, no actions are executed. However, a `watch_record` is still written to the watch history. :::: - Actions have access to the payload in the execution context. They can use it to support their execution in any way they need. For example, the payload might serve as a model for a templated email body. {{watcher}} supports the following actions: @@ -28,7 +24,6 @@ Actions have access to the payload in the execution context. They can use it to * [`pagerduty`](actions-pagerduty.md) * [`jira`](actions-jira.md) - ## Acknowledgement and throttling [actions-ack-throttle] During the watch execution, once the condition is met, a decision is made per configured action as to whether it should be throttled. The main purpose of action throttling is to prevent too many executions of the same action for the same watch. @@ -88,7 +83,6 @@ PUT _watcher/watch/error_logs_alert 1. There will be at least 15 minutes between subsequent `email_administrator` action executions. 2. See [Email action](actions-email.md) for more information. - You can also define a throttle period at the watch level. The watch-level throttle period serves as the default throttle period for all of the actions defined in the watch: ```console @@ -143,7 +137,6 @@ PUT _watcher/watch/log_event_watch 1. There will be at least 15 minutes between subsequent action executions (applies to both `email_administrator` and `notify_pager` actions) - If you do not define a throttle period at the action or watch level, the global default throttle period is applied. Initially, this is set to 5 seconds. To change the global default, configure the `xpack.watcher.execution.default_throttle_period` setting in `elasticsearch.yml`: ```yaml @@ -166,19 +159,8 @@ The following diagram illustrates the throttling decisions made for each action :alt: action throttling ::: - ## Using SSL/TLS with OpenJDK [actions-ssl-openjdk] As each distributor is free to choose how to package OpenJDK, it may happen, that even despite the exact same version, an OpenJDK distribution contains different parts under different Linux distributions. This can lead to issues with any action or input that uses TLS, like the `jira`, `pagerduty`, `slack`, or `webhook` one, because of missing CA certs. If you encounter TLS errors, when writing watches that connect to TLS endpoints, you should try to upgrade to the latest available OpenJDK distribution for your platform and if that does not help, try to upgrade to Oracle JDK. - - - - - - - - - - diff --git a/explore-analyze/alerts-cases/watcher/condition-always.md b/explore-analyze/alerts-cases/watcher/condition-always.md index 69b9c2b205..737c0c657e 100644 --- a/explore-analyze/alerts-cases/watcher/condition-always.md +++ b/explore-analyze/alerts-cases/watcher/condition-always.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/condition-always.html --- - - # Always condition [condition-always] - Use the `always` condition to perform the watch actions whenever the watch is triggered, unless they are [throttled](actions.md#actions-ack-throttle). The `always` condition enables you to perform watch actions on a fixed schedule, such as, *"Every Friday at noon, send a status report email to `sys.admin@example.com`"* diff --git a/explore-analyze/alerts-cases/watcher/condition-array-compare.md b/explore-analyze/alerts-cases/watcher/condition-array-compare.md index 4ebdddb6c8..19eb31ba9d 100644 --- a/explore-analyze/alerts-cases/watcher/condition-array-compare.md +++ b/explore-analyze/alerts-cases/watcher/condition-array-compare.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/condition-array-compare.html --- - - # Array compare condition [condition-array-compare] - Use `array_compare` to compare an array of values in the execution context to a given value. See [Table 84](condition-compare.md#condition-compare-operators) for the operators you can use. ## Using an array compare condition [_using_an_array_compare_condition] @@ -37,13 +34,10 @@ For example, the following `array_compare` condition returns `true` if there is 3. The [comparison operator](condition-compare.md#condition-compare-operators) to use. 4. The comparison value. Supports date math like the [compare condition](condition-compare.md#compare-condition-date-math). - -::::{note} +::::{note} When using fieldnames that contain a dot this condition will not work, use a [script condition](condition-script.md) instead. :::: - - ## Array-compare condition attributes [_array_compare_condition_attributes] | Name | Description | @@ -52,5 +46,3 @@ When using fieldnames that contain a dot this condition will not work, use a [sc | `.path` | The path to the field in each array element that you want to evaluate. For example, `doc_count`. Defaults to an empty string. | | `..quantifier` | How many matches are required for the comparison to evaluate to `true`: `some` or `all`. Defaults to `some`--there must be at least one match. If the array is empty, the comparison evaluates to `true` if the quantifier is set to `all` and `false` if the quantifier is set to `some`. | | `..value` | The value to compare against. | - - diff --git a/explore-analyze/alerts-cases/watcher/condition-compare.md b/explore-analyze/alerts-cases/watcher/condition-compare.md index dc04dbad4e..887dae8384 100644 --- a/explore-analyze/alerts-cases/watcher/condition-compare.md +++ b/explore-analyze/alerts-cases/watcher/condition-compare.md @@ -4,14 +4,11 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/condition-compare.html --- - - # Compare condition [condition-compare] - Use the `compare` condition to perform a simple comparison against a value in the watch payload. You can use the `compare` condition without enabling dynamic scripting. -$$$condition-compare-operators$$$ +## Compare condition operators [condition-compare-operators] | Name | Description | | --- | --- | @@ -41,37 +38,36 @@ To use the `compare` condition, you specify the value in the execution context t 1. Use dot notation to reference a value in the execution context. 2. Specify a comparison operator and the value you want to compare against. - -$$$compare-condition-date-math$$$ -When comparing dates and times, you can use date math expressions of the form `<{{expression}}>`. For example, the following expression returns `true` if the watch was executed within the last five minutes: +You can also compare two values in the execution context by specifying the compared value as a path of the form of `{{path}}`. For example, the following condition compares the `ctx.payload.aggregations.status.buckets.error.doc_count` to the `ctx.payload.aggregations.handled.buckets.true.doc_count`: ```js { "condition" : { "compare" : { - "ctx.execution_time" : { - "gte" : "<{now-5m}>" + "ctx.payload.aggregations.status.buckets.error.doc_count" : { + "not_eq" : "{{ctx.payload.aggregations.handled.buckets.true.doc_count}}" } } } } ``` -You can also compare two values in the execution context by specifying the compared value as a path of the form of `{{path}}`. For example, the following condition compares the `ctx.payload.aggregations.status.buckets.error.doc_count` to the `ctx.payload.aggregations.handled.buckets.true.doc_count`: +## Using date math in a compare condition [compare-condition-date-math] + +When comparing dates and times, you can use date math expressions of the form `<{{expression}}>`. For example, the following expression returns `true` if the watch was executed within the last five minutes: ```js { "condition" : { "compare" : { - "ctx.payload.aggregations.status.buckets.error.doc_count" : { - "not_eq" : "{{ctx.payload.aggregations.handled.buckets.true.doc_count}}" + "ctx.execution_time" : { + "gte" : "<{now-5m}>" } } } } ``` - ## Accessing values in the execution context [_accessing_values_in_the_execution_context] You use "dot-notation" to access values in the execution context. Values loaded into the execution context by the input are prefixed by `ctx.payload`. @@ -86,5 +82,3 @@ You can reference entries in arrays using their zero-based array indices. For ex | `ctx.trigger.scheduled_time` | The time this watch was supposed to be triggered. | | `ctx.metadata.*` | Any metadata associated with the watch. | | `ctx.payload.*` | The payload data loaded by the watch’s input. | - - diff --git a/explore-analyze/alerts-cases/watcher/condition-never.md b/explore-analyze/alerts-cases/watcher/condition-never.md index 431d2e5f79..246cd35abe 100644 --- a/explore-analyze/alerts-cases/watcher/condition-never.md +++ b/explore-analyze/alerts-cases/watcher/condition-never.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/condition-never.html --- - - # Never condition [condition-never] - Use the `never` condition to skip performing the watch actions whenever the watch is triggered. The watch input is processed, a record is added to the watch history, and watch execution ends. This condition is generally used for testing. ## Using the never condition [_using_the_never_condition] @@ -20,5 +17,3 @@ There are no attributes to specify for the `never` condition. To use the it, you "never" : {} } ``` - - diff --git a/explore-analyze/alerts-cases/watcher/condition-script.md b/explore-analyze/alerts-cases/watcher/condition-script.md index a55d021a21..28cf4f64dc 100644 --- a/explore-analyze/alerts-cases/watcher/condition-script.md +++ b/explore-analyze/alerts-cases/watcher/condition-script.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/condition-script.html --- - - # Script condition [condition-script] - A watch [condition](condition.md) that evaluates a script. The default scripting language is `painless`. You can use any of the scripting languages supported by Elasticsearch as long as the language supports evaluating expressions to Boolean values. Note that the `mustache` and `expression` languages are too limited to be used by this condition. For more information, see [Scripting](../../scripting.md). ## Using a script condition [_using_a_script_condition] @@ -37,7 +34,6 @@ For example, the following snippet shows a formal definition of an `inline` scri } ``` - ## Inline scripts [condition-script-inline] Inline scripts are scripts that are defined in the condition itself. The following snippet shows the formal configuration of a simple painless script that always returns `true`. @@ -50,7 +46,6 @@ Inline scripts are scripts that are defined in the condition itself. The followi } ``` - ## Stored scripts [condition-script-stored] Stored scripts refer to scripts that were [stored](../../scripting/modules-scripting-using.md) in Elasticsearch. The following snippet shows how to refer to a script by its `id`: @@ -75,7 +70,6 @@ As with [inline](#condition-script-inline) scripts, you can also specify the scr } ``` - ## Accessing the watch payload [accessing-watch-payload] A script can access the current watch execution context, including the payload data, as well as any parameters passed in through the condition definition. @@ -122,5 +116,3 @@ You can reference the following variables in the watch context: | `ctx.trigger.scheduled_time` | The time this watch was supposed to be triggered. | | `ctx.metadata.*` | Any metadata associated with the watch. | | `ctx.payload.*` | The payload data loaded by the watch’s input. | - - diff --git a/explore-analyze/alerts-cases/watcher/condition.md b/explore-analyze/alerts-cases/watcher/condition.md index 2efa5e065b..8ca57ed652 100644 --- a/explore-analyze/alerts-cases/watcher/condition.md +++ b/explore-analyze/alerts-cases/watcher/condition.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/condition.html --- - - # Conditions [condition] - When a watch is triggered, its condition determines whether or not to execute the watch actions. {{watcher}} supports the following condition types: * [`always`](condition-always.md): The condition always evaluates to `true`, so the watch actions are always performed. @@ -17,17 +14,10 @@ When a watch is triggered, its condition determines whether or not to execute th * [`array_compare`](condition-array-compare.md): compare an array of values in the watch payload to a given value to determine whether or not to execute the watch actions. * [`script`](condition-script.md): use a script to determine whether or not to execute the watch actions. -::::{note} +::::{note} If you omit the condition definition from a watch, the condition defaults to `always`. :::: - When a condition is evaluated, it has full access to the watch execution context, including the watch payload (`ctx.payload.*`). The [script](condition-script.md), [compare](condition-compare.md) and [array_compare](condition-array-compare.md) conditions can use the payload data to determine whether or not the necessary conditions are met. In addition to the watch wide condition, you can also configure conditions per [action](action-conditions.md). - - - - - - diff --git a/raw-migrated-files/cloud/cloud/ec-watcher.md b/explore-analyze/alerts-cases/watcher/enable-watcher.md similarity index 85% rename from raw-migrated-files/cloud/cloud/ec-watcher.md rename to explore-analyze/alerts-cases/watcher/enable-watcher.md index b36ce68022..71289d1282 100644 --- a/raw-migrated-files/cloud/cloud/ec-watcher.md +++ b/explore-analyze/alerts-cases/watcher/enable-watcher.md @@ -1,16 +1,12 @@ -# Enable Watcher [ec-watcher] +# Enable Watcher [enable-watcher] ::::{note} -If you are looking for Kibana alerting, check [Alerting and Actions](../../../explore-analyze/alerts-cases.md) in the Kibana Guide. +If you are looking for Kibana alerting, check [Alerts and Cases](../../../explore-analyze/alerts-cases.md). :::: - -Watcher lets you take action based on changes in your data. It is designed around the principle that, if you can query something in Elasticsearch, you can alert on it. Simply define a query, condition, schedule, the actions to take, and Watcher will do the rest. - Watcher can be enabled when configuring your cluster. You can run Alerting on a separate cluster from the cluster whose data you are actually watching. - -## Before you begin [ec_before_you_begin_8] +## Before you begin [watcher-before-you-begin] Some restrictions apply when adding alerts. To learn more, check [Restrictions for alerts (via Watcher)](../../../deploy-manage/deploy/elastic-cloud/restrictions-known-problems.md#ec-restrictions-watcher). @@ -20,19 +16,17 @@ To enable Watcher on a cluster, you may first need to perform one or several of To learn more about Kibana alerting and how to use it, check [Alerting and Actions](../../../explore-analyze/alerts-cases.md). +## Send alerts by email [watcher-allowlist] -## Send alerts by email [ec-watcher-allowlist] +You can configure notifications similar to the [operational emails](../../../deploy-manage/cloud-organization/operational-emails.md) that Elasticsearch Service sends automatically to alert you about performance issues in your clusters. -Alerting can send alerts by email. You can configure notifications similar to the [operational emails](../../../deploy-manage/cloud-organization/operational-emails.md) that Elasticsearch Service sends automatically to alert you about performance issues in your clusters. - -Watcher in Elastic Cloud is preconfigured with an email service and can be used without any additional configuration. Alternatively, a custom mail server can be configured as described in [Configuring a custom mail server](../../../explore-analyze/alerts-cases/watcher.md#ec-watcher-custom-mail-server) +Watcher in Elastic Cloud is preconfigured with an email service and can be used without any additional configuration. Alternatively, a custom mail server can be configured as described in [Configuring a custom mail server](#watcher-custom-mail-server) You can optionally add [HTML sanitization](../../../explore-analyze/alerts-cases/watcher/actions-email.md#email-html-sanitization) settings under [Elasticsearch User settings](../../../deploy-manage/deploy/elastic-cloud/edit-stack-settings.md) in the [Elasticsearch Service Console](https://cloud.elastic.co?page=docs&placement=docs-body) so that HTML elements are sanitized in the email notification. For more information on sending alerts by email, check [Email action](../../../explore-analyze/alerts-cases/watcher/actions-email.md). - -## Cloud email service limits [ec-cloud-email-service-limits] +## Cloud email service limits [cloud-email-service-limits] The following quotas apply when using the Elastic email service: @@ -41,11 +35,9 @@ The following quotas apply when using the Elastic email service: * Maximum message size (including attachments): 10 MB per message (after base64 encoding). * The email-sender can’t be customized (Any custom `From:` header will be removed) +## Advanced usage [advanced_usage] -## Advanced usage [ec_advanced_usage] - - -### Slack and PagerDuty integration [ec-advanced-usage] +### Slack and PagerDuty integration [advanced-usage] Under the hood, Alerting is configured through `elasticsearch.yml`. If you want to customize your Alerting settings, you can provide custom `elasticsearch.yml` snippet which is appended to your configuration. @@ -81,7 +73,6 @@ xpack.notification.slack: to: channel3 ``` - ### Slack Webhook account settings [slack-webhook-setting] The Slack Webhook is set for each account in the Elasticsearch Keystore with the following settings: @@ -135,8 +126,7 @@ PUT _watcher/watch/test-alarm :alt: Advanced Alerting configuration ::: - -## Configuring a custom mail server [ec-watcher-custom-mail-server] +## Configuring a custom mail server [watcher-custom-mail-server] It is possible to use a custom mail service instead of the one configured by default. It can be configured by following the [Elasticsearch documentation for configuring email accounts](https://www.elastic.co/guide/en/elasticsearch/reference/current/actions-email.html). @@ -169,4 +159,3 @@ An example on how to configure a new account from the Elastic cloud console: 6. The new email account is now set up. It will now be used by default for watcher email actions. For a full reference of all available settings, see the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/notification-settings.html#email-notification-settings). - diff --git a/explore-analyze/alerts-cases/watcher/encrypting-data.md b/explore-analyze/alerts-cases/watcher/encrypting-data.md index 2313143afe..b569382c41 100644 --- a/explore-analyze/alerts-cases/watcher/encrypting-data.md +++ b/explore-analyze/alerts-cases/watcher/encrypting-data.md @@ -13,10 +13,9 @@ To encrypt sensitive data in {{watcher}}: 1. Use the [elasticsearch-syskeygen](https://www.elastic.co/guide/en/elasticsearch/reference/current/syskeygen.html) command to create a system key file. 2. Copy the `system_key` file to all of the nodes in your cluster. - - ::::{important} - The system key is a symmetric key, so the same key must be used on every node in the cluster. - :::: + ::::{important} + The system key is a symmetric key, so the same key must be used on every node in the cluster. + :::: 3. Set the [`xpack.watcher.encrypt_sensitive_data` setting](https://www.elastic.co/guide/en/elasticsearch/reference/current/notification-settings.html): @@ -34,8 +33,6 @@ To encrypt sensitive data in {{watcher}}: 5. Delete the `system_key` file on each node in the cluster. -::::{note} +::::{note} Existing watches are not affected by these changes. Only watches that you create after following these steps have encryption enabled. :::: - - diff --git a/explore-analyze/alerts-cases/watcher/example-watches.md b/explore-analyze/alerts-cases/watcher/example-watches.md index 7264da3eca..1290704522 100644 --- a/explore-analyze/alerts-cases/watcher/example-watches.md +++ b/explore-analyze/alerts-cases/watcher/example-watches.md @@ -10,5 +10,3 @@ The following example shows how to set up a watch to: * [Monitor the status of an Elasticsearch cluster](watch-cluster-status.md) For more example watches you can use as a starting point for building custom watches, refer to the [Example watches](https://github.com/elastic/examples/tree/master/Alerting) in the Elastic Examples repo. - - diff --git a/explore-analyze/alerts-cases/watcher/input-chain.md b/explore-analyze/alerts-cases/watcher/input-chain.md index a6d243ff73..e036714149 100644 --- a/explore-analyze/alerts-cases/watcher/input-chain.md +++ b/explore-analyze/alerts-cases/watcher/input-chain.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/input-chain.html --- - - # Chain input [input-chain] - Use the `chain` input to load data from multiple sources into the watch execution context when the watch is triggered. The inputs in a chain are processed in order and the data loaded by an input can be accessed by the subsequent inputs in the chain. The `chain` input enables you to perform actions based on data from multiple sources. You can also use the data collected by one input to load data from another source. @@ -43,12 +40,10 @@ For example, the following chain input loads data from an HTTP server using the 1. The inputs in a chain are specified as an array to guarantee the order in which the inputs are processed. (JSON does not guarantee the order of arbitrary objects.) 2. Loads the `path` set by the `first` input. - ## Accessing chained input data [_accessing_chained_input_data] To reference data loaded by a particular input, you use the input’s name, `ctx.payload..`. - ## Transforming chained input data [_transforming_chained_input_data] In certain use-cases the output of the first input should be used as input in a subsequent input. This requires you to do a transform, before you pass the data on to the next input. diff --git a/explore-analyze/alerts-cases/watcher/input-http.md b/explore-analyze/alerts-cases/watcher/input-http.md index be676d6e92..afcf0f37b0 100644 --- a/explore-analyze/alerts-cases/watcher/input-http.md +++ b/explore-analyze/alerts-cases/watcher/input-http.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/input-http.html --- - - # HTTP input [input-http] - Use the `http` input to submit a request to an HTTP endpoint and load the response into the watch execution context when the watch is triggered. See [HTTP input attributes](#http-input-attributes) for all of the supported attributes. With the `http` input, you can: @@ -48,7 +45,6 @@ You can use the full Elasticsearch [Query DSL](../../query-filter/languages/quer } ``` - ## Calling Elasticsearch APIs [_calling_elasticsearch_apis] To load the data from other Elasticsearch APIs, specify the API endpoint as the `path` attribute. Use the `params` attribute to specify query string parameters. For example, the following `http` input calls the [cluster stats](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-stats.html) API and enables the `human` attribute: @@ -70,8 +66,6 @@ To load the data from other Elasticsearch APIs, specify the API endpoint as the 1. Enabling this attribute returns the `bytes` values in the response in human readable format. - - ## Calling external web services [input-http-auth-basic-example] You can use `http` input to get data from any external web service. The `http` input supports basic authentication. For example, the following input provides a username and password to access `myservice`: @@ -131,8 +125,6 @@ You can also call an API using a `Bearer token` instead of basic authentication. } ``` - - ## Using templates [_using_templates_2] The `http` input supports templating. You can use [templates](how-watcher-works.md#templates) when specifying the `path`, `body`, header values, and parameter values. @@ -152,7 +144,6 @@ For example, the following snippet uses templates to specify what index to query } ``` - ## Accessing the HTTP response [_accessing_the_http_response] If the response body is formatted in JSON or YAML, it is parsed and loaded into the execution context. If the response body is not formatted in JSON or YAML, it is loaded into the payload’s `_value` field. @@ -161,7 +152,6 @@ Conditions, transforms, and actions access the response data through the executi In addition all the headers from the response can be accessed using the `ctx.payload._headers` field as well as the HTTP status code of the response using `ctx.payload._status_code`. - ## HTTP input attributes [http-input-attributes] | Name | Required | Default | Description | @@ -193,5 +183,3 @@ You can reference the following variables in the execution context when specifyi | `ctx.trigger.triggered_time` | The time this watch was triggered. | | `ctx.trigger.scheduled_time` | The time this watch was supposed to be triggered. | | `ctx.metadata.*` | Any metadata associated with the watch. | - - diff --git a/explore-analyze/alerts-cases/watcher/input-search.md b/explore-analyze/alerts-cases/watcher/input-search.md index 1c0875daa4..658cb20d36 100644 --- a/explore-analyze/alerts-cases/watcher/input-search.md +++ b/explore-analyze/alerts-cases/watcher/input-search.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/input-search.html --- - - # Search input [input-search] - Use the `search` input to load the results of an Elasticsearch search request into the execution context when the watch is triggered. See [Search Input Attributes](#search-input-attributes) for all of the supported attributes. In the search input’s `request` object, you specify: @@ -74,7 +71,6 @@ For example, the following input loads only the total number of hits into the wa }, ``` - ## Using Templates [_using_templates] The `search` input supports [search templates](../../../solutions/search/search-templates.md). For example, the following snippet references the indexed template called `my_template` and passes a value of 23 to fill in the template’s `value` parameter: @@ -98,7 +94,6 @@ The `search` input supports [search templates](../../../solutions/search/search- } ``` - ## Applying conditions [_applying_conditions] The `search` input is often used in conjunction with the [`script`](condition-script.md) condition. For example, the following snippet adds a condition to check if the search returned more than five hits: @@ -122,7 +117,6 @@ The `search` input is often used in conjunction with the [`script`](condition-sc } ``` - ## Accessing the search results [_accessing_the_search_results] Conditions, transforms, and actions can access the search results through the watch execution context. For example: @@ -132,12 +126,10 @@ Conditions, transforms, and actions can access the search results through the wa * To access a particular hit, use its zero-based array index. For example, to get the third hit, use `ctx.payload.hits.hits.2`. * To get a field value from a particular hit, use `ctx.payload.hits.hits..fields.`. For example, to get the message field from the first hit, use `ctx.payload.hits.hits.0.fields.message`. -::::{note} +::::{note} The total number of hits in the search response is returned as an object in the response. It contains a `value`, the number of hits, and a `relation` that indicates if the value is accurate (`"eq"`) or a lower bound of the total hits that match the query (`"gte"`). You can set `track_total_hits` to true in the search request to tell Elasticsearch to always track the number of hits accurately. :::: - - ## Search Input Attributes [search-input-attributes] | Name | Required | Default | Description | @@ -161,5 +153,3 @@ You can reference the following variables in the execution context when specifyi | `ctx.trigger.triggered_time` | The time this watch was triggered. | | `ctx.trigger.scheduled_time` | The time this watch was supposed to be triggered. | | `ctx.metadata.*` | Any metadata associated with the watch. | - - diff --git a/explore-analyze/alerts-cases/watcher/input-simple.md b/explore-analyze/alerts-cases/watcher/input-simple.md index 6a31b1c26b..c8a865af5b 100644 --- a/explore-analyze/alerts-cases/watcher/input-simple.md +++ b/explore-analyze/alerts-cases/watcher/input-simple.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/input-simple.html --- - - # Simple input [input-simple] - Use the `simple` input to load static data into the execution context when the watch is triggered. This enables you to store the data centrally and reference it with templates. You can define the static data as a string (`str`), numeric value (`num`), or an object (`obj`): @@ -50,4 +47,3 @@ For example, the following watch uses the `simple` input to set the recipient na } } ``` - diff --git a/explore-analyze/alerts-cases/watcher/input.md b/explore-analyze/alerts-cases/watcher/input.md index 35551a0424..b97e268d4f 100644 --- a/explore-analyze/alerts-cases/watcher/input.md +++ b/explore-analyze/alerts-cases/watcher/input.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/input.html --- - - # Inputs [input] - When a watch is triggered, its *input* loads data into the execution context. This payload is accessible during the subsequent watch execution phases. For example, you can base a watch’s condition on the data loaded by its input. {{watcher}} supports four input types: @@ -18,12 +15,6 @@ When a watch is triggered, its *input* loads data into the execution context. Th * [`http`](input-http.md): load the results of an HTTP request into the execution context. * [`chain`](input-chain.md): use a series of inputs to load data into the execution context. -::::{note} +::::{note} If you don’t define an input for a watch, an empty payload is loaded into the execution context. :::: - - - - - - diff --git a/explore-analyze/alerts-cases/watcher/managing-watches.md b/explore-analyze/alerts-cases/watcher/managing-watches.md index 2b804fc7f7..fee30b3ca3 100644 --- a/explore-analyze/alerts-cases/watcher/managing-watches.md +++ b/explore-analyze/alerts-cases/watcher/managing-watches.md @@ -14,16 +14,14 @@ mapped_pages: * Use the [deactivate watch API](https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api-deactivate-watch.html) to deactivate watches * Use the [ack watch API](https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api-ack-watch.html) to acknowledge watches - -## Listing watches [listing-watches] +## Listing watches [listing-watches] Currently there is not dedicated API for listing the stored watches. However, since {{watcher}} stores its watches in the `.watches` index, you can list them by executing a search on this index. -::::{important} +::::{important} You can only perform read actions on the `.watches` index. You must use the {{watcher}} APIs to create, update, and delete watches. If {{es}} {security-features} are enabled, we recommend you only grant users `read` privileges on the `.watches` index. :::: - For example, the following returns the first 100 watches: ```console @@ -32,4 +30,3 @@ GET /_watcher/_query/watches "size" : 100 } ``` - diff --git a/explore-analyze/alerts-cases/watcher/transform-chain.md b/explore-analyze/alerts-cases/watcher/transform-chain.md index 25908be68d..dd39e51f2e 100644 --- a/explore-analyze/alerts-cases/watcher/transform-chain.md +++ b/explore-analyze/alerts-cases/watcher/transform-chain.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-chain.html --- - - # Chain payload transform [transform-chain] - A [{{watcher-transform}}](transform.md) that executes an ordered list of configured {{watcher-transforms}} in a chain, where the output of one transform serves as the input of the next transform in the chain. The payload that is accepted by this transform serves as the input of the first transform in the chain and the output of the last transform in the chain is the output of the `chain` transform as a whole. You can use chain {{watcher-transforms}} to build more complex transforms out of the other available transforms. For example, you can combine a [`search`](transform-search.md) {{watcher-transform}} and a [`script`](transform-script.md) {{watcher-transform}}, as shown in the following snippet: @@ -40,6 +37,4 @@ You can use chain {{watcher-transforms}} to build more complex transforms out of 2. The first transform in the chain (in this case, a `search` {{watcher-transform}}) 3. The second and final transform in the chain (in this case, a `script` {{watcher-transform}}) - This example executes a `count` search on the cluster to look for `error` events. The search results are then passed to the second `script` {{watcher-transform}}. The `script` {{watcher-transform}} extracts the total hit count and assigns it to the `error_count` field in a newly-generated payload. This new payload is the output of the `chain` {{watcher-transform}} and replaces the payload in the watch execution context. - diff --git a/explore-analyze/alerts-cases/watcher/transform-script.md b/explore-analyze/alerts-cases/watcher/transform-script.md index 0452f79511..8a821afb53 100644 --- a/explore-analyze/alerts-cases/watcher/transform-script.md +++ b/explore-analyze/alerts-cases/watcher/transform-script.md @@ -4,18 +4,14 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-script.html --- - - # Script payload transform [transform-script] - A [{{watcher-transform}}](transform.md) that executes a script on the current payload in the watch execution context and replaces it with a newly generated one. The following snippet shows how a simple script {{watcher-transform}} can be defined on the watch level: -::::{tip} +::::{tip} The `script` {{watcher-transform}} is often useful when used in combination with the [`search`](transform-search.md) {{watcher-transform}}, where the script can extract only the significant data from a search result, and by that, keep the payload minimal. This can be achieved with the [`chain`](transform-chain.md) {{watcher-transform}}. :::: - ```js { "transform" : { @@ -26,15 +22,13 @@ The `script` {{watcher-transform}} is often useful when used in combination with 1. A simple `painless` script that creates a new payload with a single `time` field holding the scheduled time. - -::::{note} +::::{note} The executed script may either return a valid model that is the equivalent of a Java™ Map or a JSON object (you will need to consult the documentation of the specific scripting language to find out what this construct is). Any other value that is returned will be assigned and accessible to/via the `_value` variable. :::: +## Script settings [transform-script-settings] -The `script` attribute may hold a string value in which case it will be treated as an inline script and the default elasticsearch script languages will be assumed (as described in [Scripting](../../scripting.md)). You can use the other scripting languages supported by Elasticsearch. For this, you need to set the `script` field to an object describing the script and its language. The following table lists the possible settings that can be configured: - -$$$transform-script-settings$$$ +The `script` attribute may hold a string value in which case it will be treated as an inline script and the default Elasticsearch script languages will be assumed (as described in [Scripting](../../scripting.md)). You can use the other scripting languages supported by Elasticsearch. For this, you need to set the `script` field to an object describing the script and its language. The following table lists the possible settings that can be configured: | Name | Required | Default | Description | | --- | --- | --- | --- | @@ -45,8 +39,6 @@ $$$transform-script-settings$$$ When using the object notation of the script, one (and only one) of `inline`, or `id` fields must be defined. -::::{note} +::::{note} In addition to the provided `params`, the scripts also have access to the [standard watch execution context parameters](how-watcher-works.md#watch-execution-context). :::: - - diff --git a/explore-analyze/alerts-cases/watcher/transform-search.md b/explore-analyze/alerts-cases/watcher/transform-search.md index ea8a0ac79b..5cf0e96d4d 100644 --- a/explore-analyze/alerts-cases/watcher/transform-search.md +++ b/explore-analyze/alerts-cases/watcher/transform-search.md @@ -4,11 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-search.html --- - - # Search payload transform [transform-search] - A [{{watcher-transform}}](transform.md) that executes a search on the cluster and replaces the current payload in the watch execution context with the returned search response. The following snippet shows how a simple search transform can be defined on the watch level: ```js @@ -43,9 +40,9 @@ Like every other search based construct, one can make use of the full search API } ``` -The following table lists all available settings for the search {{watcher-transform}}: +## Transform search settings [transform-search-settings] -$$$transform-search-settings$$$ +The following table lists all available settings for the search {{watcher-transform}}: | Name | Required | Default | Description | | --- | --- | --- | --- | @@ -136,5 +133,3 @@ The following is an example of using templates that refer to provided parameters } } ``` - - diff --git a/explore-analyze/alerts-cases/watcher/transform.md b/explore-analyze/alerts-cases/watcher/transform.md index 369cec2472..e08e75f587 100644 --- a/explore-analyze/alerts-cases/watcher/transform.md +++ b/explore-analyze/alerts-cases/watcher/transform.md @@ -4,22 +4,18 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/transform.html --- - - # Transforms [transform] - A *{{watcher-transform}}* processes and changes the payload in the watch execution context to prepare it for the watch actions. {{watcher}} supports three types of {{watcher-transforms}}: * [`search`](transform-search.md) * [`script`](transform-script.md) * [`chain`](transform-chain.md) -::::{note} +::::{note} {{watcher-transforms-cap}} are optional. When none are defined, the actions have access to the payload as loaded by the watch input. :::: - You can define {{watcher-transforms}} in two places: * As a top level construct in the watch definition. In this case, the payload is transformed before any of the watch actions are executed. @@ -58,8 +54,3 @@ The following example defines two {{watcher-transforms}}, one at the watch level 1. A watch level `transform` 2. An action level `transform` - - - - - diff --git a/explore-analyze/alerts-cases/watcher/trigger-schedule.md b/explore-analyze/alerts-cases/watcher/trigger-schedule.md index 4292e974a2..a7ef66ff44 100644 --- a/explore-analyze/alerts-cases/watcher/trigger-schedule.md +++ b/explore-analyze/alerts-cases/watcher/trigger-schedule.md @@ -4,20 +4,16 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/trigger-schedule.html --- - - # Schedule trigger [trigger-schedule] - Schedule [triggers](trigger.md) define when the watch execution should start based on date and time. All times are in UTC time unless a timezone is explicitly specified in the schedule. {{watcher}} uses the system clock to determine the current time. To ensure schedules are triggered when expected, you should synchronize the clocks of all nodes in the cluster using a time service such as [NTP](http://www.ntp.org/). -::::{note} +::::{note} {{watcher}} can’t correct for manual adjustments to the system clock. Be aware when making such changes that watch execution may be affected with watches being skipped or repeated if the adjustment covers their target execution time. This applies to changes made via NTP as well. :::: - When specifying a timezone for a watch, keep in mind the effect daylight savings time transitions may have on the schedule, especially if the watch is scheduled to run during the transition. Here’s how {{watcher}} handles watches scheduled during discontinuities: ## Gap Transitions [_gap_transitions] @@ -26,11 +22,8 @@ These occur when the clock moves forward, such as when daylight savings time sta Example: If a watch is scheduled to run daily at 1:30AM in the `Europe/London` time zone and the clock moves forward one hour from 1:00AM (GMT+0) to 2:00AM (GMT+1), the watch is executed at 2:30AM (GMT+1) which would have been 1:30AM before the transition. Subsequent executions happen at 1:30AM (GMT+1). - ## Overlap Transitions [_overlap_transitions] These occur when the clock moves backward, such as when daylight savings time ends and cause certain hours or minutes to be repeated. If your watch is scheduled to run during an overlap transition, only the first occurrence of the time causes to the watch to execute with the second being skipped. Example: If a watch is scheduled to run at 1:30 AM and the clock moves backward one hour from 2:00AM to 1:00AM, the watch is executed at 1:30AM and the second occurrence after the change is skipped. - - diff --git a/explore-analyze/alerts-cases/watcher/trigger.md b/explore-analyze/alerts-cases/watcher/trigger.md index 78874ce0e5..bd3471280d 100644 --- a/explore-analyze/alerts-cases/watcher/trigger.md +++ b/explore-analyze/alerts-cases/watcher/trigger.md @@ -4,15 +4,8 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/trigger.html --- - - # Triggers [trigger] - Every watch must have a `trigger` that defines when the watch execution process should start. When you create a watch, its trigger is registered with the appropriate *Trigger Engine*. The trigger engine is responsible for evaluating the trigger and triggering the watch when needed. {{watcher}} is designed to support different types of triggers, but only time-based [`schedule`](trigger-schedule.md) triggers are currently available. - - - - diff --git a/explore-analyze/alerts-cases/watcher/watch-cluster-status.md b/explore-analyze/alerts-cases/watcher/watch-cluster-status.md index d84f0be940..fd126f2869 100644 --- a/explore-analyze/alerts-cases/watcher/watch-cluster-status.md +++ b/explore-analyze/alerts-cases/watcher/watch-cluster-status.md @@ -11,8 +11,7 @@ You can easily configure a basic watch to monitor the health of your Elasticsear * [Add a condition](#health-add-condition) that evaluates the health status to determine if action is required. * [Take action](#health-take-action) if the cluster is RED. - -## Schedule the watch and add an input [health-add-input] +## Schedule the watch and add an input [health-add-input] A watch [schedule](trigger-schedule.md) controls how often a watch is triggered. The watch [input](input.md) gets the data that you want to evaluate. @@ -29,7 +28,6 @@ PUT _watcher/watch/cluster_health_watch 1. Schedules are typically configured to run less frequently. This example sets the interval to 10 seconds to you can easily see the watches being triggered. Since this watch runs so frequently, don’t forget to [delete the watch](#health-delete) when you’re done experimenting. - To get the status of your cluster, you can call the [cluster health API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-health.html): ```console @@ -99,8 +97,7 @@ GET .watcher-history*/_search } ``` - -## Add a condition [health-add-condition] +## Add a condition [health-add-condition] A [condition](condition.md) evaluates the data you’ve loaded into the watch and determines if any action is required. Since you’ve defined an input that loads the cluster status into the watch, you can define a condition that checks that status. @@ -131,7 +128,6 @@ PUT _watcher/watch/cluster_health_watch 1. Schedules are typically configured to run less frequently. This example sets the interval to 10 seconds to you can easily see the watches being triggered. - If you check the watch history, you’ll see that the condition result is recorded as part of the `watch_record` each time the watch executes. To check to see if the condition was met, you can run the following query. @@ -145,8 +141,7 @@ GET .watcher-history*/_search?pretty } ``` - -## Take action [health-take-action] +## Take action [health-take-action] Recording `watch_records` in the watch history is nice, but the real power of {{watcher}} is being able to do something in response to an alert. A watch’s [actions](actions.md) define what to do when the watch condition is true—​you can send emails, call third-party webhooks, or write documents to an Elasticsearch index or log when the watch condition is met. @@ -207,12 +202,10 @@ xpack.notification.email.account: 2. Replace `` with your Gmail user name (typically your Gmail address). 3. Replace `` with your Gmail password. - -::::{note} +::::{note} If you have advanced security options enabled for your email account, you need to take additional steps to send email from {{watcher}}. For more information, see [Configuring email accounts](actions-email.md#configuring-email). :::: - You can check the watch history or the `status_index` to see that the action was performed. ```console @@ -224,8 +217,7 @@ GET .watcher-history*/_search?pretty } ``` - -## Delete the watch [health-delete] +## Delete the watch [health-delete] Since the `cluster_health_watch` is configured to run every 10 seconds, make sure you delete it when you’re done experimenting. Otherwise, you’ll spam yourself indefinitely. @@ -234,4 +226,3 @@ To remove the watch, use the [delete watch API](https://www.elastic.co/guide/en/ ```console DELETE _watcher/watch/cluster_health_watch ``` - diff --git a/explore-analyze/alerts-cases/watcher/watcher-getting-started.md b/explore-analyze/alerts-cases/watcher/watcher-getting-started.md index a41e00d9d1..4ba5dc5b04 100644 --- a/explore-analyze/alerts-cases/watcher/watcher-getting-started.md +++ b/explore-analyze/alerts-cases/watcher/watcher-getting-started.md @@ -5,15 +5,13 @@ mapped_pages: # Getting started with Watcher [watcher-getting-started] -$$$watch-log-data$$$ To set up a watch to start sending alerts: * [Schedule the watch and define an input](#log-add-input). * [Add a condition](#log-add-condition) that checks to see if an alert needs to be sent. * [Configure an action](#log-take-action) to send an alert when the condition is met. - -## Schedule the watch and define an input [log-add-input] +## Schedule the watch and define an input [log-add-input] A watch [schedule](trigger-schedule.md) controls how often a watch is triggered. The watch [input](input.md) gets the data that you want to evaluate. @@ -56,8 +54,7 @@ GET .watcher-history*/_search?pretty } ``` - -## Add a condition [log-add-condition] +## Add a condition [log-add-condition] A [condition](condition.md) evaluates the data you’ve loaded into the watch and determines if any action is required. Now that you’ve loaded log errors into the watch, you can define a condition that checks to see if any errors were found. @@ -87,7 +84,6 @@ PUT _watcher/watch/log_error_watch 1. The [compare](condition-compare.md) condition lets you easily compare against values in the execution context. - For this compare condition to evaluate to `true`, you need to add an event to the `logs` index that contains an error. For example, the following request adds a 404 error to the `logs` index: ```console @@ -116,8 +112,7 @@ GET .watcher-history*/_search?pretty } ``` - -## Configure an action [log-take-action] +## Configure an action [log-take-action] Recording watch records in the watch history is nice, but the real power of {{watcher}} is being able to do something when the watch condition is met. A watch’s [actions](actions.md) define what to do when the watch condition evaluates to `true`. You can send emails, call third-party webhooks, write documents to an Elasticsearch index, or log messages to the standard Elasticsearch log files. @@ -152,8 +147,7 @@ PUT _watcher/watch/log_error_watch } ``` - -## Delete the Watch [log-delete] +## Delete the Watch [log-delete] Since the `log_error_watch` is configured to run every 10 seconds, make sure you delete it when you’re done experimenting. Otherwise, the noise from this sample watch will make it hard to see what else is going on in your watch history and log file. @@ -163,17 +157,14 @@ To remove the watch, use the [delete watch API](https://www.elastic.co/guide/en/ DELETE _watcher/watch/log_error_watch ``` - -## Required security privileges [required-security-privileges] +## Required security privileges [required-security-privileges] To enable users to create and manipulate watches, assign them the `watcher_admin` security role. Watcher admins can also view watches, watch history, and triggered watches. To allow users to view watches and the watch history, assign them the `watcher_user` security role. Watcher users cannot create or manipulate watches; they are only allowed to execute read-only watch operations. - -## Where to go next [next-steps] +## Where to go next [next-steps] * See [*How {{watcher}} works*](how-watcher-works.md) for more information about the anatomy of a watch and the watch lifecycle. * See [*Example watches*](example-watches.md) for more examples of setting up a watch. * See the [Example Watches](https://github.com/elastic/examples/tree/master/Alerting) in the Elastic Examples repo for additional sample watches you can use as a starting point for building custom watches. - diff --git a/explore-analyze/alerts-cases/watcher/watcher-limitations.md b/explore-analyze/alerts-cases/watcher/watcher-limitations.md index 4887e59e4a..ea07e0d06f 100644 --- a/explore-analyze/alerts-cases/watcher/watcher-limitations.md +++ b/explore-analyze/alerts-cases/watcher/watcher-limitations.md @@ -4,19 +4,14 @@ mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-limitations.html --- - - # Limitations [watcher-limitations] - - ## Watches are not updated when file based scripts change [_watches_are_not_updated_when_file_based_scripts_change] When you refer to a file script in a watch, the watch itself is not updated if you change the script on the filesystem. Currently, the only way to reload a file script in a watch is to delete the watch and recreate it. - ## Security integration [_security_integration] When the {{security-features}} are enabled, a watch stores information about what the user who stored the watch is allowed to execute **at that time**. This means, if those permissions change over time, the watch will still be able to execute with the permissions that existed when the watch was created. diff --git a/raw-migrated-files/kibana/kibana/watcher-ui.md b/explore-analyze/alerts-cases/watcher/watcher-ui.md similarity index 88% rename from raw-migrated-files/kibana/kibana/watcher-ui.md rename to explore-analyze/alerts-cases/watcher/watcher-ui.md index e44a0c7fb1..aad72aac3d 100644 --- a/raw-migrated-files/kibana/kibana/watcher-ui.md +++ b/explore-analyze/alerts-cases/watcher/watcher-ui.md @@ -1,24 +1,20 @@ -# Watcher [watcher-ui] - -Watcher is an {{es}} feature that you can use to create actions based on conditions, which are periodically evaluated using queries on your data. Watches are helpful for analyzing mission-critical and business-critical streaming data. For example, you might watch application logs for performance outages or audit access logs for security threats. +# Watcher UI [watcher-ui] Go to the **Watcher** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). With this UI, you can: -* [Create a simple threshold watch](../../../explore-analyze/alerts-cases/watcher.md#watcher-create-threshold-alert) -* [View your watch history and action status](../../../explore-analyze/alerts-cases/watcher.md#watcher-getting-started) -* [Deactivate and delete a watch](../../../explore-analyze/alerts-cases/watcher.md#watcher-deactivate) -* [Create an advanced watch using API syntax](../../../explore-analyze/alerts-cases/watcher.md#watcher-create-advanced-watch) +* [Create a simple threshold watch](#watcher-create-threshold-alert) +* [View your watch history and action status](#watcher-getting-started) +* [Deactivate and delete a watch](#watcher-deactivate) +* [Create an advanced watch using API syntax](#watcher-create-advanced-watch) ![Watcher list](../../../images/kibana-watches.png "") [Alerting on cluster and index events](../../../explore-analyze/alerts-cases/watcher.md) is a good source for detailed information on how watches work. If you are using the UI to create a threshold watch, take a look at the different watcher actions. If you are creating an advanced watch, you should be familiar with the parts of a watch—input, schedule, condition, and actions. ::::{note} -There are limitations in **Watcher** that affect {{kib}}. For information, refer to [Alerting](../../../explore-analyze/alerts-cases/watcher/watcher-limitations.md). +There are limitations in **Watcher** that affect {{kib}}. For information, refer to [Limitations](watcher-limitations.md). :::: - - ## Watcher security [watcher-security] To use Watcher in {{kib}}, you must have the built-in `kibana_admin` role and either of these Watcher roles: @@ -32,15 +28,12 @@ To manage roles, go to the **Roles** management page, or use the [role APIs](htt If you are creating a threshold watch, you must also have the `view_index_metadata` index privilege. See [Index management](../../../manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md) for detailed information. :::: - - ## Create a threshold alert [watcher-create-threshold-alert] A threshold alert is one of the most common types of watches that you can create. This alert periodically checks when your data is above, below, equals, or is in between a certain threshold within a given time interval. The following example walks you through creating a threshold alert. The alert is triggered when the maximum total CPU usage on a machine goes above a certain percentage. The example uses [Metricbeat](https://www.elastic.co/products/beats/metricbeat) to collect metrics from your systems and services. [Learn more](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-installation-configuration.html) on how to install and get started with Metricbeat. - ### Define the watch input and schedule [_define_the_watch_input_and_schedule] 1. Click **Create** and then select **Create threshold alert**. @@ -53,8 +46,6 @@ The following example walks you through creating a threshold alert. The alert is ![Input and schedule for threshold alert](../../../images/kibana-create-threshold-alert-created.png "") - - ### Add a condition [_add_a_condition] You should now see a panel with default conditions and a visualization of the data based on those conditions. The condition evaluates the data you’ve loaded into the watch and determines if any action is required. @@ -70,8 +61,6 @@ You should now see a panel with default conditions and a visualization of the da ![Condition for threshold alert](../../../images/kibana-threshold-alert-condition.png "") - - ### Add an action [_add_an_action] Now that the condition is set, you must add an action. The action triggers when the watch condition is met. For a complete list of actions and how to configure them, see [Adding conditions to actions](../../../explore-analyze/alerts-cases/watcher/action-conditions.md). @@ -89,18 +78,14 @@ In this example, you’ll configure an email action. You must have an [email acc The alert appears on the Watcher overview page, where you can drill down into the watch history and status. - - ### Delete the alert [_delete_the_alert] In this example, you set the threshold to 25% so you can see the watch execute its actions. In a real-world scenario, this threshold is likely too low because the alerts will be too frequent. Once you are done experimenting, you should delete the alert. Find the alert on the Watcher overview page and click the trash icon in the **Actions** column. - ### Edit the alert [_edit_the_alert] Alternatively, you can keep the alert and adjust the threshold value. To edit an alert, find the alert on the Watcher overview page and click the pencil icon in the **Actions** column. - ## View watch history and status [watcher-getting-started] The Watcher overview page lists your watches, including the state of each watch, when its condition was last met, when the condition was last checked, and whether any of its actions have been acknowledged, throttled, or failed to execute. Each watch can be in one of three states: @@ -111,21 +96,18 @@ The Watcher overview page lists your watches, including the state of each watch, From this page you can drill down into a watch to investigate its history and status. - ### View watch history [_view_watch_history] The **Execution history** tab shows each time the watch is triggered and the results of the query, whether the condition was met, and what actions were taken. ![Execution history tab](../../../images/kibana-execution-history.png "") - ### Acknowledge action status [_acknowledge_action_status] The **Action statuses** tab lists all actions associated with the watch and the state of each action. Some actions can be acknowledged, which will prevent too many executions of that action for the relevant watch. See [Acknowledgement and throttling](../../../explore-analyze/alerts-cases/watcher/actions.md#actions-ack-throttle) for details. ![Action status tab](../../../images/kibana-alerts-status.png "") - ## Deactivate and delete a watch [watcher-deactivate] Actions for deactivating and deleting a watch are on each watch detail page: @@ -133,19 +115,16 @@ Actions for deactivating and deleting a watch are on each watch detail page: * **Deactivate a watch** if you know a situation is planned that will cause a false alarm. You can reactivate the watch when the situation is resolved. * **Delete a watch** to permanently remove it from the system. You can delete the watch you are currently viewing, or go to the Watcher overview, and delete watches in bulk. - ## Create an advanced watch [watcher-create-advanced-watch] Advanced watches are for users who are more familiar with {{es}} query syntax and the Watcher framework. The UI is aligned with using the REST APIs. For more information, see [Query DSL](../../../explore-analyze/query-filter/languages/querydsl.md). - ### Create the watch [_create_the_watch] On the Watch overview page, click **Create** and choose **Create advanced watch**. An advanced watch requires a name and ID. Name is a user-friendly way to identify the watch, and ID refers to the identifier used by {{es}}. Refer to [Watch definition](../../../explore-analyze/alerts-cases/watcher/how-watcher-works.md#watch-definition) for how to input the watch JSON. ![Create advanced watch](../../../images/kibana-advanced-watch-create.png "") - ### Simulate the watch [_simulate_the_watch] The **Simulate** tab allows you to override parts of the watch, and then run a simulation. Be aware of these implementation details on overrides: @@ -161,11 +140,9 @@ After starting the simulation, you’ll see a results screen. For more informati ![Create advanced watch](../../../images/kibana-advanced-watch-simulate.png "") - ### Examples of advanced watches [_examples_of_advanced_watches] Refer to these examples for creating an advanced watch: -* [Watch the status of an {{es}} cluster](../../../explore-analyze/alerts-cases/watcher/watch-cluster-status.md) -* [Watch event data](https://www.elastic.co/guide/en/elasticsearch/reference/current/example-watches.html) - +* [Watch the status of an {{es}} cluster](watch-cluster-status.md) +* [Watch event data](example-watches.md) diff --git a/explore-analyze/scripting/painless-lab.md b/explore-analyze/scripting/painless-lab.md index c11c52726e..0906ededc2 100644 --- a/explore-analyze/scripting/painless-lab.md +++ b/explore-analyze/scripting/painless-lab.md @@ -9,8 +9,7 @@ mapped_pages: This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features. :::: - -The **Painless Lab** is an interactive code editor that lets you test and debug [Painless scripts](modules-scripting-painless.md) in real-time. You can use the Painless scripting language to create [{{kib}} runtime fields](../find-and-organize/data-views.md#runtime-fields), process [reindexed data](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html), define complex [Watcher conditions](../alerts-cases/watcher.md#watcher-create-advanced-watch), and work with data in other contexts. +The **Painless Lab** is an interactive code editor that lets you test and debug [Painless scripts](modules-scripting-painless.md) in real-time. You can use the Painless scripting language to create [{{kib}} runtime fields](../find-and-organize/data-views.md#runtime-fields), process [reindexed data](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html), define complex [Watcher conditions](../alerts-cases/watcher.md), and work with data in other contexts. Find **Painless Lab** by navigating to the **Developer tools** page using the navigation menu or the [global search field](../../explore-analyze/find-and-organize/find-apps-and-objects.md). @@ -18,4 +17,3 @@ Find **Painless Lab** by navigating to the **Developer tools** page using the na :alt: Painless Lab :class: screenshot ::: - diff --git a/explore-analyze/toc.yml b/explore-analyze/toc.yml index cb10bceb41..36c70c7bc8 100644 --- a/explore-analyze/toc.yml +++ b/explore-analyze/toc.yml @@ -357,6 +357,8 @@ toc: children: - file: alerts-cases/watcher/watcher-getting-started.md - file: alerts-cases/watcher/how-watcher-works.md + - file: alerts-cases/watcher/enable-watcher.md + - file: alerts-cases/watcher/watcher-ui.md - file: alerts-cases/watcher/encrypting-data.md - file: alerts-cases/watcher/input.md children: diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/xpack-alerting.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/xpack-alerting.md deleted file mode 100644 index d0809eff95..0000000000 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/xpack-alerting.md +++ /dev/null @@ -1,42 +0,0 @@ -# Watcher [xpack-alerting] - -::::{tip} -{{kib}} Alerting provides a set of built-in actions and alerts that are integrated with applications such as APM, Metrics, Security, and Uptime. You can use {{kib}} Alerting to detect complex conditions within different {{kib}} apps and trigger actions when those conditions are met. For more information, see [Alerting and actions](../../../explore-analyze/alerts-cases.md). -:::: - - -You can use Watcher to watch for changes or anomalies in your data and perform the necessary actions in response. For example, you might want to: - -* Monitor social media as another way to detect failures in user-facing automated systems like ATMs or ticketing systems. When the number of tweets and posts in an area exceeds a threshold of significance, notify a service technician. -* Monitor your infrastructure, tracking disk usage over time. Open a helpdesk ticket when any servers are likely to run out of free space in the next few days. -* Track network activity to detect malicious activity, and proactively change firewall configuration to reject the malicious user. -* Monitor Elasticsearch, and send immediate notification to the system administrator if nodes leave the cluster or query throughput exceeds an expected range. -* Track application response times and if page-load time exceeds SLAs for more than 5 minutes, open a helpdesk ticket. If SLAs are exceeded for an hour, page the administrator on duty. - -All of these use-cases share a few key properties: - -* The relevant data or changes in data can be identified with a periodic Elasticsearch query. -* The results of the query can be checked against a condition. -* One or more actions are taken if the condition is true — an email is sent, a 3rd party system is notified, or the query results are stored. - - -## How watches work [_how_watches_work] - -The {{alert-features}} provide an API for creating, managing and testing *watches*. A watch describes a single alert and can contain multiple notification actions. - -A watch is constructed from four simple building blocks: - -Schedule -: A schedule for running a query and checking the condition. - -Query -: The query to run as input to the condition. Watches support the full Elasticsearch query language, including aggregations. - -Condition -: A condition that determines whether or not to execute the actions. You can use simple conditions (always true), or use scripting for more sophisticated scenarios. - -Actions -: One or more actions, such as sending email, pushing data to 3rd party systems through a webhook, or indexing the results of the query. - -A full history of all watches is maintained in an Elasticsearch index. This history keeps track of each time a watch is triggered and records the results from the query, whether the condition was met, and what actions were taken. - diff --git a/raw-migrated-files/toc.yml b/raw-migrated-files/toc.yml index df39317fcc..51bcb0e1ea 100644 --- a/raw-migrated-files/toc.yml +++ b/raw-migrated-files/toc.yml @@ -243,7 +243,6 @@ toc: - file: cloud/cloud/ec-traffic-filtering-vpc.md - file: cloud/cloud/ec-upgrade-deployment.md - file: cloud/cloud/ec-user-privileges.md - - file: cloud/cloud/ec-watcher.md - file: cloud/cloud/ec-working-with-elasticsearch.md - file: curator/elasticsearch-client-curator/index.md children: @@ -621,7 +620,6 @@ toc: - file: elasticsearch/elasticsearch-reference/snapshots-restore-snapshot.md - file: elasticsearch/elasticsearch-reference/starting-elasticsearch.md - file: elasticsearch/elasticsearch-reference/stopping-elasticsearch.md - - file: elasticsearch/elasticsearch-reference/xpack-alerting.md - file: elasticsearch/elasticsearch-reference/xpack-autoscaling.md - file: elasticsearch/elasticsearch-reference/xpack-rollup.md - file: ingest-docs/fleet/index.md @@ -660,7 +658,6 @@ toc: - file: kibana/kibana/upgrade-migrations-rolling-back.md - file: kibana/kibana/upgrade.md - file: kibana/kibana/using-kibana-with-security.md - - file: kibana/kibana/watcher-ui.md - file: kibana/kibana/xpack-security-authorization.md - file: kibana/kibana/xpack-security-fips-140-2.md - file: kibana/kibana/xpack-security.md