From 25f2b06fb9b381cf234e9c66dd0b0dcaff0bb685 Mon Sep 17 00:00:00 2001 From: Joe Peeples Date: Wed, 16 Oct 2024 09:31:12 -0400 Subject: [PATCH 1/2] Permissions for alert suppression in machine learning rules (#5819) * Update ESS docs: ML rule req for alert suppression * Update serverless docs, and align with ESS (cherry picked from commit 632deff5790dea90c78306316897127c83b5a51b) # Conflicts: # docs/serverless/advanced-entity-analytics/ml-requirements.mdx # docs/serverless/alerts/alert-suppression.mdx --- .../advanced-entity-analytics/ml-req.asciidoc | 4 + docs/detections/alert-suppression.asciidoc | 6 +- .../ml-requirements.mdx | 30 +++++ docs/serverless/alerts/alert-suppression.mdx | 117 ++++++++++++++++++ 4 files changed, 155 insertions(+), 2 deletions(-) create mode 100644 docs/serverless/advanced-entity-analytics/ml-requirements.mdx create mode 100644 docs/serverless/alerts/alert-suppression.mdx diff --git a/docs/advanced-entity-analytics/ml-req.asciidoc b/docs/advanced-entity-analytics/ml-req.asciidoc index f42f61e513..edab94a672 100644 --- a/docs/advanced-entity-analytics/ml-req.asciidoc +++ b/docs/advanced-entity-analytics/ml-req.asciidoc @@ -7,6 +7,10 @@ To run and create {ml} jobs and rules, you need all of these: * There must be at least one {ml} node in your cluster * The `machine_learning_admin` user role +Additionally, to configure <> for {ml} rules, your role needs the following {kibana-ref}/kibana-role-management.html#adding_index_privileges[index privilege]: + +* `read` permission for the `.ml-anomalies-*` index + For more information, go to {ml-docs}/setup.html[Set up {ml-features}]. [IMPORTANT] diff --git a/docs/detections/alert-suppression.asciidoc b/docs/detections/alert-suppression.asciidoc index 73f0537840..1d9070bd03 100644 --- a/docs/detections/alert-suppression.asciidoc +++ b/docs/detections/alert-suppression.asciidoc @@ -4,7 +4,9 @@ .Requirements and notices [sidebar] -- -Alert suppression requires a https://www.elastic.co/pricing[Platinum or higher subscription]. +* Alert suppression requires a https://www.elastic.co/pricing[Platinum or higher subscription]. + +* {ml-cap} rules have <> for alert suppression. preview::["Alert suppression is in technical preview for threshold, indicator match, event correlation, and new terms rules. The functionality may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features."] -- @@ -17,7 +19,7 @@ Alert suppression allows you to reduce the number of repeated or duplicate detec * <> (non-sequence queries only) * <> * <> -* <> +* <> Normally, when a rule meets its criteria repeatedly, it creates multiple alerts, one for each time the rule's criteria are met. When alert suppression is configured, duplicate qualifying events are grouped, and only one alert is created for each group. Depending on the rule type, you can configure alert suppression to create alerts each time the rule runs, or once within a specified time window. You can also specify multiple fields to group events by unique combinations of values. diff --git a/docs/serverless/advanced-entity-analytics/ml-requirements.mdx b/docs/serverless/advanced-entity-analytics/ml-requirements.mdx new file mode 100644 index 0000000000..d3c22a4f21 --- /dev/null +++ b/docs/serverless/advanced-entity-analytics/ml-requirements.mdx @@ -0,0 +1,30 @@ +--- +slug: /serverless/security/ml-requirements +title: ((ml-cap)) job and rule requirements +description: Requirements for using ((ml)) jobs and rules. +tags: [ 'serverless', 'security', 'reference', 'manage' ] +--- + + +
+ +To run and create ((ml)) jobs and rules, you need the appropriate user role. + +Additionally, for custom roles, to configure alert suppression for ((ml)) rules, your role needs the following index privilege: + +* `read` permission for the `.ml-anomalies-*` index + +For more information, go to [Set up ((ml-features))](((ml-docs))/setup.html). + + + +Some roles give +access to the results of _all_ ((anomaly-jobs)), irrespective of whether the user +has access to the source indices. Likewise, a user who has full or read-only +access to ((ml-features)) within a given ((kib)) space can view the results of _all_ +((anomaly-jobs)) that are visible in that space. You must carefully consider who +is given these roles and feature privileges; ((anomaly-job)) results may propagate +field values that contain sensitive information from the source indices to the +results. + + diff --git a/docs/serverless/alerts/alert-suppression.mdx b/docs/serverless/alerts/alert-suppression.mdx new file mode 100644 index 0000000000..f6cc5b08ef --- /dev/null +++ b/docs/serverless/alerts/alert-suppression.mdx @@ -0,0 +1,117 @@ +--- +slug: /serverless/security/alert-suppression +title: Suppress detection alerts +description: Reduce noise from rules that create repeated or duplicate alerts. +tags: [ 'serverless', 'security', 'how-to' ] +status: in review +--- + + +
+ + + - ((ml-cap)) rules have additional requirements for alert suppression. + + - Alert suppression is in technical preview for threshold, indicator match, event correlation, and new terms rules. The functionality may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. + + +Alert suppression allows you to reduce the number of repeated or duplicate detection alerts created by these detection rule types: + +* +* +* +* (non-sequence queries only) +* +* +* + +Normally, when a rule meets its criteria repeatedly, it creates multiple alerts, one for each time the rule's criteria are met. When alert suppression is configured, duplicate qualifying events are grouped, and only one alert is created for each group. Depending on the rule type, you can configure alert suppression to create alerts each time the rule runs, or once within a specified time window. You can also specify multiple fields to group events by unique combinations of values. + +The ((security-app)) displays several indicators in the Alerts table and the alert details flyout when a detection alert is created with alert suppression enabled. You can view the original events associated with suppressed alerts by investigating the alert in Timeline. + + +Alert suppression is not available for Elastic prebuilt rules. However, if you want to suppress alerts for a prebuilt rule, you can duplicate it, then configure alert suppression on the duplicated rule. + + +## Configure alert suppression + +You can configure alert suppression when you create or edit a supported rule type. Refer to documentation for creating , , , , , , or rules for detailed instructions. + +1. When configuring the rule type (the **Define rule** step for a new rule, or the **Definition** tab for an existing rule), specify how you want to group events for alert suppression: + + * **Custom query rule, indicator match, threshold, event correlation (non-sequence queries only), new terms, ((esql)), or ((ml)) rules:** In **Suppress alerts by**, enter 1-3 field names to group events by the fields' values. + * **Threshold rule:** In **Group by**, enter up to 3 field names to group events by the fields' values, or leave the setting empty to group all qualifying events together. + + + If you specify a field with multiple values, alerts with that field are handled as follows: + + * **Custom query or threshold rules:** Alerts are grouped by each unique value. For example, if you suppress alerts by `destination.ip` of `[127.0.0.1, 127.0.0.2, 127.0.0.3]`, alerts will be suppressed separately for each value of `127.0.0.1`, `127.0.0.2`, and `127.0.0.3`. + * **Indicator match, event correlation (non-sequence queries only), new terms, ((esql)), or ((ml)) rules:** Alerts with the specified field name and identical array values are grouped together. For example, if you suppress alerts by `destination.ip` of `[127.0.0.1, 127.0.0.2, 127.0.0.3]`, alerts with the entire array are grouped and only one alert is created for the group. + + + +1. If available, select how often to create alerts for duplicate events: + + + Both options are available for custom query, indicator match, event correlation, new terms, ((esql)), and ((ml)) rules. Threshold rules only have the **Per time period** option. + + + * **Per rule execution**: Create an alert each time the rule runs and an event meets its criteria. + * **Per time period**: Create one alert for all qualifying events that occur within a specified time window, beginning from when an event first meets the rule criteria and creates the alert. + + For example, if a rule runs every 5 minutes but you don't need alerts that frequently, you can set the suppression time period to a longer time, such as 1 hour. If the rule meets its criteria, it creates an alert at that time, and for the next hour, it'll suppress any subsequent qualifying events. + + + +1. Under **If a suppression field is missing**, choose how to handle events with missing suppression fields (events in which one or more of the **Suppress alerts by** fields don't exist): + + + These options are not available for threshold rules. + + + * **Suppress and group alerts for events with missing fields**: Create one alert for each group of events with missing fields. Missing fields get a `null` value, which is used to group and suppress alerts. + * **Do not suppress alerts for events with missing fields**: Create a separate alert for each matching event. This basically falls back to normal alert creation for events with missing suppression fields. + +1. Configure other rule settings, then save and enable the rule. + + +* Use the **Rule preview** before saving the rule to visualize how alert suppression will affect the alerts created, based on historical data. +* If a rule times out while suppression is turned on, try shortening the rule's time or turn off suppression to improve the rule's performance. + + +## Confirm suppressed alerts + +The ((security-app)) displays several indicators of whether a detection alert was created with alert suppression enabled, and how many duplicate alerts were suppressed. + + +After an alert is moved to the `Closed` status, it will no longer suppress new alerts. To prevent interruptions or unexpected changes in suppression, avoid closing alerts before the suppression interval ends. + + +* **Alerts** table — Icon in the **Rule** column. Hover to display the number of suppressed alerts: + + + +* **Alerts** table — Column for suppressed alerts count. Select **Fields** to open the fields browser, then add `kibana.alert.suppression.docs_count` to the table. + + + +* Alert details flyout — **Insights** section: + + + +## Investigate events for suppressed alerts + +With alert suppression, detection alerts aren't created for the grouped source events, but you can still retrieve the events for further analysis or investigation. Do one of the following to open Timeline with the original events associated with both the created alert and the suppressed alerts: + +* **Alerts** table — Select **Investigate in timeline** in the **Actions** column. + + + +* Alert details flyout — Select **Take action** → **Investigate in timeline**. + +## Alert suppression limit by rule type + +Some rule types have a maximum number of alerts that can be suppressed (custom query rules don't have a suppression limit): + +* **Threshold, event correlation (non-sequence queries only, ((esql)), and ((ml)):** The maximum number is the value you choose for the rule's **Max alerts per run** advanced setting, which is `100` by default. +* **Indicator match and new terms:** The maximum number is five times the value you choose for the the rule's **Max alerts per run** advanced setting. The default value is `100`, which means the default maximum limit for indicator match rules and new terms rules is `500`. \ No newline at end of file From 3ac26f352e88010ca7ee46a1b1a6ffa2433d7250 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Wed, 16 Oct 2024 13:33:09 +0000 Subject: [PATCH 2/2] Delete docs/serverless directory and its contents --- .../ml-requirements.mdx | 30 ----- docs/serverless/alerts/alert-suppression.mdx | 117 ------------------ 2 files changed, 147 deletions(-) delete mode 100644 docs/serverless/advanced-entity-analytics/ml-requirements.mdx delete mode 100644 docs/serverless/alerts/alert-suppression.mdx diff --git a/docs/serverless/advanced-entity-analytics/ml-requirements.mdx b/docs/serverless/advanced-entity-analytics/ml-requirements.mdx deleted file mode 100644 index d3c22a4f21..0000000000 --- a/docs/serverless/advanced-entity-analytics/ml-requirements.mdx +++ /dev/null @@ -1,30 +0,0 @@ ---- -slug: /serverless/security/ml-requirements -title: ((ml-cap)) job and rule requirements -description: Requirements for using ((ml)) jobs and rules. -tags: [ 'serverless', 'security', 'reference', 'manage' ] ---- - - -
- -To run and create ((ml)) jobs and rules, you need the appropriate user role. - -Additionally, for custom roles, to configure alert suppression for ((ml)) rules, your role needs the following index privilege: - -* `read` permission for the `.ml-anomalies-*` index - -For more information, go to [Set up ((ml-features))](((ml-docs))/setup.html). - - - -Some roles give -access to the results of _all_ ((anomaly-jobs)), irrespective of whether the user -has access to the source indices. Likewise, a user who has full or read-only -access to ((ml-features)) within a given ((kib)) space can view the results of _all_ -((anomaly-jobs)) that are visible in that space. You must carefully consider who -is given these roles and feature privileges; ((anomaly-job)) results may propagate -field values that contain sensitive information from the source indices to the -results. - - diff --git a/docs/serverless/alerts/alert-suppression.mdx b/docs/serverless/alerts/alert-suppression.mdx deleted file mode 100644 index f6cc5b08ef..0000000000 --- a/docs/serverless/alerts/alert-suppression.mdx +++ /dev/null @@ -1,117 +0,0 @@ ---- -slug: /serverless/security/alert-suppression -title: Suppress detection alerts -description: Reduce noise from rules that create repeated or duplicate alerts. -tags: [ 'serverless', 'security', 'how-to' ] -status: in review ---- - - -
- - - - ((ml-cap)) rules have additional requirements for alert suppression. - - - Alert suppression is in technical preview for threshold, indicator match, event correlation, and new terms rules. The functionality may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. - - -Alert suppression allows you to reduce the number of repeated or duplicate detection alerts created by these detection rule types: - -* -* -* -* (non-sequence queries only) -* -* -* - -Normally, when a rule meets its criteria repeatedly, it creates multiple alerts, one for each time the rule's criteria are met. When alert suppression is configured, duplicate qualifying events are grouped, and only one alert is created for each group. Depending on the rule type, you can configure alert suppression to create alerts each time the rule runs, or once within a specified time window. You can also specify multiple fields to group events by unique combinations of values. - -The ((security-app)) displays several indicators in the Alerts table and the alert details flyout when a detection alert is created with alert suppression enabled. You can view the original events associated with suppressed alerts by investigating the alert in Timeline. - - -Alert suppression is not available for Elastic prebuilt rules. However, if you want to suppress alerts for a prebuilt rule, you can duplicate it, then configure alert suppression on the duplicated rule. - - -## Configure alert suppression - -You can configure alert suppression when you create or edit a supported rule type. Refer to documentation for creating , , , , , , or rules for detailed instructions. - -1. When configuring the rule type (the **Define rule** step for a new rule, or the **Definition** tab for an existing rule), specify how you want to group events for alert suppression: - - * **Custom query rule, indicator match, threshold, event correlation (non-sequence queries only), new terms, ((esql)), or ((ml)) rules:** In **Suppress alerts by**, enter 1-3 field names to group events by the fields' values. - * **Threshold rule:** In **Group by**, enter up to 3 field names to group events by the fields' values, or leave the setting empty to group all qualifying events together. - - - If you specify a field with multiple values, alerts with that field are handled as follows: - - * **Custom query or threshold rules:** Alerts are grouped by each unique value. For example, if you suppress alerts by `destination.ip` of `[127.0.0.1, 127.0.0.2, 127.0.0.3]`, alerts will be suppressed separately for each value of `127.0.0.1`, `127.0.0.2`, and `127.0.0.3`. - * **Indicator match, event correlation (non-sequence queries only), new terms, ((esql)), or ((ml)) rules:** Alerts with the specified field name and identical array values are grouped together. For example, if you suppress alerts by `destination.ip` of `[127.0.0.1, 127.0.0.2, 127.0.0.3]`, alerts with the entire array are grouped and only one alert is created for the group. - - - -1. If available, select how often to create alerts for duplicate events: - - - Both options are available for custom query, indicator match, event correlation, new terms, ((esql)), and ((ml)) rules. Threshold rules only have the **Per time period** option. - - - * **Per rule execution**: Create an alert each time the rule runs and an event meets its criteria. - * **Per time period**: Create one alert for all qualifying events that occur within a specified time window, beginning from when an event first meets the rule criteria and creates the alert. - - For example, if a rule runs every 5 minutes but you don't need alerts that frequently, you can set the suppression time period to a longer time, such as 1 hour. If the rule meets its criteria, it creates an alert at that time, and for the next hour, it'll suppress any subsequent qualifying events. - - - -1. Under **If a suppression field is missing**, choose how to handle events with missing suppression fields (events in which one or more of the **Suppress alerts by** fields don't exist): - - - These options are not available for threshold rules. - - - * **Suppress and group alerts for events with missing fields**: Create one alert for each group of events with missing fields. Missing fields get a `null` value, which is used to group and suppress alerts. - * **Do not suppress alerts for events with missing fields**: Create a separate alert for each matching event. This basically falls back to normal alert creation for events with missing suppression fields. - -1. Configure other rule settings, then save and enable the rule. - - -* Use the **Rule preview** before saving the rule to visualize how alert suppression will affect the alerts created, based on historical data. -* If a rule times out while suppression is turned on, try shortening the rule's time or turn off suppression to improve the rule's performance. - - -## Confirm suppressed alerts - -The ((security-app)) displays several indicators of whether a detection alert was created with alert suppression enabled, and how many duplicate alerts were suppressed. - - -After an alert is moved to the `Closed` status, it will no longer suppress new alerts. To prevent interruptions or unexpected changes in suppression, avoid closing alerts before the suppression interval ends. - - -* **Alerts** table — Icon in the **Rule** column. Hover to display the number of suppressed alerts: - - - -* **Alerts** table — Column for suppressed alerts count. Select **Fields** to open the fields browser, then add `kibana.alert.suppression.docs_count` to the table. - - - -* Alert details flyout — **Insights** section: - - - -## Investigate events for suppressed alerts - -With alert suppression, detection alerts aren't created for the grouped source events, but you can still retrieve the events for further analysis or investigation. Do one of the following to open Timeline with the original events associated with both the created alert and the suppressed alerts: - -* **Alerts** table — Select **Investigate in timeline** in the **Actions** column. - - - -* Alert details flyout — Select **Take action** → **Investigate in timeline**. - -## Alert suppression limit by rule type - -Some rule types have a maximum number of alerts that can be suppressed (custom query rules don't have a suppression limit): - -* **Threshold, event correlation (non-sequence queries only, ((esql)), and ((ml)):** The maximum number is the value you choose for the rule's **Max alerts per run** advanced setting, which is `100` by default. -* **Indicator match and new terms:** The maximum number is five times the value you choose for the the rule's **Max alerts per run** advanced setting. The default value is `100`, which means the default maximum limit for indicator match rules and new terms rules is `500`. \ No newline at end of file