From 381d6acbbc867de6bc4734176807b84eff18a22a Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Tue, 25 Mar 2025 14:50:00 -0400 Subject: [PATCH 1/5] First draft --- solutions/security/detect-and-alert.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/solutions/security/detect-and-alert.md b/solutions/security/detect-and-alert.md index ecfeea2ee4..67df689885 100644 --- a/solutions/security/detect-and-alert.md +++ b/solutions/security/detect-and-alert.md @@ -42,25 +42,26 @@ To make sure you can access Detections and manage rules, see [Detections require -## Compatibility with cold and frozen tier nodes [cold-tier-detections] +## Manage data in cold and frozen tiers [cold-tier-detections] ```yaml {applies_to} stack: ``` -Cold and frozen [data tiers](/manage-data/lifecycle/data-tiers.md) hold time series data that is only accessed occasionally. In {{stack}} version >=7.11.0, {{elastic-sec}} supports cold but not frozen tier data for the following {{es}} indices: +Cold data tiers store time series data that is accessed infrequently and rarely updated, while frozen data tiers hold time series data that is accessed even less frequently and never updated. If you are automating searches across different [data tiers](/manage-data/lifecycle/data-tiers.md) using rules, consider the following best practices and limitations. -* Index patterns specified in `securitySolution:defaultIndex` -* Index patterns specified in the definitions of detection rules, except for indicator match rules -* Index patterns specified in the data sources selector on various {{security-app}} pages +### Best practices [best-practices-data-tiers] -{{elastic-sec}} does **NOT** support either cold or frozen tier data for the following {{es}} indices: +* **Retention in hot tier**: We recommend keeping data in the hot tier for at least 24 hours. {{ilm-cap}} policies that roll over data more frequently than once every 24 hours can increase the volume of frozen data queried by rules, leading to performance issues. +* **Replicas for Mission-Critical Data**: Your data should have replicas if it must be constantly available. Since frozen tiers don't support replicas, shard unavailability can cause partial rule run failures. Shard unavailability may be also encountered during or after {stack} upgrades. If this happens, you can [manually run](/solutions/security/detect-and-alert/manage-detection-rules.md#manually-run-rules) the rule over the affected time period. -* Index patterns controlled by {{elastic-sec}}, including alerts and list indices -* Index patterns specified in the definition of indicator match rules +### Limitations [limitations-data-tiers] -Using either cold or frozen tier data for unsupported indices may result in detection rule timeouts and overall performance degradation. +Data tiers are a powerful and useful tool. When using them, keep the following limitations in mind: +* {{ilm-cap}} policies for indices controlled by {{elastic-sec}}, including alerts and list indices, must not be modified. +* Indicator match rule performance can be severely impacted by querying data in frozen tiers. +* Cold and frozen source data must have an {{ilm}} policy that keeps it in the hot or warm tiers for at least one day. ## Limited support for indicator match rules [support-indicator-rules] From 31d0aeadddea7ff7dbe45491f7aa29daa65901b9 Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Tue, 25 Mar 2025 14:55:10 -0400 Subject: [PATCH 2/5] lowercase --- solutions/security/detect-and-alert.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/solutions/security/detect-and-alert.md b/solutions/security/detect-and-alert.md index 67df689885..e87d8b3359 100644 --- a/solutions/security/detect-and-alert.md +++ b/solutions/security/detect-and-alert.md @@ -53,7 +53,7 @@ Cold data tiers store time series data that is accessed infrequently and rarely ### Best practices [best-practices-data-tiers] * **Retention in hot tier**: We recommend keeping data in the hot tier for at least 24 hours. {{ilm-cap}} policies that roll over data more frequently than once every 24 hours can increase the volume of frozen data queried by rules, leading to performance issues. -* **Replicas for Mission-Critical Data**: Your data should have replicas if it must be constantly available. Since frozen tiers don't support replicas, shard unavailability can cause partial rule run failures. Shard unavailability may be also encountered during or after {stack} upgrades. If this happens, you can [manually run](/solutions/security/detect-and-alert/manage-detection-rules.md#manually-run-rules) the rule over the affected time period. +* **Replicas for mission-critical data**: Your data should have replicas if it must be constantly available. Since frozen tiers don't support replicas, shard unavailability can cause partial rule run failures. Shard unavailability may be also encountered during or after {stack} upgrades. If this happens, you can [manually run](/solutions/security/detect-and-alert/manage-detection-rules.md#manually-run-rules) the rule over the affected time period. ### Limitations [limitations-data-tiers] From 42a8acd346644f0b15dc879c1a7f4939a89f3f8f Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Wed, 2 Apr 2025 13:17:10 -0400 Subject: [PATCH 3/5] Marshall's input --- solutions/security/detect-and-alert.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/solutions/security/detect-and-alert.md b/solutions/security/detect-and-alert.md index c047dc42ab..0eb123c0f8 100644 --- a/solutions/security/detect-and-alert.md +++ b/solutions/security/detect-and-alert.md @@ -53,7 +53,7 @@ Cold data tiers store time series data that is accessed infrequently and rarely ### Best practices [best-practices-data-tiers] * **Retention in hot tier**: We recommend keeping data in the hot tier for at least 24 hours. {{ilm-cap}} policies that roll over data more frequently than once every 24 hours can increase the volume of frozen data queried by rules, leading to performance issues. -* **Replicas for mission-critical data**: Your data should have replicas if it must be constantly available. Since frozen tiers don't support replicas, shard unavailability can cause partial rule run failures. Shard unavailability may be also encountered during or after {stack} upgrades. If this happens, you can [manually run](/solutions/security/detect-and-alert/manage-detection-rules.md#manually-run-rules) the rule over the affected time period. +* **Replicas for mission-critical data**: Your data should have replicas if it must be highly available. Since frozen tiers don't support replicas, shard unavailability can cause partial rule run failures. Shard unavailability may be also encountered during or after {stack} upgrades. If this happens, you can [manually rerun](/solutions/security/detect-and-alert/manage-detection-rules.md#manually-run-rules) rules over the affected time period once the shards are available. ### Limitations [limitations-data-tiers] @@ -61,7 +61,7 @@ Data tiers are a powerful and useful tool. When using them, keep the following l * {{ilm-cap}} policies for indices controlled by {{elastic-sec}}, including alerts and list indices, must not be modified. * Indicator match rule performance can be severely impacted by querying data in frozen tiers. -* Cold and frozen source data must have an {{ilm}} policy that keeps it in the hot or warm tiers for at least one day. +* Source data must have an {{ilm}} policy that keeps it in the hot or warm tiers for at least one day before moving to cold or frozen tiers. ## Limited support for indicator match rules [support-indicator-rules] From b7133d6e5f84a8964415c44d88c96d64d883c277 Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Fri, 4 Apr 2025 14:20:53 -0400 Subject: [PATCH 4/5] Mike P.'s input --- solutions/security/detect-and-alert.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/solutions/security/detect-and-alert.md b/solutions/security/detect-and-alert.md index 0eb123c0f8..3426feb388 100644 --- a/solutions/security/detect-and-alert.md +++ b/solutions/security/detect-and-alert.md @@ -48,12 +48,12 @@ To make sure you can access Detections and manage rules, see [Detections require stack: ``` -Cold data tiers store time series data that is accessed infrequently and rarely updated, while frozen data tiers hold time series data that is accessed even less frequently and never updated. If you are automating searches across different [data tiers](/manage-data/lifecycle/data-tiers.md) using rules, consider the following best practices and limitations. +Cold [data tiers](/manage-data/lifecycle/data-tiers.md) store time series data that's accessed infrequently and rarely updated, while frozen data tiers hold time series data that's accessed even less frequently and never updated. If you're automating searches across different data tiers using rules, consider the following best practices and limitations. ### Best practices [best-practices-data-tiers] -* **Retention in hot tier**: We recommend keeping data in the hot tier for at least 24 hours. {{ilm-cap}} policies that roll over data more frequently than once every 24 hours can increase the volume of frozen data queried by rules, leading to performance issues. -* **Replicas for mission-critical data**: Your data should have replicas if it must be highly available. Since frozen tiers don't support replicas, shard unavailability can cause partial rule run failures. Shard unavailability may be also encountered during or after {stack} upgrades. If this happens, you can [manually rerun](/solutions/security/detect-and-alert/manage-detection-rules.md#manually-run-rules) rules over the affected time period once the shards are available. +* **Retention in hot tier**: We recommend keeping data in the hot tier ({{ilm-cap}} hot phase) for at least 24 hours. {{ilm-cap}} policies that move ingested data from the hot phase to another phase (for example, cold or frozen) in less than 24 hours may cause performance issues and/or rule execution errors. +* **Replicas for mission-critical data**: Your data should have replicas if it must be highly available. Since frozen tiers don't support replicas, shard unavailability can cause partial rule run failures. Shard unavailability may be also encountered during or after {{stack}} upgrades. If this happens, you can [manually rerun](/solutions/security/detect-and-alert/manage-detection-rules.md#manually-run-rules) rules over the affected time period once the shards are available. ### Limitations [limitations-data-tiers] From 1023403c1f4517c236e631b42f15af4589140124 Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Thu, 10 Apr 2025 16:11:06 -0400 Subject: [PATCH 5/5] Same fixes --- solutions/security/detect-and-alert.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/solutions/security/detect-and-alert.md b/solutions/security/detect-and-alert.md index 3426feb388..36975003b4 100644 --- a/solutions/security/detect-and-alert.md +++ b/solutions/security/detect-and-alert.md @@ -57,11 +57,10 @@ Cold [data tiers](/manage-data/lifecycle/data-tiers.md) store time series data t ### Limitations [limitations-data-tiers] -Data tiers are a powerful and useful tool. When using them, keep the following limitations in mind: +Data tiers are a powerful and useful tool. When using them, keep the following in mind: -* {{ilm-cap}} policies for indices controlled by {{elastic-sec}}, including alerts and list indices, must not be modified. -* Indicator match rule performance can be severely impacted by querying data in frozen tiers. -* Source data must have an {{ilm}} policy that keeps it in the hot or warm tiers for at least one day before moving to cold or frozen tiers. +* To avoid rule failures, do not modify {{ilm}} policies for {elastic-sec}-controlled indices, such as alert and list indices. +* Source data must have an {{ilm}} policy that keeps it in the hot or warm tiers for at least 24 hours before moving to cold or frozen tiers. ## Limited support for indicator match rules [support-indicator-rules]