From e4d20a267ecab950890054144bdce9eac18c3d81 Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Tue, 18 Mar 2025 14:59:25 -0400 Subject: [PATCH 1/6] First draft --- .../detections-logsdb-impact.asciidoc | 18 +++++++++++++++--- 1 file changed, 15 insertions(+), 3 deletions(-) diff --git a/docs/detections/detections-logsdb-impact.asciidoc b/docs/detections/detections-logsdb-impact.asciidoc index f24b8a0d13..1275da9ca7 100644 --- a/docs/detections/detections-logsdb-impact.asciidoc +++ b/docs/detections/detections-logsdb-impact.asciidoc @@ -1,7 +1,11 @@ [[detections-logsdb-index-mode-impact]] = Using logsdb index mode with {elastic-sec} -NOTE: To use the {ref}/mapping-source-field.html#synthetic-source[synthetic `_source`] feature, you must have the appropriate subscription. Refer to the subscription page for https://www.elastic.co/subscriptions/cloud[Elastic Cloud] and {subscriptions}[Elastic Stack/self-managed] for the breakdown of available features and their associated subscription tiers. +.Requirements +[sidebar] +-- +To use the {ref}/mapping-source-field.html#synthetic-source[synthetic `_source`] feature, you must have the appropriate subscription. Refer to the subscription page for https://www.elastic.co/subscriptions/cloud[Elastic Cloud] and {subscriptions}[Elastic Stack/self-managed] for the breakdown of available features and their associated subscription tiers. +-- This topic explains the impact of using logsdb index mode with {elastic-sec}. @@ -9,9 +13,15 @@ With logsdb index mode, the original `_source` field is not stored in the index When the `_source` is reconstructed, {ref}/mapping-source-field.html#synthetic-source-modifications[modifications] are possible. Therefore, there could be a mismatch between users' expectations and how fields are formatted. -Continue reading to find out how this affects specific {elastic-sec} components. +Continue reading to learn how logsdb index mode affects CPU and storage usage and how it affects specific {elastic-sec} components. -NOTE: Logsdb index mode is fully supported, and is recommended for new {elastic-sec} deployments. Logsdb is not recommended for existing {elastic-sec} deployments unless users fully understand and accept the documented changes to detection alert documents, runtime fields, and rule actions (refer to the sections below), and have ensured that their deployment has sufficient excess hot data tier CPU capacity to support the logsdb ingesting and indexing process. Enabling logsdb without sufficient excess hot data tier CPU capacity may result in data ingestion backups and or security detection rule timeouts and errors. +NOTE: Logsdb index mode is fully supported, and is recommended for all {elastic-sec} deployments. Users with existing {elastic-sec} deployments are advised to fully understand and accept the documented changes to detection alert documents, runtime fields, and rule actions (refer to the sections below), and ensure that their deployment has sufficient excess hot data tier CPU capacity to support the logsdb ingest and indexing process. Enabling logsdb without sufficient excess hot data tier CPU capacity may result in data ingestion backups and/or security detection rule timeouts and errors. + +[discrete] +[[logsdb-cpu-storage-effects]] +== CPU and storage effects + +Logsdb index mode significantly reduces storage needs by using slightly more CPU during ingest. After enabling logsdb for your data sources, you may need to adjust cluster sizing in response to the new CPU and storage needs. To learn more about how logsdb index mode optimized CPU and storage usage, check out https://www.elastic.co/search-labs/blog/elasticsearch-logsdb-index-mode[our blog]. [discrete] [[logsdb-alerts]] @@ -65,3 +75,5 @@ The following will not work with synthetic source (logsdb index mode enabled): ---- "source": """ emit(params._source['agent.name'] + "_____" + doc['agent.name'].value ); """ ---- + +Also note that runtime fields with scripts that reference `params._source` may need to be updated. Scripts that currently use dotted field names to access source fields will need to be converted to use the nested access pattern instead, unless the object being accessed has `subobjects` set to `false`. Fields that are not mapped also need to be accessed in scripts using the nested access pattern (for example, `params._source['foo']['bar']['baz']` or `params._source.foo.bar.baz`, not `params._source['foo.bar.baz']`). To learn more about how synthetic source names fields and changes that you may need to make to your scripts, refer to {ref}/mapping-source-field.html#synthetic-source-modifications-field-names[Fields named as they are mapped]. \ No newline at end of file From 0c74bdafc27358510fced49cd07329f885baa01b Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Tue, 18 Mar 2025 15:29:04 -0400 Subject: [PATCH 2/6] Redundant --- docs/detections/detections-logsdb-impact.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/detections/detections-logsdb-impact.asciidoc b/docs/detections/detections-logsdb-impact.asciidoc index 1275da9ca7..59893441b8 100644 --- a/docs/detections/detections-logsdb-impact.asciidoc +++ b/docs/detections/detections-logsdb-impact.asciidoc @@ -13,7 +13,7 @@ With logsdb index mode, the original `_source` field is not stored in the index When the `_source` is reconstructed, {ref}/mapping-source-field.html#synthetic-source-modifications[modifications] are possible. Therefore, there could be a mismatch between users' expectations and how fields are formatted. -Continue reading to learn how logsdb index mode affects CPU and storage usage and how it affects specific {elastic-sec} components. +Continue reading to learn how logsdb index mode affects CPU and storage usage and specific {elastic-sec} components. NOTE: Logsdb index mode is fully supported, and is recommended for all {elastic-sec} deployments. Users with existing {elastic-sec} deployments are advised to fully understand and accept the documented changes to detection alert documents, runtime fields, and rule actions (refer to the sections below), and ensure that their deployment has sufficient excess hot data tier CPU capacity to support the logsdb ingest and indexing process. Enabling logsdb without sufficient excess hot data tier CPU capacity may result in data ingestion backups and/or security detection rule timeouts and errors. From b88dcdcf5d684d43fecd6bb0ae9b1e5003272829 Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Tue, 18 Mar 2025 16:41:03 -0400 Subject: [PATCH 3/6] Update detections-logsdb-impact.asciidoc --- docs/detections/detections-logsdb-impact.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/detections/detections-logsdb-impact.asciidoc b/docs/detections/detections-logsdb-impact.asciidoc index 59893441b8..e789e2e412 100644 --- a/docs/detections/detections-logsdb-impact.asciidoc +++ b/docs/detections/detections-logsdb-impact.asciidoc @@ -18,8 +18,8 @@ Continue reading to learn how logsdb index mode affects CPU and storage usage an NOTE: Logsdb index mode is fully supported, and is recommended for all {elastic-sec} deployments. Users with existing {elastic-sec} deployments are advised to fully understand and accept the documented changes to detection alert documents, runtime fields, and rule actions (refer to the sections below), and ensure that their deployment has sufficient excess hot data tier CPU capacity to support the logsdb ingest and indexing process. Enabling logsdb without sufficient excess hot data tier CPU capacity may result in data ingestion backups and/or security detection rule timeouts and errors. [discrete] -[[logsdb-cpu-storage-effects]] -== CPU and storage effects +[[logsdb-cpu-storage]] +== CPU and storage Logsdb index mode significantly reduces storage needs by using slightly more CPU during ingest. After enabling logsdb for your data sources, you may need to adjust cluster sizing in response to the new CPU and storage needs. To learn more about how logsdb index mode optimized CPU and storage usage, check out https://www.elastic.co/search-labs/blog/elasticsearch-logsdb-index-mode[our blog]. From 5ed6baa727691bec956ac3c003ad5ed7943e065c Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Tue, 18 Mar 2025 16:44:03 -0400 Subject: [PATCH 4/6] change tense --- docs/detections/detections-logsdb-impact.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/detections/detections-logsdb-impact.asciidoc b/docs/detections/detections-logsdb-impact.asciidoc index e789e2e412..3765c81d76 100644 --- a/docs/detections/detections-logsdb-impact.asciidoc +++ b/docs/detections/detections-logsdb-impact.asciidoc @@ -21,7 +21,7 @@ NOTE: Logsdb index mode is fully supported, and is recommended for all {elastic- [[logsdb-cpu-storage]] == CPU and storage -Logsdb index mode significantly reduces storage needs by using slightly more CPU during ingest. After enabling logsdb for your data sources, you may need to adjust cluster sizing in response to the new CPU and storage needs. To learn more about how logsdb index mode optimized CPU and storage usage, check out https://www.elastic.co/search-labs/blog/elasticsearch-logsdb-index-mode[our blog]. +Logsdb index mode significantly reduces storage needs by using slightly more CPU during ingest. After enabling logsdb for your data sources, you may need to adjust cluster sizing in response to the new CPU and storage needs. To learn more about how logsdb index mode optimizes CPU and storage usage, check out https://www.elastic.co/search-labs/blog/elasticsearch-logsdb-index-mode[our blog]. [discrete] [[logsdb-alerts]] From 73ae97374aa3ead55acbc6ece00d30bb1b1336e6 Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Fri, 21 Mar 2025 09:35:56 -0400 Subject: [PATCH 5/6] Small fixes --- docs/detections/detections-logsdb-impact.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/detections/detections-logsdb-impact.asciidoc b/docs/detections/detections-logsdb-impact.asciidoc index 3765c81d76..45da66b00a 100644 --- a/docs/detections/detections-logsdb-impact.asciidoc +++ b/docs/detections/detections-logsdb-impact.asciidoc @@ -21,7 +21,7 @@ NOTE: Logsdb index mode is fully supported, and is recommended for all {elastic- [[logsdb-cpu-storage]] == CPU and storage -Logsdb index mode significantly reduces storage needs by using slightly more CPU during ingest. After enabling logsdb for your data sources, you may need to adjust cluster sizing in response to the new CPU and storage needs. To learn more about how logsdb index mode optimizes CPU and storage usage, check out https://www.elastic.co/search-labs/blog/elasticsearch-logsdb-index-mode[our blog]. +Logsdb index mode significantly reduces storage needs by using slightly more CPU during ingest. After enabling logsdb index mode for your data sources, you may need to adjust cluster sizing in response to the new CPU and storage needs. To learn more about how logsdb index mode optimizes CPU and storage usage, check out https://www.elastic.co/search-labs/blog/elasticsearch-logsdb-index-mode[our blog]. [discrete] [[logsdb-alerts]] @@ -76,4 +76,4 @@ The following will not work with synthetic source (logsdb index mode enabled): "source": """ emit(params._source['agent.name'] + "_____" + doc['agent.name'].value ); """ ---- -Also note that runtime fields with scripts that reference `params._source` may need to be updated. Scripts that currently use dotted field names to access source fields will need to be converted to use the nested access pattern instead, unless the object being accessed has `subobjects` set to `false`. Fields that are not mapped also need to be accessed in scripts using the nested access pattern (for example, `params._source['foo']['bar']['baz']` or `params._source.foo.bar.baz`, not `params._source['foo.bar.baz']`). To learn more about how synthetic source names fields and changes that you may need to make to your scripts, refer to {ref}/mapping-source-field.html#synthetic-source-modifications-field-names[Fields named as they are mapped]. \ No newline at end of file +Also note that runtime fields with scripts that reference `params._source` may need to be updated. Scripts that currently use dotted field names to access source fields must be converted to use the nested access pattern instead, unless the object being accessed has `subobjects` set to `false`. Fields that are not mapped also need to be accessed in scripts using the nested access pattern (for example, `params._source['foo']['bar']['baz']` or `params._source.foo.bar.baz`, not `params._source['foo.bar.baz']`). To learn more about how synthetic source names fields and changes that you may need to make to your scripts, refer to {ref}/mapping-source-field.html#synthetic-source-modifications-field-names[Fields named as they are mapped]. \ No newline at end of file From e706b990c6fb20a2c676be38603d88300c9fdaf2 Mon Sep 17 00:00:00 2001 From: "nastasha.solomon" Date: Fri, 21 Mar 2025 09:43:23 -0400 Subject: [PATCH 6/6] One more change --- docs/detections/detections-logsdb-impact.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/detections/detections-logsdb-impact.asciidoc b/docs/detections/detections-logsdb-impact.asciidoc index 45da66b00a..588825dc69 100644 --- a/docs/detections/detections-logsdb-impact.asciidoc +++ b/docs/detections/detections-logsdb-impact.asciidoc @@ -15,7 +15,7 @@ When the `_source` is reconstructed, {ref}/mapping-source-field.html#synthetic-s Continue reading to learn how logsdb index mode affects CPU and storage usage and specific {elastic-sec} components. -NOTE: Logsdb index mode is fully supported, and is recommended for all {elastic-sec} deployments. Users with existing {elastic-sec} deployments are advised to fully understand and accept the documented changes to detection alert documents, runtime fields, and rule actions (refer to the sections below), and ensure that their deployment has sufficient excess hot data tier CPU capacity to support the logsdb ingest and indexing process. Enabling logsdb without sufficient excess hot data tier CPU capacity may result in data ingestion backups and/or security detection rule timeouts and errors. +NOTE: Logsdb index mode is fully supported, and is recommended for all {elastic-sec} deployments. Users with existing {elastic-sec} deployments are advised to fully understand and accept the documented changes to detection alert documents, runtime fields, and rule actions (refer to the sections below), and ensure that their deployment has sufficient excess hot data tier CPU capacity to support the logsdb ingest and indexing process. Enabling logsdb index mode without sufficient excess hot data tier CPU capacity may result in data ingestion backups and/or security detection rule timeouts and errors. [discrete] [[logsdb-cpu-storage]]