From 2e664aa73b1ed360316e66dfd6916d0946551039 Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Wed, 12 Feb 2025 14:05:13 -0600 Subject: [PATCH 1/4] add content --- manage-data/lifecycle.md | 60 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 56 insertions(+), 4 deletions(-) diff --git a/manage-data/lifecycle.md b/manage-data/lifecycle.md index e7a5400759..ee9b8c0ca4 100644 --- a/manage-data/lifecycle.md +++ b/manage-data/lifecycle.md @@ -4,13 +4,65 @@ mapped_urls: - https://www.elastic.co/guide/en/serverless/current/project-setting-data.html --- -# Data lifecycle - % What needs to be done: Refine % GitHub issue: docs-projects#374 % Use migrated content from existing pages that map to this page: -% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/data-management.md -% - [ ] ./raw-migrated-files/docs-content/serverless/project-setting-data.md \ No newline at end of file +% - [x] ./raw-migrated-files/elasticsearch/elasticsearch-reference/data-management.md +% - [ ] ./raw-migrated-files/docs-content/serverless/project-setting-data.md + +# Data lifecycle + +The data you store in {{es}} generally falls into one of two categories: + +* **Content**: a collection of items you want to search, such as a catalog of products +* **Time series data**: a stream of continuously-generated timestamped data, such as log entries + +**Content** might be frequently updated, but the value of the content remains relatively constant over time. You want to be able to retrieve items quickly regardless of how old they are. + +**Time series data** keeps accumulating over time, so you need strategies for balancing the value of the data against the cost of storing it. As it ages, it tends to become less important and less-frequently accessed, so you can move it to less expensive, less performant hardware. For your oldest data, what matters is that you have access to the data. It’s ok if queries take longer to complete. + +To help you manage your data, {{es}} offers you the following options: {{ilm-cap}}, Data stream lifecycle, and Elastic Curator. + +::::{note} +[Data rollup](/manage-data/lifecycle/rollup.md) is a deprecated {{es}} feature that allows you to manage the amount of data that is stored in your cluster, similar to the downsampling functionality of {{ilm-init}} and data stream lifecycle. This feature should not be used for new deployments. +:::: + +## {{ilm-init}} [ilm] + +::::{tip} +{{ilm-init}} is not available on {{es-serverless}}. + +:::{dropdown} Why? +In an {{ecloud}} or self-managed environment, ILM lets you automatically transition indices through data tiers according to your performance needs and retention requirements. This allows you to balance hardware costs with performance. {{es-serverless}} eliminates this complexity by optimizing your cluster performance for you. + +[Data stream lifecycle](#data-stream-lifecycle) is an optimized lifecycle tool that lets you focus on the most common lifecycle management needs, without unnecessary hardware-centric concepts like data tiers. +::: +:::: + +**{{ilm-init}}** can be used to manage both indices and data streams. It allows you to do the following: + +* Define the retention period of your data. The retention period is the minimum time your data will be stored in {{es}}. Data older than this period can be deleted by {{es}}. +* Define [multiple tiers](/manage-data/lifecycle/data-tiers.md) of data nodes with different performance characteristics. +* Automatically transition indices through the data tiers according to your performance needs and retention policies. +* Leverage [searchable snapshots](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md) stored in a remote repository to provide resiliency for your older indices while reducing operating costs and maintaining search performance. +* Perform [asynchronous searches](/solutions/search/async-search-api.md) of data stored on less-performant hardware. + +**[Read more in {{ilm-cap}} ->](/manage-data/lifecycle/index-lifecycle-management.md)** + +## Data stream lifecycle [data-stream-lifecycle] + +**Data stream lifecycle** is less feature rich but is focused on simplicity. It allows you to do the following: + +* Define the retention period of your data. The retention period is the minimum time your data will be stored in {{es}}. Data older than this period can be deleted by {{es}} at a later time. +* Improve the performance of your data stream by performing background operations that will optimize the way your data stream is stored. + +**[Read more in Data stream lifecycle ->](/manage-data/lifecycle/data-stream.md)** + +## Elastic Curator [elastic-curator] + +**Elastic Curator** is a tool that allows you to manage your indices and snapshots using user-defined filters and predefined actions. If ILM provides the functionality to manage your index lifecycle, and you have at least a Basic license, consider using ILM in place of Curator. Many stack components make use of ILM by default. + +**[Read more in Elastic Curator ->](/manage-data/lifecycle/curator.md)** From 0a448ffd8697058b78e61ffbc908f8a6eb855c81 Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Wed, 12 Feb 2025 14:06:08 -0600 Subject: [PATCH 2/4] put deprecated product last --- manage-data/toc.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/manage-data/toc.yml b/manage-data/toc.yml index 91b422474b..d6d363de34 100644 --- a/manage-data/toc.yml +++ b/manage-data/toc.yml @@ -98,7 +98,7 @@ toc: - file: ingest/sample-data.md - file: ingest/transform-enrich.md children: - - file: ingest/transform-enrich/ingest-pipelines-serverless.md + - file: ingest/transform-enrich/ingest-pipelines-serverless.md - file: ingest/transform-enrich/ingest-pipelines.md children: - file: ingest/transform-enrich/example-parse-logs.md @@ -138,6 +138,7 @@ toc: - file: lifecycle/data-stream/tutorial-update-existing-data-stream.md - file: lifecycle/data-stream/tutorial-data-stream-retention.md - file: lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md + - file: lifecycle/curator.md - file: lifecycle/rollup.md children: - file: lifecycle/rollup/getting-started-with-rollups.md @@ -145,7 +146,6 @@ toc: - file: lifecycle/rollup/rollup-aggregation-limitations.md - file: lifecycle/rollup/rollup-search-limitations.md - file: lifecycle/rollup/migrating-from-rollup-to-downsampling.md - - file: lifecycle/curator.md - file: migrate.md children: - file: migrate/migrate-from-a-self-managed-cluster-with-a-self-signed-certificate-using-remote-reindex.md From 6215a35e99cb98fa8bc74d6922351d9522d33b04 Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Thu, 13 Feb 2025 10:23:08 -0600 Subject: [PATCH 3/4] remove mapped page, clean up comments --- manage-data/lifecycle.md | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/manage-data/lifecycle.md b/manage-data/lifecycle.md index ee9b8c0ca4..0c3a612c65 100644 --- a/manage-data/lifecycle.md +++ b/manage-data/lifecycle.md @@ -1,18 +1,8 @@ --- mapped_urls: - https://www.elastic.co/guide/en/elasticsearch/reference/current/data-management.html - - https://www.elastic.co/guide/en/serverless/current/project-setting-data.html --- -% What needs to be done: Refine - -% GitHub issue: docs-projects#374 - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/elasticsearch/elasticsearch-reference/data-management.md -% - [ ] ./raw-migrated-files/docs-content/serverless/project-setting-data.md - # Data lifecycle The data you store in {{es}} generally falls into one of two categories: From 84e5d7d9b763513194200c0b3c2ee4ce8e92eb00 Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Thu, 13 Feb 2025 10:29:38 -0600 Subject: [PATCH 4/4] add mapped page --- manage-data/ingest/tools.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/manage-data/ingest/tools.md b/manage-data/ingest/tools.md index cac5389533..fe82b7d985 100644 --- a/manage-data/ingest/tools.md +++ b/manage-data/ingest/tools.md @@ -3,6 +3,7 @@ mapped_urls: - https://www.elastic.co/guide/en/cloud/current/ec-cloud-ingest-data.html - https://www.elastic.co/guide/en/fleet/current/beats-agent-comparison.html - https://www.elastic.co/guide/en/kibana/current/connect-to-elasticsearch.html + - https://www.elastic.co/guide/en/serverless/current/project-setting-data.html - https://www.elastic.co/customer-success/data-ingestion - https://github.com/elastic/ingest-docs/pull/1373 --- @@ -28,4 +29,4 @@ mapped_urls: $$$supported-outputs-beats-and-agent$$$ -$$$additional-capabilities-beats-and-agent$$$ \ No newline at end of file +$$$additional-capabilities-beats-and-agent$$$