diff --git a/manage-data/lifecycle/curator.md b/manage-data/lifecycle/curator.md index aa6e65b5be..9549157bd8 100644 --- a/manage-data/lifecycle/curator.md +++ b/manage-data/lifecycle/curator.md @@ -7,6 +7,8 @@ applies_to: # Elasticsearch Curator -Similar to {{ilm-cap}} ({{ilm-init}}), Elasticsearch Curator can help you manage index lifecycles. **If {{ilm-init}} provides the functionality to manage your index lifecycle and you have at least a Basic license, use {{ilm-init}} instead of Curator.** Many {{stack}} components use {{ilm-init}} by default. +{{es}} Curator is a tool that helps you automate and manage index lifecycles, providing advanced control over your {{es}} data. While most use cases are now handled by {{ilm-cap}} ({{ilm-init}})—included with a Basic license—Curator remains a valuable option for scenarios requiring additional functionality or customization. -If you're looking for additional functionality for managing your index lifecycle, you can read more about how Elasticsearch Curator may help in [Curator index management](curator://reference/index.md). +* If {{ilm-init}} provides the functionality to manage your index lifecycle and you have at least a Basic license, use {{ilm-init}} instead of Curator. Many {{stack}} components use {{ilm-init}} by default. + +* If you're looking for additional functionality for managing your index lifecycle, you can read more in [Curator index management](curator://reference/index.md) about how {{es}} Curator can help for your use case. diff --git a/manage-data/lifecycle/data-stream.md b/manage-data/lifecycle/data-stream.md index 50dae71b30..7f076a6660 100644 --- a/manage-data/lifecycle/data-stream.md +++ b/manage-data/lifecycle/data-stream.md @@ -1,4 +1,5 @@ --- +navigation_title: Data stream lifecycle mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/data-stream-lifecycle.html applies_to: @@ -8,19 +9,21 @@ products: - id: elasticsearch --- -# Data stream lifecycle [data-stream-lifecycle] +# Data stream lifecycle in {{es}} [data-stream-lifecycle] -A data stream lifecycle is the built-in mechanism [data streams](/manage-data/data-store/data-streams.md) use to manage their lifecycle. It enables you to easily automate the management of your data streams according to your retention requirements. For example, you could configure the lifecycle to: +A data stream lifecycle in {{es}} is a built-in automation mechanism for managing data retention, performance, and storage optimization. By configuring rollover, retention, and downsampling rules for your [data streams](/manage-data/data-store/data-streams.md), you can ensure that time-series data is efficiently maintained and that older data is removed when no longer needed. This page explains how the lifecycle works, its key features, and how to configure it for both new and existing data streams. -* Ensure that data indexed in the data stream will be kept at least for the retention time you defined. -* Ensure that data older than the retention period will be deleted automatically by {{es}} at a later time. +Data stream lifecycle manages your data streams according to your retention requirements. For example, you can configure the lifecycle to: -To achieve that, it supports: +* Ensure that data indexed in the data stream is kept at least for the retention time you defined. +* Ensure that data older than the retention period is deleted automatically by {{es}} at a later time. + +To achieve these, data stream lifecycle supports: * Automatic [rollover](index-lifecycle-management/rollover.md), which chunks your incoming data in smaller pieces to facilitate better performance and backwards incompatible mapping changes. * Configurable retention, which allows you to configure the time period for which your data is guaranteed to be stored. {{es}} is allowed at a later time to delete data older than this time period. Retention can be configured on the data stream level or on a global level. Read more about the different options in this [tutorial](data-stream/tutorial-data-stream-retention.md). -A data stream lifecycle also supports downsampling the data stream backing indices. See [the downsampling example](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-lifecycle) for more details. +Data stream lifecycle also supports downsampling the data stream backing indices. Refer to [the downsampling example](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-lifecycle) for more details. ## Data stream lifecycle availability @@ -43,31 +46,21 @@ In intervals configured by [`data_streams.lifecycle.poll_interval`](elasticsearc ::::{important} We use the `generation_time` instead of the creation time because this ensures that all data in the backing index have passed the retention period. As a result, the retention period is not the exact time data gets deleted, but the minimum time data will be stored. :::: - - ::::{note} Steps `2-4` apply only to backing indices that are not already managed by {{ilm-init}}, meaning that these indices either do not have an {{ilm-init}} policy defined, or if they do, they have [`index.lifecycle.prefer_ilm`](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#index-lifecycle-prefer-ilm) set to `false`. :::: - - ## Configuring data stream lifecycle [data-stream-lifecycle-configuration] Since the lifecycle is configured on the data stream level, the process to configure a lifecycle on a new data stream and on an existing one differ. Four tutorials are available to help you set up and manage data streams with data stream lifecycle: -* To create a new data stream with a lifecycle, add the data stream lifecycle as part of the index template that matches the name of your data stream. See [Tutorial: Create a data stream with a lifecycle](data-stream/tutorial-create-data-stream-with-lifecycle.md) for the detailed steps. When a write operation with the name of your data stream reaches {{es}} then the data stream will be created with the respective data stream lifecycle. -* To update the lifecycle settings for an individual, existing data stream, use the [data stream lifecycle APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-data-stream). See [Tutorial: Update existing data stream](data-stream/tutorial-update-existing-data-stream.md) for details. -* Retention settings for data streams can be configured both individually, at the data stream level, and globally, for all data streams in a cluster. To learn more, refer to [Tutorial: Configure data stream retention](/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md). -* To migrate an existing {{ilm-init}} managed data stream to data stream lifecycle, follow the steps in [Tutorial: Migrate ILM managed data stream to data stream lifecycle](data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md). +* To create a new data stream with a lifecycle, add the data stream lifecycle as part of the index template that matches the name of your data stream. Refer to [](data-stream/tutorial-create-data-stream-with-lifecycle.md) for the detailed steps. When a write operation with the name of your data stream reaches {{es}} then the data stream is created with the respective data stream lifecycle. +* To update the lifecycle settings for an individual, existing data stream, use the [data stream lifecycle APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-data-stream). Refer to [](data-stream/tutorial-update-existing-data-stream.md) for details. +* Retention settings for data streams can be configured both individually, at the data stream level, and globally, for all data streams in a cluster. To learn more, refer to [](/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md). +* To migrate an existing {{ilm-init}} managed data stream to data stream lifecycle, follow the steps in [](data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md). ::::{note} Updating the data stream lifecycle of an existing data stream is different from updating the settings or the mapping, because it is applied on the data stream level and not on the individual backing indices. :::: - - - - - - diff --git a/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md b/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md index 6677096b5a..7a950b08c7 100644 --- a/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md +++ b/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md @@ -8,9 +8,9 @@ products: - id: elasticsearch --- -# Tutorial: Create a data stream with a lifecycle [tutorial-manage-new-data-stream] +# Creating a data stream with a lifecycle [tutorial-manage-new-data-stream] -To create a data stream with a built-in lifecycle, follow these steps: +Follow these steps to create an {{es}} data stream with a configured lifecycle. Learn how to set the retention period for your data and to retrieve the lifecycle configuration details. 1. [Create an index template](#create-index-template-with-lifecycle) 2. [Create a data stream](#create-data-stream-with-lifecycle) diff --git a/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md b/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md index b6aabb53c6..017af6c49c 100644 --- a/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md +++ b/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md @@ -8,9 +8,11 @@ products: - id: elasticsearch --- -# Tutorial: Configure data stream retention [tutorial-manage-data-stream-retention] +# Setting retention for {{es}} data streams [tutorial-manage-data-stream-retention] -This tutorial demonstrates lifecycle retention, showing how to define, configure, and apply it. Keep in mind that the following options apply only to data streams that are managed by the data stream lifecycle. +This tutorial shows how to configure retention settings for data streams in {{es}}, helping you to control storage costs and compliance. Learn to adjust lifecycle policies so old data is deleted or moved automatically. + +The following options apply only to data streams managed by data stream lifecycle. 1. [What is data stream retention?](#what-is-retention) 2. [How to configure retention?](#retention-configuration) diff --git a/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md b/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md index 2c86131781..a9ee0de488 100644 --- a/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md +++ b/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md @@ -8,10 +8,11 @@ products: - id: elasticsearch --- -# Tutorial: Migrate ILM managed data stream to data stream lifecycle [tutorial-migrate-data-stream-from-ilm-to-dsl] +# Migrating {{ilm-init}}-managed data streams to data stream lifecycle [tutorial-migrate-data-stream-from-ilm-to-dsl] -In this tutorial we’ll look at migrating an existing data stream from [Index Lifecycle Management ({{ilm-init}})](../index-lifecycle-management.md) to [data stream lifecycle](../data-stream.md). The existing {{ilm-init}} managed backing indices will continue to be managed by {{ilm-init}} until they age out and get deleted by {{ilm-init}}; however, the new backing indices will be managed by data stream lifecycle. This way, a data stream is gradually migrated away from being managed by {{ilm-init}} to being managed by data stream lifecycle. As we’ll see, {{ilm-init}} and data stream lifecycle can co-manage a data stream; however, an index can only be managed by one system at a time. +This tutorial describes how to migrate a data stream from [Index Lifecycle Management ({{ilm-init}})](../index-lifecycle-management.md) to the newer [data stream lifecycle](../data-stream.md). It explains migration steps, compatibility considerations, and validation best practices. +During the migration, existing {{ilm-init}} managed backing indices continue to be managed by {{ilm-init}} until they age out and are deleted by {{ilm-init}}. Newly created backing indices are managed by data stream lifecycle. This way, a data stream is gradually migrated from being managed by {{ilm-init}} to being managed by data stream lifecycle. {{ilm-init}} and data stream lifecycle can co-manage a data stream, however an index can be managed by only one system at a time. :::{admonition} Configure data retention policies for Streams :applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"} @@ -25,8 +26,7 @@ To migrate a data stream from {{ilm-init}} to data stream lifecycle using APIs y 1. Update the index template that’s backing the data stream to set [prefer_ilm](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#index-lifecycle-prefer-ilm) to `false`, and to configure data stream lifecycle. 2. Configure the data stream lifecycle for the *existing* data stream using the [lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-lifecycle). -For more details see the [migrate to data stream lifecycle](#migrate-from-ilm-to-dsl) section. - +For more details refer to [Migrate to data stream lifecycle](#migrate-from-ilm-to-dsl). ## Setup ILM managed data stream [setup-test-data] diff --git a/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md b/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md index c6c20387f0..7276ff29ea 100644 --- a/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md +++ b/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md @@ -8,9 +8,9 @@ products: - id: elasticsearch --- -# Tutorial: Update the lifecycle of a data stream [tutorial-manage-existing-data-stream] +# Update the lifecycle of a data stream [tutorial-manage-existing-data-stream] -Follow these steps to configure or remove data stream lifecycle settings for an existing, individual data stream: +Follow these steps to configure or remove data stream lifecycle settings for an existing, individual data stream. - [Set a data stream’s lifecycle](#set-lifecycle) - [Remove the lifecycle for a data stream](#delete-lifecycle) diff --git a/manage-data/lifecycle/data-tiers.md b/manage-data/lifecycle/data-tiers.md index fc6bdec9e5..ed59bf1414 100644 --- a/manage-data/lifecycle/data-tiers.md +++ b/manage-data/lifecycle/data-tiers.md @@ -1,4 +1,5 @@ --- +navigation_title: Data tiers mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/data-tiers.html - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-disable-data-tier.html @@ -11,9 +12,11 @@ products: - id: cloud-hosted --- -# Data tiers +# {{es}} data tiers: hot, warm, cold, and frozen storage explained -A *data tier* is a collection of [nodes](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md) within a cluster that share the same [data node role](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#node-roles), and a hardware profile that’s appropriately sized for the role. Elastic recommends that nodes in the same tier share the same hardware profile to avoid [hot spotting](/troubleshoot/elasticsearch/hotspotting.md). +{{es}} organizes data into storage tiers to balance performance, cost, and accessibility. Each tier—from hot for frequently accessed data to frozen for rarely queried datasets—has specific hardware and storage characteristics. This guide explains how to configure, manage, and automate data placement across tiers for both time series and general content data. + +Each *data tier* is a collection of [nodes](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md) in an {{es}} cluster that share the same [data node role](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#node-roles), and a hardware profile that’s appropriately sized for the role. Elastic recommends that nodes in the same tier share the same hardware profile to avoid [hot spotting](/troubleshoot/elasticsearch/hotspotting.md). :::{admonition} Serverless manages data storage for you By abstracting cluster management tasks, {{serverless-full}} adjusts data storage and scaling based on your workload. Certain [project settings](/deploy-manage/deploy/elastic-cloud/project-settings.md) allow you to customize how your data is stored and calibrate the performance of your data. @@ -468,7 +471,7 @@ When {{es}} creates an index as part of a [data stream](/manage-data/data-store/ At the time of index creation, you can override the default setting by explicitly setting the preferred value in one of two ways: -* Using an [index template](/manage-data/data-store/templates.md). Refer to [Automate rollover with ILM](/manage-data/lifecycle/index-lifecycle-management.md) for details. +* Using an [index template](/manage-data/data-store/templates.md). Refer to [](/manage-data/lifecycle/index-lifecycle-management.md) for details. * Within the [create index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create) request body. You can override this setting after index creation by [updating the index setting](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings) to the preferred value. diff --git a/manage-data/lifecycle/index-lifecycle-management.md b/manage-data/lifecycle/index-lifecycle-management.md index c1ef6bb1d3..add40b0cd6 100644 --- a/manage-data/lifecycle/index-lifecycle-management.md +++ b/manage-data/lifecycle/index-lifecycle-management.md @@ -1,4 +1,5 @@ --- +navigation_title: Index lifecycle managment mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html - https://www.elastic.co/guide/en/elasticsearch/reference/current/overview-index-lifecycle-management.html @@ -12,9 +13,11 @@ products: - id: elasticsearch --- -# Index lifecycle management +# {{ilm-cap}} ({{ilm-init}}) in {{es}} -{{ilm-cap}} ({{ilm-init}}) provides an integrated and streamlined way to manage your time series data. You can configure {{ilm-init}} policies to automatically manage indices according to your performance, resiliency, and retention requirements. For example, you could use {{ilm-init}} to: +{{ilm}} ({{ilm-init}}) automates the management of time-based indices, such as logs and metrics. Using {{ILM-init}} policies, you can streamline index rollover, retention, and deletion to optimize performance, reliability, and storage costs for your data. + +For example, you can use {{ilm-init}} to: * Spin up a new index when an index reaches a certain size or number of documents * Create a new index each day, week, or month and archive previous ones @@ -74,10 +77,10 @@ For example, if you are indexing metrics data from a fleet of ATMs into Elastics You can create and manage index lifecycle policies through {{kib}}'s [Index Management](/manage-data/data-store/index-basics.md#index-management) UI or the [{{ilm-init}} APIs]({{es-apis}}group/endpoint-ilm). For more details on creating and managing index lifecycle policies refer to: -* [Configure a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md) -* [View the lifecycle status of an index or datastream](/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md) -* [Update or switch a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/policy-updates.md) -* [Restore a managed data stream or index](/manage-data/lifecycle/index-lifecycle-management/restore-managed-data-stream-index.md) +* [Create an {{ilm}} policy](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md) +* [Check the {{ilm-init}} status of {{es}} indices and data streams](/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md) +* [Update or switch an {{es}} index lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/policy-updates.md) +* [Restore managed indices and manage {{ilm-init}} actions](/manage-data/lifecycle/index-lifecycle-management/restore-managed-data-stream-index.md) * [](/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md) Default {{ilm}} policies are created automatically when you install an [Elastic Integration](integration-docs://reference/index.md), or when you use {{agent}}, {{beats}}, or the {{ls}} {{es}} output plugin to send data to the {{stack}}. @@ -90,7 +93,7 @@ To automatically back up your indices and manage snapshots, use [snapshot lifecy ## Pausing and troubleshooting {{ilm-init}} -In case you want to temporarily pause the {{ilm-init}} service while you perform maintenance, make other changes to your cluster, or do any troubleshooting, refer to [Start and stop {{ilm-init}}](/manage-data/lifecycle/index-lifecycle-management/start-stop-index-lifecycle-management.md). +In case you want to temporarily pause the {{ilm-init}} service while you perform maintenance, make other changes to your cluster, or do any troubleshooting, refer to [](/manage-data/lifecycle/index-lifecycle-management/start-stop-index-lifecycle-management.md). In the event of any issues running {{ilm-init}}, refer to [Fix index lifecycle management errors](/troubleshoot/elasticsearch/index-lifecycle-management-errors.md) for detailed troubleshooting guidance. @@ -98,8 +101,8 @@ In the event of any issues running {{ilm-init}}, refer to [Fix index lifecycle m For existing hot-warm deployments that are currently using index curation, migrating to ILM gives you more fine-grained control over the lifecycle of each index. Read more in: -* [Manage existing indices](/manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md) -* [Migrate to index lifecycle management](/manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md) -* [Migrate index allocation filters to node roles](/manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md) +* [](/manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md) +* [](/manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md) +* [](/manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md) -You can also set up {{ilm-init}} to manage an existing set of indices that do not already have a managed lifecycle. The {{ilm-init}} policy that you apply should not contain a rollover action, because the policy won't be carried forward when the rollover action creates a new index. Refer to [Manually apply a lifecycle policy to an index](/manage-data/lifecycle/index-lifecycle-management/policy-apply.md) to learn more. \ No newline at end of file +You can also set up {{ilm-init}} to manage an existing set of indices that do not already have a managed lifecycle. The {{ilm-init}} policy that you apply should not contain a rollover action, because the policy isn't carried forward when the rollover action creates a new index. Refer to [](/manage-data/lifecycle/index-lifecycle-management/policy-apply.md) to learn more. \ No newline at end of file diff --git a/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md b/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md index f2f305b3a1..5f4145eb6e 100644 --- a/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md +++ b/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md @@ -1,5 +1,5 @@ --- -navigation_title: Configure a lifecycle policy +navigation_title: Create an {{ilm-init}} policy mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html applies_to: @@ -8,15 +8,17 @@ products: - id: elasticsearch --- -# Configure a lifecycle policy [set-up-lifecycle-policy] +# Create an index lifecycle management policy in {{es}} [set-up-lifecycle-policy] -An [{{ilm}}](/manage-data/lifecycle/index-lifecycle-management.md) ({{ilm-init}}) policy defines how your indices are managed over time, automating when and how they transition as they age. You can use {{ilm-init}} to manage both indices and [data streams](/manage-data/data-store/data-streams.md). There are fewer configuration steps required to set up ILM with data streams. In comparison, configuring ILM with indices requires you to create an initial managed index and alias in addition to defining a policy and creating a template to apply it. This page describes the steps to configure an {{ilm-init}} lifecycle policy for both scenarios. +An [{{ilm}}](/manage-data/lifecycle/index-lifecycle-management.md) ({{ilm-init}}) policy defines how indices transition through different phases over time. This guide explains how to create a new {{ilm-init}} policy with configurable rollover, retention, and deletion rules, and then apply the policy using an index template. + +You can use {{ilm-init}} to manage both indices and [data streams](/manage-data/data-store/data-streams.md). There are fewer configuration steps required to set up ILM with data streams. In comparison, configuring ILM with indices requires you to create an initial managed index and alias in addition to defining a policy and creating a template to apply it. This page describes the steps to configure an {{ilm-init}} lifecycle policy for both scenarios. :::{note} This page is specifically about using {{ilm-init}} with indices or data streams. If you're looking for a simpler data streams lifecycle management option that focuses on a data retention period, refer to [Data stream lifecycle](/manage-data/lifecycle/data-stream.md). Check [Data lifecycle](/manage-data/lifecycle.md) to compare these lifecycle management options. ::: -**Consider these aspects when configuring an {{ilm-init}} policy:** +**Consider these aspects when creating an {{ilm-init}} policy:** * To manage an index or data stream with {{ilm-init}}, you need to specify a valid policy in the `index.lifecycle.name` index setting. diff --git a/manage-data/lifecycle/index-lifecycle-management/ilm-tutorials.md b/manage-data/lifecycle/index-lifecycle-management/ilm-tutorials.md index 0e0cccceb7..406eb1b1d2 100644 --- a/manage-data/lifecycle/index-lifecycle-management/ilm-tutorials.md +++ b/manage-data/lifecycle/index-lifecycle-management/ilm-tutorials.md @@ -1,20 +1,21 @@ --- -navigation_title: ILM tutorials +navigation_title: ILM automation tutorials applies_to: stack: ga products: - id: elasticsearch --- -# {{ilm-cap}} tutorials +# ILM automation tutorials A collection of tutorials is available to help you control the lifecycle of your data using {{ilm}}. ## Configuring rollover -When you continuously index timestamped documents into {{es}}, you typically use a [data stream](../../data-store/data-streams.md) so you can periodically [roll over](rollover.md) to a new index. This enables you to implement a [hot-warm-cold architecture](../data-tiers.md) to meet the performance requirements for your newest data, control costs over time, enforce retention policies, and still get the most out of your data. -To simplify index management and automate rollover, select one of the scenarios that best applies to your situation: +{{ilm}} ({{ilm-init}}) in {{es}} is a powerful feature that helps automate the management of your indices through their entire lifecycle — from creation to rollover, to retention and deletion. Whether you're handling large volumes of time series data or general content, {{ilm-init}} enables you to optimize performance, control storage costs, and enforce data retention policies with ease. These tutorials guide you step-by-step through setting up {{ilm-init}} policies, configuring automated rollover, and monitoring your indices, so you can efficiently manage your {{es}} data lifecycle. + +When you continuously index timestamped documents into {{es}}, you typically use a [data stream](../../data-store/data-streams.md) so you can periodically [roll over](rollover.md) to a new index. To simplify index management and automate rollover, select one of the scenarios that best applies to your situation: * **Roll over data streams with ILM.** When ingesting write-once, timestamped data that doesn't change, follow the steps in [](/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-with-data-streams.md) for simple, automated data stream rollover. ILM-managed backing indices are automatically created under a single data stream alias. ILM also tracks and transitions the backing indices through the lifecycle automatically. * **Roll over time series indices with ILM.** Data streams are best suited for [append-only](../../data-store/data-streams.md#data-streams-append-only) use cases. If you need to update or delete existing time series data, you can perform update or delete operations directly on the data stream backing index. If you frequently send multiple documents using the same `_id` expecting last-write-wins, you may want to use an index alias with a write index instead. You can still use {{ilm-init}} to manage and roll over the alias’s indices. Follow the steps in [](/manage-data/lifecycle/index-lifecycle-management/tutorial-time-series-without-data-streams.md) for more information. diff --git a/manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md b/manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md index 3161eca1f9..211fbb419f 100644 --- a/manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md +++ b/manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md @@ -1,5 +1,5 @@ --- -navigation_title: Index lifecycle +navigation_title: ILM phases and actions mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html applies_to: @@ -8,8 +8,9 @@ products: - id: elasticsearch --- -# Index lifecycle [ilm-index-lifecycle] +# {{ilm-cap}} phases and actions [ilm-index-lifecycle] +{{ilm-cap}} ({{ilm-init}}) in {{es}} defines how indices transition through different phases—hot, warm, cold, frozen, and delete—based on age and usage. Each phase supports specific actions, from rollover and force merge to searchable snapshots and delete. This guide explains the purpose of each phase, the timing of transitions, and the actions available to manage index performance and storage efficiently. [{{ilm-cap}}](../index-lifecycle-management.md) ({{ilm-init}}) defines five index lifecycle *phases*: diff --git a/manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md b/manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md index 643b1c8ab3..547f2c5307 100644 --- a/manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md +++ b/manage-data/lifecycle/index-lifecycle-management/manage-existing-indices.md @@ -28,7 +28,7 @@ If you’ve been using [](/manage-data/lifecycle/curator.md) or some other mecha The simplest way to transition to managing your periodic indices with {{ilm-init}} is to [configure an index template](configure-lifecycle-policy.md#apply-policy-template) to apply a lifecycle policy to new indices. A number of [example {{ilm-init}} polices](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md#ilm-rollover-ex) are available, showing how index rollover can be initiated based on different criteria. -Once the index you are writing to is being managed by {{ilm-init}}, you can [manually apply a policy](/manage-data/lifecycle/index-lifecycle-management/policy-apply.md) to your older indices. Define a separate policy for your older indices that omits the rollover action. Rollover is used to manage where new data goes, so isn’t applicable. +Once the index you are writing to is being managed by {{ilm-init}}, you can manually [apply a policy to your older indices](/manage-data/lifecycle/index-lifecycle-management/policy-apply.md). Define a separate policy for your older indices that omits the rollover action. Rollover is used to manage where new data goes, so isn’t applicable. Keep in mind that policies applied to existing indices compare the `min_age` for each phase to the original creation date of the index, and might proceed through multiple phases immediately. If your policy performs resource-intensive operations like force merge, you don’t want to have a lot of indices performing those operations all at once when you switch over to {{ilm-init}}. diff --git a/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md b/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md index f18a4c4008..e854912800 100644 --- a/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md +++ b/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md @@ -6,7 +6,9 @@ products: - id: elasticsearch --- -# Manage the lifecycle policy for integrations data [ilm-manage-lifecycle-policy-integrations-data] +# Managing lifecycle polices for integrations data [ilm-manage-lifecycle-policy-integrations-data] + +Learn to apply and manage lifecycle policies for integrations data stored in {{es}}. This documentation covers default policies for Elastic integrations, how to override them, and how to align retention rules with your requirements. An Elastic integration is a pre-packaged collection of assets that provides an effective, simplified way to monitor a product, system, or service, with minimal required setup. Most integrations rely on {{agent}} as an ingest mechanism, and the policies used to govern installed integrations are managed in {{fleet}}. diff --git a/manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md b/manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md index c7dce31ab3..9c27efd5cf 100644 --- a/manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md +++ b/manage-data/lifecycle/index-lifecycle-management/migrate-index-allocation-filters-to-node-roles.md @@ -7,9 +7,10 @@ products: - id: elasticsearch --- -# Migrate index allocation filters to node roles [migrate-index-allocation-filters] +# Migrate index allocation filters to {{ilm-init}} node roles [migrate-index-allocation-filters] -If you currently use [custom node attributes](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#custom-node-attributes) and [attribute-based allocation filters](../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md) to move indices through [data tiers](../data-tiers.md) in a [hot-warm-cold architecture](https://www.elastic.co/blog/implementing-hot-warm-cold-in-elasticsearch-with-index-lifecycle-management), we recommend that you switch to using the built-in node roles and automatic [data tier allocation](../data-tiers.md#data-tier-allocation). Using node roles enables {{ilm-init}} to automatically move indices between data tiers. + +If your cluster currently relies on [custom node attributes](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md#custom-node-attributes) and [attribute-based allocation filters](../../../deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md) to move indices between [data tiers](../data-tiers.md), migrating to {{es}} node roles simplifies management. Node roles work with {{ilm}} ({{ilm-init}}) to automatically move indices through hot, warm, cold, and frozen tiers, reducing manual configuration and ensuring efficient shard allocation. ::::{note} While we recommend relying on automatic data tier allocation to manage your data in a hot-warm-cold architecture, you can still use attribute-based allocation filters to control shard allocation for other purposes. diff --git a/manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md b/manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md index 19a6d70ee2..e834b1c215 100644 --- a/manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md +++ b/manage-data/lifecycle/index-lifecycle-management/migrate-index-management.md @@ -12,7 +12,7 @@ products: - id: cloud-hosted --- -# Migrate to {{ilm-init}} on {{ech}} or {{ece}} deployments +# Migrate to {{ilm}} ({{ilm-init}}) on {{ech}} or {{ece}} deployments ::::{important} Index curation is deprecated. Any deployments using index curation are prompted to migrate to ILM. diff --git a/manage-data/lifecycle/index-lifecycle-management/policy-apply.md b/manage-data/lifecycle/index-lifecycle-management/policy-apply.md index d50d10a1f9..1b70570b1f 100644 --- a/manage-data/lifecycle/index-lifecycle-management/policy-apply.md +++ b/manage-data/lifecycle/index-lifecycle-management/policy-apply.md @@ -1,16 +1,15 @@ --- -navigation_title: Manually apply a policy to an index +navigation_title: Apply a policy to an existing index applies_to: stack: ga products: - id: elasticsearch --- -# Manually apply a lifecycle policy to an index [apply-policy-manually] +# Apply an index lifecycle policy to an existing {{es}} index [apply-policy-manually] -When you create a new {{es}} index, it automatically picks up settings, mappings, and aliases from any matching [index template](/manage-data/data-store/templates.md#index-templates) and its component templates. If the template specifies a lifecycle policy, that policy is applied to the new index as soon as it's created so that the index will be managed by {{ilm-init}}. This process is described in detail in [Configure a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md). -You can also apply a lifecycle policy manually to existing indices that are not yet managed, as described on this page. This is useful if you want to: +While index templates are typically used to automatically apply {{ilm}} ({{ilm-init}}) policies to new indices, you might need to apply a policy to an existing index. This page explains how to manually assign an {{ilm-init}} policy to an index through {{kib}} or using the {{es}} API, ensuring your existing data is managed according to lifecycle rules. This is useful if you want to: * Configure the indices to move through different [data tiers](/manage-data/lifecycle/data-tiers.md) as they age. * Perform [lifecycle actions](elasticsearch://reference/elasticsearch/index-lifecycle-actions/index.md) such as downsampling or shrinking. * Delete the indices when they reach a certain age. @@ -18,7 +17,7 @@ You can also apply a lifecycle policy manually to existing indices that are not :::{warning} Consider the following before manually applying a policy to an index: -* If an index is currently managed by an ILM policy you must first remove that policy before applying a new one. To remove or make changes to an {{ilm-init}} policy currently applied to one or more indices, or to switch indices to use a different policy, refer to [Update or switch a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/policy-updates.md). +* If an index is currently managed by an {{ilm-init}} policy you must first remove that policy before applying a new one. To remove or make changes to an {{ilm-init}} policy currently applied to one or more indices, or to switch indices to use a different policy, refer to [Update or switch an {{es}} index lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/policy-updates.md). * Do not manually apply a policy that uses the rollover action. Policies that use rollover must be applied by the [index template](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md#apply-policy-template). Otherwise, the policy is not carried forward when the rollover action creates a new index. For an approach to migrating existing indices that supports rollover, refer to [](/manage-data/lifecycle/index-lifecycle-management/migrate-ilm.md). ::: diff --git a/manage-data/lifecycle/index-lifecycle-management/policy-updates.md b/manage-data/lifecycle/index-lifecycle-management/policy-updates.md index 60c8bc76d5..0047da73c4 100644 --- a/manage-data/lifecycle/index-lifecycle-management/policy-updates.md +++ b/manage-data/lifecycle/index-lifecycle-management/policy-updates.md @@ -1,4 +1,5 @@ --- +navigation_title: Update or switch an index lifecycle policy mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/update-lifecycle-policy.html applies_to: @@ -7,15 +8,15 @@ products: - id: elasticsearch --- -# Update or switch a lifecycle policy +# Update or switch an {{es}} index lifecycle policy -You can change how the lifecycle of an index or collection of rolling indices is managed by modifying the current policy or switching to a different policy. +{{ilm}} ({{ilm-init}}) enables automated handling of index phases and data retention. You can update a policy to modify how indices progress through phases or switch an index to a different policy to change its lifecycle management. This page explains how to safely apply updates using {{kib}} or the {{es}} API. To ensure that policy updates don’t put an index into a state where it can’t exit the current phase, the phase definition is cached in the index metadata when it enters the phase. If changes can be safely applied, {{ilm-init}} updates the cached phase definition. If they cannot, phase execution continues using the cached definition. When the index advances to the next phase, it uses the phase definition from the updated policy. -To configure an index not currently managed by {{ilm-init}} to be governed by an lifecycle policy, refer to [Manually apply a policy to an index](/manage-data/lifecycle/index-lifecycle-management/policy-apply.md). +To configure an index not currently managed by {{ilm-init}} to be governed by an lifecycle policy, refer to [Apply an index lifecycle policy to an existing {{es}} index](/manage-data/lifecycle/index-lifecycle-management/policy-apply.md). ## How changes are applied [ilm-apply-changes] diff --git a/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md b/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md index 8760cd9349..0344c789ab 100644 --- a/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md +++ b/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md @@ -1,5 +1,5 @@ --- -navigation_title: View the lifecycle status of an index or data stream +navigation_title: Check the {{ilm-init}} status of indices and data streams mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/update-lifecycle-policy.html applies_to: @@ -8,11 +8,9 @@ products: - id: elasticsearch --- -# View the lifecycle status of an index or data stream [view-lifecycle-status] +# Check the {{ilm-init}} status of {{es}} indices and data streams [view-lifecycle-status] -For any existing managed index or data stream in your cluster, you can access the {{ilm-init}} policy applied to it and its current status. - -You can view the lifecycle status of an index or data stream in {{kib}} or using the {{es}} API. +{{ilm}} ({{ilm-init}}) automates the handling of index phases and data retention. You can check the current {{ilm-init}} policy applied to an index or data stream and see its lifecycle phase, actions, and other status details using {{kib}} or the {{ilm-init}} Explain API. :::{tip} If you're investigating an {{ilm-init}}-related problem, refer to [Troubleshoot index and snapshot lifecycle management](/troubleshoot/elasticsearch/start-ilm.md) and [Fix index lifecycle management errors](/troubleshoot/elasticsearch/index-lifecycle-management-errors.md) in the {{es}} chapter of the **Troubleshoot** section. diff --git a/manage-data/lifecycle/index-lifecycle-management/restore-managed-data-stream-index.md b/manage-data/lifecycle/index-lifecycle-management/restore-managed-data-stream-index.md index 523de2898f..57d8ef6c43 100644 --- a/manage-data/lifecycle/index-lifecycle-management/restore-managed-data-stream-index.md +++ b/manage-data/lifecycle/index-lifecycle-management/restore-managed-data-stream-index.md @@ -7,13 +7,15 @@ products: - id: elasticsearch --- -# Restore a managed data stream or index [index-lifecycle-and-snapshots] +# Restore managed indices and manage {{ilm-init}} actions [index-lifecycle-and-snapshots] + +When restoring a managed {{es}} index or data stream, {{ilm}} ({{ilm-init}}) resumes running the associated policies automatically. You can temporarily stop {{ilm-init}} to control the running of lifecycle actions, ensuring restored indices progress through phases according to your operational needs. To [restore](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-snapshot-restore) managed indices, ensure that the {{ilm-init}} policies referenced by the indices exist. If necessary, you can restore {{ilm-init}} policies by setting [`include_global_state`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-snapshot-restore) to `true`. -When you restore a managed index or a data stream with managed backing indices, {{ilm-init}} automatically resumes executing the restored indices' policies. A restored index’s `min_age` is relative to when it was originally created or rolled over, not its restoration time. Policy actions are performed on the same schedule whether or not an index has been restored from a snapshot. If you restore an index that was accidentally deleted half way through its month long lifecycle, it proceeds normally through the last two weeks of its lifecycle. +A restored index’s `min_age` is relative to when it was originally created or rolled over, not its restoration time. Policy actions are performed on the same schedule irrespective of whether an index has been restored from a snapshot. If you restore an index that was accidentally deleted half way through its month long lifecycle, it proceeds normally through the last two weeks of its lifecycle. -In some cases, you might want to prevent {{ilm-init}} from immediately executing its policy on a restored index. For example, if you are restoring an older snapshot you might want to prevent it from rapidly progressing through all of its lifecycle phases. You might want to add or update documents before it’s marked read-only or shrunk, or prevent the index from being immediately deleted. +You might want to prevent {{ilm-init}} from immediately executing its policy on a restored index. For example, if you are restoring an older snapshot you might want to prevent it from rapidly progressing through all its lifecycle phases. You might want to add or update documents before it’s marked read-only or shrunk, or prevent the index from being immediately deleted. To prevent {{ilm-init}} from executing a restored index’s policy: diff --git a/manage-data/lifecycle/index-lifecycle-management/rollover.md b/manage-data/lifecycle/index-lifecycle-management/rollover.md index a90e0fd37b..3acf99287e 100644 --- a/manage-data/lifecycle/index-lifecycle-management/rollover.md +++ b/manage-data/lifecycle/index-lifecycle-management/rollover.md @@ -1,4 +1,5 @@ --- +navigation_title: Index rollover and routing allocation mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/index-rollover.html applies_to: @@ -7,14 +8,11 @@ products: - id: elasticsearch --- -# About rollover [index-rollover] +# Index rollover and routing allocation in {{es}} [index-rollover] -In {{es}}, the [rollover action](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md) replaces your active write index with a new one whenever your index grows beyond a specified size, age, or number of documents. -This is particularly useful for time-series data, such as logs or metrics where index growth is continuous, in order to meet performance and retention requirements. +When working with time series data such as logs or metrics, writing to a single index indefinitely can hurt performance and resource usage. {{es}} uses index rollover and routing allocation to optimize ingestion, search, and storage. The [rollover action](elasticsearch://reference/elasticsearch/index-lifecycle-actions/ilm-rollover.md) replaces your active write index with a new one whenever your index grows beyond a specified size, age, or number of documents. This is particularly useful for time-series data, such as logs or metrics where index growth is continuous. Without rollover, a single index would continue to grow, causing search performance to drop and having a higher administrative burden on the cluster. -Without rollover, a single index would continue to grow, causing search performance to drop and having a higher administrative burden on the cluster. - -The rollover feature is an important part of how [index lifecycle](../index-lifecycle-management/index-lifecycle.md) ({{ilm-init}}) and [data stream lifecycles](../data-stream.md) ({{dlm-init}}) work to keep your indices fast and manageable. By switching the write target of an index, the rollover action provides the following benefits: +The rollover feature is an important part of how [{{ilm}}](../index-lifecycle-management/index-lifecycle.md) ({{ilm-init}}) and [data stream lifecycles](../data-stream.md) ({{dlm-init}}) work to keep your indices fast and manageable. By switching the write target of an index, the rollover action provides the following benefits: * **Lifecycle** - works with lifecycle management ({{ilm-init}} or {{dlm-init}}) to transition the index through its lifecycle actions and allows for granular control over retention cycles * **Optimized performance** - keeps shard sizes within [recommended limits](/deploy-manage/production-guidance/optimize-performance/size-shards.md) (10-50 GB) @@ -113,4 +111,4 @@ When an alias or data stream is rolled over, the previous write index’s rollov :::: **Alias-based naming pattern**
-When configured correctly, the newly created write index will have a similar name to one that's been rolled over, however the six-digit, zero-padded suffix will be incremented. For example before rollover, the write index was called `my-index-000001` and after rollover, the newly created index becomes `my-index-000002` and also becomes the new write index. The alias typically shares the base name which in this example is `my-index`. +When configured correctly, the newly created write index has a similar name to one that's been rolled over, however the six-digit, zero-padded suffix is incremented. For example, before rollover the write index was named `my-index-000001`, and after rollover the newly created index is named `my-index-000002` and also becomes the new write index. The alias typically shares the base name which in this example is `my-index`. diff --git a/manage-data/lifecycle/index-lifecycle-management/skip-rollover.md b/manage-data/lifecycle/index-lifecycle-management/skip-rollover.md index 992cb591b0..76139d0336 100644 --- a/manage-data/lifecycle/index-lifecycle-management/skip-rollover.md +++ b/manage-data/lifecycle/index-lifecycle-management/skip-rollover.md @@ -1,4 +1,5 @@ --- +navigation_title: Skip rollover mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/skipping-rollover.html applies_to: @@ -7,16 +8,15 @@ products: - id: elasticsearch --- -# Skip rollover [skipping-rollover] +# Skip rollover in {{ilm}} ({{ilm-init}}) [skipping-rollover] You can use {{ilm}} to manage index lifecycle transitions without rolling over your indices. For example, when working with data that isn't continuously ingested, you might prefer to create a new index on a set schedule, such as the first Monday of each month. In this case, you can still use an {{ilm-init}} policy to automatically move indices through [lifecycle phases](/manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md#ilm-phase-transitions) as they age and control when new indices are created by disabling the rollover action. You can configure indices to skip rollover either as part of an {{ilm-init}} policy or manually by adjusting the index settings. - ## Skip rollover through an {{ilm-init}} policy -When you [configure a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md) in {{kib}}, you can choose whether or not rollover is enabled. +When you [create a lifecycle policy](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md) in {{kib}}, you can choose to enable or disable rollover. To update the automatic rollover setting in an {{ilm-init}} policy: diff --git a/manage-data/lifecycle/index-lifecycle-management/start-stop-index-lifecycle-management.md b/manage-data/lifecycle/index-lifecycle-management/start-stop-index-lifecycle-management.md index 1bd0f96561..1e5b51ff03 100644 --- a/manage-data/lifecycle/index-lifecycle-management/start-stop-index-lifecycle-management.md +++ b/manage-data/lifecycle/index-lifecycle-management/start-stop-index-lifecycle-management.md @@ -10,6 +10,8 @@ products: # Start and stop index lifecycle management [start-stop-ilm] +By default, the {{ilm-init}} service is in a `RUNNING` state and manages all indices that have lifecycle policies. You can stop and restart the {{ilm}} service as needed. + Follow these steps to check the current {{ilm-init}} status, and to stop or restart it as needed. ### Get {{ilm-init}} status diff --git a/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md b/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md index 2dd41cb91a..a067d63e29 100644 --- a/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md +++ b/manage-data/lifecycle/index-lifecycle-management/tutorial-customize-built-in-policies.md @@ -1,4 +1,5 @@ --- +navigation_title: Customize built-in {{ilm-init}} policies mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/example-using-index-lifecycle-policy.html applies_to: @@ -7,11 +8,12 @@ products: - id: elasticsearch --- -# Customize duplicates of built-in {{ilm-init}} policies +# Customize built-in {{ilm-init}} policies in {{es}} -{{es}} includes a set of built-in {{ilm-init}} policies that define how managed indices transition across [data tiers](/manage-data/lifecycle/data-tiers.md) and what [actions](/manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md#ilm-phase-actions), such as rollover, downsampling, or shrinking, are performed at each phase. -This tutorial demonstrates how to create a customized copy of a built-in {{ilm-init}} policy to better fit your data retention, performance, or storage requirements. You should never edit managed policies directly, because updates to {{es}} or Elastic integrations might overwrite those changes. Instead, you can duplicate a built-in policy, modify the duplicate, and assign it to your index or component templates. +{{es}} includes several built-in {{ilm}} (ilm-init) policies to help manage your logs and metrics efficiently. This tutorial shows you how to safely duplicate and customize these policies to optimize [rollover](/manage-data/lifecycle/index-lifecycle-management/rollover.md) and other [actions](/manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md#ilm-phase-actions), control resource usage across hot and warm [data tiers](/manage-data/lifecycle/data-tiers.md), and enforce retention rules for your indices. + +Follow these steps to create a customized copy of a built-in {{ilm-init}} policy. You should never edit managed policies directly, because updates to {{es}} or Elastic integrations might overwrite those changes. Instead, you can duplicate a built-in policy, modify the duplicate, and assign it to your index or component templates. While this tutorial uses [{{agent}}](/reference/fleet/index.md) and its built-in `logs@lifecycle` policy as an example, the same process can be applied to any built-in policies. Common examples include: @@ -35,9 +37,9 @@ The process involves three main steps: Once applied, your customized policy will govern any new indices created after the change. Existing indices will continue to use their current lifecycle policy until they roll over. If you want the policy to take effect immediately, you can manually [roll over](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-rollover) the data stream. :::{tip} -* If you're using [Elastic integrations](https://docs.elastic.co/en/integrations) and are not yet familiar with which data streams are associated with them, refer to [Manage the lifecycle policy for integrations data](/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md). +* If you're using [Elastic integrations](https://docs.elastic.co/en/integrations) and are not yet familiar with which data streams are associated with them, refer to [](/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data.md). -* If you're looking for a more advanced use case, such as customizing an ILM policy for a selected set of data streams in one or more integrations or namespaces, check the set of tutorials in [Customize data retention policies](/reference/fleet/data-streams-ilm-tutorial.md) in the {{fleet}} and {{agent}} reference documentation. +* If you're looking for a more advanced use case, such as customizing an ILM policy for a selected set of data streams in one or more integrations or namespaces, check the set of tutorials in [](/reference/fleet/data-streams-ilm-tutorial.md) in the {{fleet}} and {{agent}} reference documentation. These tutorials go into greater depth about creating and using `@custom` component templates. For example, the tutorial [Apply an ILM policy to all data streams generated from {{fleet}} integrations across all namespaces](/reference/fleet/data-streams-scenario1.md) shows how to create and use the `logs@custom` and `metrics@custom` component templates to customize {{ilm-init}} policies associated with data streams in integrations. ::: diff --git a/manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md b/manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md index 91c5714cb8..2df0cd4168 100644 --- a/manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md +++ b/manage-data/lifecycle/index-lifecycle-management/tutorial-general-content-with-data-streams.md @@ -257,7 +257,7 @@ Create a data stream using the [_data_stream API](https://www.elastic.co/docs/ap PUT /_data_stream/movetods ``` -You can [view the lifecycle status of your data stream](/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md), including details about its associated ILM policy. +You can [check the lifecycle status of your data stream](/manage-data/lifecycle/index-lifecycle-management/policy-view-status.md), including details about its associated {{ilm-init}} policy. ### Optional: Reindex your data with a data stream [manage-general-content-with-data-streams-reindex] diff --git a/manage-data/lifecycle/rollup.md b/manage-data/lifecycle/rollup.md index b6575fa5c3..64607b2e74 100644 --- a/manage-data/lifecycle/rollup.md +++ b/manage-data/lifecycle/rollup.md @@ -41,7 +41,7 @@ Details about setting up and configuring Rollup are covered in [Create Job API]( The Rollup feature exposes a new search endpoint (`/_rollup_search` versus the standard `/_search`) which knows how to search over rolled-up data. Importantly, this endpoint accepts 100% normal {{es}} Query DSL. Your application does not need to learn a new DSL to inspect historical data, it can simply reuse existing queries and dashboards. -There are some limitations to the functionality available; not all queries and aggregations are supported, certain search features (such as highlighting) are disabled, and available fields depend on how the rollup was configured. These limitations are covered more in [Rollup Search limitations](/manage-data/lifecycle/rollup/rollup-search-limitations.md). +There are some limitations to the functionality available; not all queries and aggregations are supported, certain search features (such as highlighting) are disabled, and available fields depend on how the rollup was configured. These limitations are covered more in [](/manage-data/lifecycle/rollup/rollup-search-limitations.md). But if your queries, aggregations and dashboards only use the available functionality, redirecting them to historical data is trivial. diff --git a/manage-data/lifecycle/rollup/getting-started-kibana.md b/manage-data/lifecycle/rollup/getting-started-kibana.md index f06e185966..4dc928f346 100644 --- a/manage-data/lifecycle/rollup/getting-started-kibana.md +++ b/manage-data/lifecycle/rollup/getting-started-kibana.md @@ -9,7 +9,7 @@ products: - id: kibana --- -# Get started with the rollups in {{kib}} +# Get started with rollups in {{kib}} ::::{admonition} Deprecated in 8.11.0. :class: warning @@ -17,7 +17,7 @@ products: Rollups are deprecated and will be removed in a future version. Use [downsampling](/manage-data/data-store/data-streams/downsampling-time-series-data-stream.md) instead. :::: -A rollup job is a periodic task that aggregates data from indices specified by an index pattern, and then rolls it into a new index. Rollup indices are a good way to compactly store months or years of historical data for use in visualizations and reports. +A rollup job is a periodic task that aggregates data from indices specified by an index pattern, and then rolls it into a new index. Rollup indices are a good way to compactly store months or years of historical data for use in visualizations and reports. This documentation covers creating rollup jobs and visualizing rolled-up data. You can go to the **Rollup Jobs** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md). diff --git a/manage-data/lifecycle/rollup/migrating-from-rollup-to-downsampling.md b/manage-data/lifecycle/rollup/migrating-from-rollup-to-downsampling.md index 766ddf4cd6..36eb5459d6 100644 --- a/manage-data/lifecycle/rollup/migrating-from-rollup-to-downsampling.md +++ b/manage-data/lifecycle/rollup/migrating-from-rollup-to-downsampling.md @@ -11,12 +11,12 @@ products: -# Migrating from Rollup to downsampling [rollup-migrating-to-downsampling] +# Migrating from rollups to downsampling in {{es}} [rollup-migrating-to-downsampling] -Rollup and downsampling are two different features that allow historical metrics to be rolled up. From a high level rollup is more flexible compared to downsampling, but downsampling is a more robust and easier feature to downsample metrics. +Understand how to transition from legacy rollup jobs to {{es}} downsampling. This documentation explains key differences between the two and outlines migration steps. -The following aspects of downsampling are easier or more robust: +The following aspects of downsampling are easier or more robust than using rollup jobs: * No need to schedule jobs. Downsampling is integrated with Index Lifecycle Management (ILM) and Data Stream Lifecycle (DSL). * No separate search API. Downsampled indices can be accessed using the search API and {{esql}}. diff --git a/manage-data/lifecycle/rollup/rollup-aggregation-limitations.md b/manage-data/lifecycle/rollup/rollup-aggregation-limitations.md index 58f5a86c96..664d3b6828 100644 --- a/manage-data/lifecycle/rollup/rollup-aggregation-limitations.md +++ b/manage-data/lifecycle/rollup/rollup-aggregation-limitations.md @@ -1,4 +1,5 @@ --- +navigation_title: Rollup aggregation limitations mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/rollup-agg-limitations.html applies_to: @@ -8,7 +9,7 @@ products: - id: elasticsearch --- -# Rollup aggregation limitations [rollup-agg-limitations] +# Rollup aggregation limitations in {{es}} [rollup-agg-limitations] ::::{admonition} Deprecated in 8.11.0. :class: warning @@ -16,9 +17,7 @@ products: Rollups will be removed in a future version. [Migrate](migrating-from-rollup-to-downsampling.md) to [downsampling](../../data-store/data-streams/downsampling-time-series-data-stream.md) instead. :::: - -There are some limitations to how fields can be rolled up / aggregated. This page highlights the major limitations so that you are aware of them. - +When you work with indexed data in {{es}}, there are some limitations to how the document fields can be rolled up or aggregated. This page highlights the major limitations so that you are aware of them. ## Limited aggregation components [_limited_aggregation_components] diff --git a/manage-data/lifecycle/rollup/rollup-search-limitations.md b/manage-data/lifecycle/rollup/rollup-search-limitations.md index 83a67e1ab2..b9371aa12b 100644 --- a/manage-data/lifecycle/rollup/rollup-search-limitations.md +++ b/manage-data/lifecycle/rollup/rollup-search-limitations.md @@ -1,4 +1,5 @@ --- +navigation_title: Rollup search limitations mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/rollup-search-limitations.html applies_to: @@ -8,7 +9,7 @@ products: - id: elasticsearch --- -# Rollup search limitations [rollup-search-limitations] +# Rollup search limitations in {{es}} [rollup-search-limitations] ::::{admonition} Deprecated in 8.11.0. :class: warning @@ -16,8 +17,7 @@ products: Rollups will be removed in a future version. [Migrate](migrating-from-rollup-to-downsampling.md) to [downsampling](../../data-store/data-streams/downsampling-time-series-data-stream.md) instead. :::: - -While we feel the Rollup function is extremely flexible, the nature of summarizing data means there will be some limitations. Once live data is thrown away, you will always lose some flexibility. +Learn about the search constraints when querying rolled-up data. While the rollup function is flexible, the nature of summarizing data means there are some limitations. Once live data is discarded, some loss of flexibility is inevitable. This page highlights the major limitations so that you are aware of them. diff --git a/manage-data/lifecycle/rollup/understanding-groups.md b/manage-data/lifecycle/rollup/understanding-groups.md index 0a6cfd0829..a69e0ce3e2 100644 --- a/manage-data/lifecycle/rollup/understanding-groups.md +++ b/manage-data/lifecycle/rollup/understanding-groups.md @@ -1,4 +1,5 @@ --- +navigation_title: Rollup groups and configuration mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/rollup-understanding-groups.html applies_to: @@ -8,7 +9,7 @@ products: - id: elasticsearch --- -# Understanding groups [rollup-understanding-groups] +# {{es}} rollup groups and configuration [rollup-understanding-groups] ::::{admonition} Deprecated in 8.11.0. :class: warning @@ -16,6 +17,7 @@ products: Rollups will be removed in a future version. [Migrate](migrating-from-rollup-to-downsampling.md) to [downsampling](../../data-store/data-streams/downsampling-time-series-data-stream.md) instead. :::: +Learn how to configure {{es}} rollup jobs with groupings for rollup queries and aggregations, as well as best practices for grouping time series data. To preserve flexibility, Rollup Jobs are defined based on how future queries may need to use the data. Traditionally, systems force the admin to make decisions about what metrics to rollup and on what interval. For example, the average of `cpu_time` on an hourly basis. This is limiting; if, in the future, the admin wishes to see the average of `cpu_time` on an hourly basis *and* partitioned by `host_name`, they are out of luck.