From 72659c61f6e84107830743b030ef04a7a8a19a49 Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Tue, 9 Sep 2025 16:49:46 -0400 Subject: [PATCH 1/5] Fix up data stream lifecycle tutorials --- manage-data/lifecycle/data-stream.md | 9 ++++---- ...orial-create-data-stream-with-lifecycle.md | 22 +++++++++++-------- .../tutorial-data-stream-retention.md | 2 +- ...ed-data-stream-to-data-stream-lifecycle.md | 4 +--- .../tutorial-update-existing-data-stream.md | 11 +++++----- 5 files changed, 26 insertions(+), 22 deletions(-) diff --git a/manage-data/lifecycle/data-stream.md b/manage-data/lifecycle/data-stream.md index 4d239e40b5..63b1010539 100644 --- a/manage-data/lifecycle/data-stream.md +++ b/manage-data/lifecycle/data-stream.md @@ -48,11 +48,12 @@ Steps `2-4` apply only to backing indices that are not already managed by {{ilm- Since the lifecycle is configured on the data stream level, the process to configure a lifecycle on a new data stream and on an existing one differ. -In the following sections, we will go through the following tutorials: +Four tutorials are available to help you set up and manage data streams with data stream lifecycle: -* To create a new data stream with a lifecycle, you need to add the data stream lifecycle as part of the index template that matches the name of your data stream (see [Tutorial: Create a data stream with a lifecycle](data-stream/tutorial-create-data-stream-with-lifecycle.md)). When a write operation with the name of your data stream reaches {{es}} then the data stream will be created with the respective data stream lifecycle. -* To update the lifecycle of an existing data stream you need to use the [data stream lifecycle APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-data-stream) to edit the lifecycle on the data stream itself (see [Tutorial: Update existing data stream](data-stream/tutorial-update-existing-data-stream.md)). -* Migrate an existing {{ilm-init}} managed data stream to Data stream lifecycle using [Tutorial: Migrate ILM managed data stream to data stream lifecycle](data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md). +* To create a new data stream with a lifecycle, add the data stream lifecycle as part of the index template that matches the name of your data stream. See [Tutorial: Create a data stream with a lifecycle](data-stream/tutorial-create-data-stream-with-lifecycle.md) to learn more. When a write operation with the name of your data stream reaches {{es}} then the data stream will be created with the respective data stream lifecycle. +* To update the lifecycle settings for an existing data stream, use the [data stream lifecycle APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-data-stream). See [Tutorial: Update existing data stream](data-stream/tutorial-update-existing-data-stream.md) for details. +* To fine-tune the retention settings for a data stream managed by data stream lifecycle, using both cluster and data stream level settings, refer to [Tutorial: Configure data stream retention](/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md). +* To migrate an existing {{ilm-init}} managed data stream to data stream lifecycle, follow the steps in [Tutorial: Migrate ILM managed data stream to data stream lifecycle](data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md). ::::{note} Updating the data stream lifecycle of an existing data stream is different from updating the settings or the mapping, because it is applied on the data stream level and not on the individual backing indices. diff --git a/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md b/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md index f9808f44a9..fb66ada7fe 100644 --- a/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md +++ b/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md @@ -30,7 +30,7 @@ You can use the [create index template API](https://www.elastic.co/docs/api/doc/ ```console PUT _index_template/my-index-template { - "index_patterns": ["my-data-stream*"], + "index_patterns": ["my-data-stream-test"], <1> "data_stream": { }, "priority": 500, "template": { @@ -43,7 +43,7 @@ PUT _index_template/my-index-template } } ``` - +<1> In this case the index template will be applied to a data stream named `my-data-stream-test`. You can optionally use a wildcard (`*`) in the index pattern to match all data streams created (either manually or using a an indexing request) that have a name matching the specified pattern. ## Create a data stream [create-data-stream-with-lifecycle] @@ -52,13 +52,13 @@ You can create a data stream in two ways: 1. By manually creating the stream using the [create data stream API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-create-data-stream). The stream’s name must still match one of your template’s index patterns. ```console - PUT _data_stream/my-data-stream + PUT _data_stream/my-data-stream-test ``` 2. By [indexing requests](../../data-store/data-streams/use-data-stream.md#add-documents-to-a-data-stream) that target the stream’s name. This name must match one of your index template’s index patterns. ```console - PUT my-data-stream/_bulk + PUT my-data-stream-test/_bulk { "create":{ } } { "@timestamp": "2099-05-06T16:21:15.000Z", "message": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736" } { "create":{ } } @@ -72,7 +72,7 @@ You can create a data stream in two ways: You can use the [get data stream lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get-data-lifecycle) to see the data stream lifecycle of your data stream and the [explain data stream lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-explain-data-lifecycle) to see the exact state of each backing index. ```console -GET _data_stream/my-data-stream/_lifecycle +GET _data_stream/my-data-stream-test/_lifecycle ``` The result will look like this: @@ -81,7 +81,7 @@ The result will look like this: { "data_streams": [ { - "name": "my-data-stream", <1> + "name": "my-data-stream-test", <1> "lifecycle": { "enabled": true, <2> "data_retention": "7d", <3> @@ -103,16 +103,20 @@ The result will look like this: If you want to see more information about how the data stream lifecycle is applied on individual backing indices use the [explain data stream lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-explain-data-lifecycle): ```console -GET .ds-my-data-stream-*/_lifecycle/explain +GET .ds-my-data-stream-test/_lifecycle/explain ``` +:::{tip} +You can use a wildcard (`*`) in the data stream name to retrieve the lifecycle status for all data streams matching a pattern. +::: + The result will look like this: ```console-result { "indices": { - ".ds-my-data-stream-2023.04.19-000001": { - "index": ".ds-my-data-stream-2023.04.19-000001", <1> + ".ds-my-data-stream-test-2023.04.19-000001": { + "index": ".ds-my-data-stream-test-2023.04.19-000001", <1> "managed_by_lifecycle": true, <2> "index_creation_date_millis": 1681918009501, "time_since_index_creation": "1.6m", <3> diff --git a/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md b/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md index a7c8df6e41..db9a23816d 100644 --- a/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md +++ b/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md @@ -8,7 +8,7 @@ products: - id: elasticsearch --- -# Tutorial: Data stream retention [tutorial-manage-data-stream-retention] +# Tutorial: Configure data stream retention [tutorial-manage-data-stream-retention] In this tutorial, we are going to go over the data stream lifecycle retention; we will define it, go over how it can be configured and how it can gets applied. Keep in mind, the following options apply only to data streams that are managed by the data stream lifecycle. diff --git a/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md b/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md index 682446c623..55fee8e505 100644 --- a/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md +++ b/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md @@ -312,8 +312,6 @@ GET _data_stream/dsl-data-stream 3. The new write index received the `false` value for the `prefer_ilm` setting, as we configured in the index template 4. The new write index is managed by `Data stream lifecycle` - - ## Migrate data stream back to ILM [migrate-from-dsl-to-ilm] We can easily change this data stream to be managed by {{ilm-init}} because we didn’t remove the {{ilm-init}} policy when we [updated the index template](#update-index-template-for-dsl). @@ -333,7 +331,7 @@ PUT _data_stream/dsl-data-stream/_lifecycle } ``` -1. The `enabled` flag can be ommitted and defaults to `true` however, here we explicitly configure it to `false` Let’s check the state of the data stream: +1. The `enabled` flag can be omitted and defaults to `true` however, here we explicitly configure it to `false` Let’s check the state of the data stream: ```console diff --git a/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md b/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md index d3ff700aaa..be6b6de13f 100644 --- a/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md +++ b/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md @@ -8,13 +8,14 @@ products: - id: elasticsearch --- -# Tutorial: Update existing data stream [tutorial-manage-existing-data-stream] +# Tutorial: Update the lifecycle of a data stream [tutorial-manage-existing-data-stream] -To update the lifecycle of an existing data stream, perform the following actions: +Follow these steps to configure or to remove data stream lifecycle settings for an existing data stream: -1. [Set a data stream’s lifecycle](#set-lifecycle) -2. [Remove lifecycle for a data stream](#delete-lifecycle) +- [Set a data stream’s lifecycle](#set-lifecycle) +- [Remove the lifecycle for a data stream](#delete-lifecycle) +Note that these steps are for data stream lifecycle only. For the steps to configure {{ilm}}, refer to the [{{ilm-init}} documentation](/manage-data/lifecycle/index-lifecycle-management.md). For a comparison between the two, refer to [](/manage-data/lifecycle.md). ## Set a data stream’s lifecycle [set-lifecycle] @@ -143,7 +144,7 @@ The response will look like: ::: ::::: -## Remove lifecycle for a data stream [delete-lifecycle] +## Remove the lifecycle for a data stream [delete-lifecycle] To remove the lifecycle of a data stream you can use the **Index Management** tools in {{kib}} or the {{es}} [delete lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-delete-data-lifecycle). From 8285904cc2a5d5702e886e6d03f535d9df07aaf6 Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Wed, 10 Sep 2025 12:07:38 -0400 Subject: [PATCH 2/5] fixup --- manage-data/lifecycle/data-stream.md | 6 +++--- .../tutorial-create-data-stream-with-lifecycle.md | 2 +- .../data-stream/tutorial-update-existing-data-stream.md | 5 +++-- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/manage-data/lifecycle/data-stream.md b/manage-data/lifecycle/data-stream.md index 63b1010539..ca352b1f3e 100644 --- a/manage-data/lifecycle/data-stream.md +++ b/manage-data/lifecycle/data-stream.md @@ -50,9 +50,9 @@ Since the lifecycle is configured on the data stream level, the process to confi Four tutorials are available to help you set up and manage data streams with data stream lifecycle: -* To create a new data stream with a lifecycle, add the data stream lifecycle as part of the index template that matches the name of your data stream. See [Tutorial: Create a data stream with a lifecycle](data-stream/tutorial-create-data-stream-with-lifecycle.md) to learn more. When a write operation with the name of your data stream reaches {{es}} then the data stream will be created with the respective data stream lifecycle. -* To update the lifecycle settings for an existing data stream, use the [data stream lifecycle APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-data-stream). See [Tutorial: Update existing data stream](data-stream/tutorial-update-existing-data-stream.md) for details. -* To fine-tune the retention settings for a data stream managed by data stream lifecycle, using both cluster and data stream level settings, refer to [Tutorial: Configure data stream retention](/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md). +* To create a new data stream with a lifecycle, add the data stream lifecycle as part of the index template that matches the name of your data stream. See [Tutorial: Create a data stream with a lifecycle](data-stream/tutorial-create-data-stream-with-lifecycle.md) for the detailed steps. When a write operation with the name of your data stream reaches {{es}} then the data stream will be created with the respective data stream lifecycle. +* To update the lifecycle settings for an individual, existing data stream, use the [data stream lifecycle APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-data-stream). See [Tutorial: Update existing data stream](data-stream/tutorial-update-existing-data-stream.md) for details. +* Retention settings for data streams can be configured both individually, at the data stream level, and globally, for all data streams in a cluster. To learn more, refer to [Tutorial: Configure data stream retention](/manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md). * To migrate an existing {{ilm-init}} managed data stream to data stream lifecycle, follow the steps in [Tutorial: Migrate ILM managed data stream to data stream lifecycle](data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md). ::::{note} diff --git a/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md b/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md index fb66ada7fe..268b7d93c7 100644 --- a/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md +++ b/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md @@ -43,7 +43,7 @@ PUT _index_template/my-index-template } } ``` -<1> In this case the index template will be applied to a data stream named `my-data-stream-test`. You can optionally use a wildcard (`*`) in the index pattern to match all data streams created (either manually or using a an indexing request) that have a name matching the specified pattern. +1. In this case the index template will be applied to a data stream named `my-data-stream-test`. You can optionally use a wildcard (`*`) in the index pattern to match all data streams created (either manually or using a an indexing request) that have a name matching the specified pattern. ## Create a data stream [create-data-stream-with-lifecycle] diff --git a/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md b/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md index be6b6de13f..038165d1e6 100644 --- a/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md +++ b/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md @@ -1,4 +1,5 @@ --- +navigation_title: Tutorial: Update the lifecycle of a data stream mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/tutorial-manage-existing-data-stream.html applies_to: @@ -8,9 +9,9 @@ products: - id: elasticsearch --- -# Tutorial: Update the lifecycle of a data stream [tutorial-manage-existing-data-stream] +# Tutorial: Update the lifecycle of an individual data stream [tutorial-manage-existing-data-stream] -Follow these steps to configure or to remove data stream lifecycle settings for an existing data stream: +Follow these steps to configure or remove data stream lifecycle settings for an existing, individual data stream: - [Set a data stream’s lifecycle](#set-lifecycle) - [Remove the lifecycle for a data stream](#delete-lifecycle) From 8dfd534aec70a76865894be14ece69d6d18d4b94 Mon Sep 17 00:00:00 2001 From: David Kilfoyle Date: Thu, 11 Sep 2025 10:49:23 -0400 Subject: [PATCH 3/5] Remove nav title --- .../data-stream/tutorial-update-existing-data-stream.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md b/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md index 038165d1e6..89e0bcabeb 100644 --- a/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md +++ b/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md @@ -1,5 +1,4 @@ --- -navigation_title: Tutorial: Update the lifecycle of a data stream mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/tutorial-manage-existing-data-stream.html applies_to: @@ -9,7 +8,7 @@ products: - id: elasticsearch --- -# Tutorial: Update the lifecycle of an individual data stream [tutorial-manage-existing-data-stream] +# Tutorial: Update the lifecycle of a data stream [tutorial-manage-existing-data-stream] Follow these steps to configure or remove data stream lifecycle settings for an existing, individual data stream: From e59747e88a374324aab3c0dd8387341e1f296d3b Mon Sep 17 00:00:00 2001 From: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> Date: Fri, 12 Sep 2025 12:38:49 -0400 Subject: [PATCH 4/5] Update manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md Co-authored-by: Vlada Chirmicci --- .../data-stream/tutorial-create-data-stream-with-lifecycle.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md b/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md index 268b7d93c7..81dc85b775 100644 --- a/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md +++ b/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md @@ -107,7 +107,7 @@ GET .ds-my-data-stream-test/_lifecycle/explain ``` :::{tip} -You can use a wildcard (`*`) in the data stream name to retrieve the lifecycle status for all data streams matching a pattern. +You can use a wildcard (`*`) in the data stream name to retrieve the lifecycle status for all data streams matching the pattern. ::: The result will look like this: From 27028435f5aac0295162848ffcfea27e43d5de71 Mon Sep 17 00:00:00 2001 From: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> Date: Fri, 12 Sep 2025 12:39:33 -0400 Subject: [PATCH 5/5] Update manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md Co-authored-by: Vlada Chirmicci --- .../data-stream/tutorial-create-data-stream-with-lifecycle.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md b/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md index 81dc85b775..6677096b5a 100644 --- a/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md +++ b/manage-data/lifecycle/data-stream/tutorial-create-data-stream-with-lifecycle.md @@ -43,7 +43,7 @@ PUT _index_template/my-index-template } } ``` -1. In this case the index template will be applied to a data stream named `my-data-stream-test`. You can optionally use a wildcard (`*`) in the index pattern to match all data streams created (either manually or using a an indexing request) that have a name matching the specified pattern. +1. In this case the index template will be applied to a data stream named `my-data-stream-test`. You can optionally use a wildcard (`*`) in the index pattern to match all data streams created (either manually or using an indexing request) that have a name matching the specified pattern. ## Create a data stream [create-data-stream-with-lifecycle]