From d34c01976c36268c835df19c7c43f32657be3bae Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Wed, 12 Feb 2025 14:39:25 -0600 Subject: [PATCH 1/7] add data tiers content --- manage-data/lifecycle/data-tiers.md | 448 +++++++++++++++++++++++++++- 1 file changed, 436 insertions(+), 12 deletions(-) diff --git a/manage-data/lifecycle/data-tiers.md b/manage-data/lifecycle/data-tiers.md index 1cc0a4bfc8..52fa0130d6 100644 --- a/manage-data/lifecycle/data-tiers.md +++ b/manage-data/lifecycle/data-tiers.md @@ -4,29 +4,453 @@ mapped_urls: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-disable-data-tier.html --- -# Data tiers - % What needs to be done: Refine % GitHub issue: docs-projects#375 % Use migrated content from existing pages that map to this page: -% - [ ] ./raw-migrated-files/elasticsearch/elasticsearch-reference/data-tiers.md -% - [ ] ./raw-migrated-files/cloud/cloud-enterprise/ece-disable-data-tier.md +% - [x] ./raw-migrated-files/elasticsearch/elasticsearch-reference/data-tiers.md +% - [x] ./raw-migrated-files/cloud/cloud-enterprise/ece-disable-data-tier.md + +# Data tiers + +A *data tier* is a collection of [nodes](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html) within a cluster that share the same [data node role](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#node-roles), and a hardware profile that’s appropriately sized for the role. Elastic recommends that nodes in the same tier share the same hardware profile to avoid [hot spotting](/troubleshoot/elasticsearch/hotspotting.md). + +## Available data tiers [available-tier] + +The data tiers that you use, and the way that you use them, depends on the data’s [category](/manage-data/lifecycle.md). The following data tiers are can be used with each data category: + +**Content data**: + +* [Content tier](/manage-data/lifecycle/data-tiers.md#content-tier) nodes handle the indexing and query load for non-timeseries indices, such as a product catalog. + +**Time series data**: + +* [Hot tier](/manage-data/lifecycle/data-tiers.md#hot-tier) nodes handle the indexing load for time series data, such as logs or metrics. They hold your most recent, most-frequently-accessed data. +* [Warm tier](/manage-data/lifecycle/data-tiers.md#warm-tier) nodes hold time series data that is accessed less-frequently and rarely needs to be updated. +* [Cold tier](/manage-data/lifecycle/data-tiers.md#cold-tier) nodes hold time series data that is accessed infrequently and not normally updated. To save space, you can keep [fully mounted indices](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#fully-mounted) of [{{search-snaps}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-searchable-snapshot.html) on the cold tier. These fully mounted indices eliminate the need for replicas, reducing required disk space by approximately 50% compared to the regular indices. +* [Frozen tier](/manage-data/lifecycle/data-tiers.md#frozen-tier) nodes hold time series data that is accessed rarely and never updated. The frozen tier stores [partially mounted indices](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#partially-mounted) of [{{search-snaps}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-searchable-snapshot.html) exclusively. This extends the storage capacity even further — by up to 20 times compared to the warm tier. + +::::{tip} +The performance of an {{es}} node is often limited by the performance of the underlying storage and hardware profile. For example hardware profiles, refer to Elastic Cloud’s [instance configurations](https://www.elastic.co/guide/en/cloud/current/ec-reference-hardware.html). Review our recommendations for optimizing your storage for [indexing](/deploy-manage/production-guidance/optimize-performance/indexing-speed.md#indexing-use-faster-hardware) and [search](/deploy-manage/production-guidance/optimize-performance/search-speed.md#search-use-faster-hardware). +:::: + +::::{important} +{{es}} assumes nodes within a data tier share the same hardware profile (such as CPU, RAM, disk capacity). Data tiers with unequally resourced nodes have a higher risk of [hot spotting](/troubleshoot/elasticsearch/hotspotting.md). +:::: + +The way data tiers are used often depends on the data’s category: + +* Content data remains on the [content tier](/manage-data/lifecycle/data-tiers.md#content-tier) for its entire data lifecycle. +* Time series data may progress through the descending temperature data tiers (hot, warm, cold, and frozen) according to your performance, resiliency, and data retention requirements. + + You can automate these lifecycle transitions using the [data stream lifecycle](/manage-data/data-store/index-types/data-streams.md), or custom [{{ilm}}](/manage-data/lifecycle/index-lifecycle-management.md). + +Learn more about each data tier, including when and how it should be used. + +### Content tier [content-tier] + +Data stored in the content tier is generally a collection of items such as a product catalog or article archive. Unlike time series data, the value of the content remains relatively constant over time, so it doesn’t make sense to move it to a tier with different performance characteristics as it ages. Content data typically has long data retention requirements, and you want to be able to retrieve items quickly regardless of how old they are. + +Content tier nodes are usually optimized for query performance—​they prioritize processing power over IO throughput so they can process complex searches and aggregations and return results quickly. While they are also responsible for indexing, content data is generally not ingested at as high a rate as time series data such as logs and metrics. From a resiliency perspective the indices in this tier should be configured to use one or more replicas. + +The content tier is required and is often deployed within the same node grouping as the hot tier. System indices and other indices that aren’t part of a data stream are automatically allocated to the content tier. + + +### Hot tier [hot-tier] + +The hot tier is the {{es}} entry point for time series data and holds your most-recent, most-frequently-searched time series data. Nodes in the hot tier need to be fast for both reads and writes, which requires more hardware resources and faster storage (SSDs). For resiliency, indices in the hot tier should be configured to use one or more replicas. + +The hot tier is required. New indices that are part of a [data stream](/manage-data/data-store/index-types/data-streams.md) are automatically allocated to the hot tier. + + +### Warm tier [warm-tier] + +Time series data can move to the warm tier once it is being queried less frequently than the recently-indexed data in the hot tier. The warm tier typically holds data from recent weeks. Updates are still allowed, but likely infrequent. Nodes in the warm tier generally don’t need to be as fast as those in the hot tier. For resiliency, indices in the warm tier should be configured to use one or more replicas. + + +### Cold tier [cold-tier] + +When you no longer need to search time series data regularly, it can move from the warm tier to the cold tier. While still searchable, this tier is typically optimized for lower storage costs rather than search speed. + +For better storage savings, you can keep [fully mounted indices](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#fully-mounted) of [{{search-snaps}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-searchable-snapshot.html) on the cold tier. Unlike regular indices, these fully mounted indices don’t require replicas for reliability. In the event of a failure, they can recover data from the underlying snapshot instead. This potentially halves the local storage needed for the data. A snapshot repository is required to use fully mounted indices in the cold tier. Fully mounted indices are read-only. + +Alternatively, you can use the cold tier to store regular indices with replicas instead of using {{search-snaps}}. This lets you store older data on less expensive hardware but doesn’t reduce required disk space compared to the warm tier. + + +### Frozen tier [frozen-tier] + +Once data is no longer being queried, or being queried rarely, it may move from the cold tier to the frozen tier where it stays for the rest of its life. + +The frozen tier requires a snapshot repository. The frozen tier uses [partially mounted indices](/deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#partially-mounted) to store and load data from a snapshot repository. This reduces local storage and operating costs while still letting you search frozen data. Because {{es}} must sometimes fetch frozen data from the snapshot repository, searches on the frozen tier are typically slower than on the cold tier. + +## Configure data tiers [configure-data-tiers] + +Follow the instructions for your deployment type to configure data tiers. + +### {{ess}} or {{ece}} [configure-data-tiers-cloud] + +The default configuration for an {{ecloud}} deployment includes a shared tier for hot and content data. This tier is required and can’t be removed. + +#### Add a data tier + +To add a warm, cold, or frozen tier when you create a deployment: + +1. On the **Create deployment** page, click **Advanced Settings**. +2. Click **+ Add capacity** for any data tiers to add. +3. Click **Create deployment** at the bottom of the page to save your changes. + +:::{image} ../../images/elasticsearch-reference-ess-advanced-config-data-tiers.png +:alt: {{ecloud}}'s deployment Advanced configuration page +:class: screenshot +::: + +To add a data tier to an existing deployment: + +1. Log in to the [{{ecloud}} console](https://cloud.elastic.co?page=docs&placement=docs-body). +2. On the **Deployments** page, select your deployment. +3. In your deployment menu, select **Edit**. +4. Click **+ Add capacity** for any data tiers to add. +5. Click **Save** at the bottom of the page to save your changes. + +#### Disable a data tier [ece-update-data-tier-allocation-rules] + +The process of disabling a data tier depends on whether we are dealing with [searchable snapshots](#ece-disable-searchable-snapshot-data-tier) or [regular indices](#ece-disable-non-searchable-snapshot-data-tier). + +The hot and warm tiers store regular indices, while the frozen tier stores searchable snapshots. However, the cold tier can store either regular indices or searchable snapshots. To check if a cold tier contains searchable snapshots perform the following request: + +```sh +GET /_cat/indices/restored-* +``` + +##### Non-searchable snapshot data tier [ece-disable-non-searchable-snapshot-data-tier] + +Elastic Cloud Enterprise tries to move all data from the nodes that are removed during plan changes. To disable a non-searchable snapshot data tier (e.g., hot, warm, or cold tier), make sure that all data on that tier can be re-allocated by reconfiguring the relevant shard allocation filters. You’ll also need to temporarily stop your index lifecycle management (ILM) policies to prevent new indices from being moved to the data tier you want to disable. + +To learn more about ILM for Elastic Cloud Enterprise, or shard allocation filtering, check the following documentation: + +* [Configure index management](https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-index-management.html) +* [Create your index lifecyle policy](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html) +* [Managing the index lifecycle](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html) +* [Shard allocation filters](https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-allocation-filtering.html) + +To make sure that all data can be migrated from the data tier you want to disable, follow these steps: + +1. Determine which nodes will be removed from the cluster. + + 1. [Log into the Cloud UI](/deploy-manage/deploy/cloud-enterprise/log-into-cloud-ui.md). + 2. From the **Deployments** page, select your deployment. + + Narrow the list by name, ID, or choose from several other filters. To further define the list, use a combination of filters. + + 3. Filter the list of instances by the Data tier you want to disable. + + :::{image} ../../images/cloud-enterprise-ec-ce-remove-tier-filter-instances.png + :alt: A screenshot showing a filtered instance list + ::: + + Note the listed instance IDs. In this example, it would be Instance 2 and Instance 3. + +2. Stop ILM. + + ```sh + POST /_ilm/stop + ``` + +3. Determine which shards need to be moved. + + ```sh + GET /_cat/shards + ``` + + Parse the output, looking for shards allocated to the nodes to be removed from the cluster. Note that `Instance #2` is shown as `instance-0000000002` in the output. + + :::{image} ../../images/cloud-enterprise-ec-ce-remove-tier-filtered-cat-shards.png + :alt: A screenshot showing a filtered shard list + ::: + +4. Move shards off the nodes to be removed from the cluster. + + You must remove any [index-level shard allocation filters](https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-allocation-filtering.html) from the indices on the nodes to be removed. ILM uses different rules depending on the policy and version of Elasticsearch. Check the index settings to determine which rule to use: + + ```sh + GET /my-index/_settings + ``` + + 1. Updating data tier based allocation inclusion rules. + + Data tier based ILM policies use `index.routing.allocation.include` to allocate shards to the appropriate tier. The indices that use this method have index routing settings similar to the following example: + + ```sh + { + ... + "routing": { + "allocation": { + "include": { + "_tier_preference": "data_warm,data_hot" + } + } + } + ... + } + ``` + + You must remove the relevant tier from the inclusion rules. For example, to disable the warm tier, the `data_warm` tier preference should be removed: + + ```sh + PUT /my-index/_settings + { + "routing": { + "allocation": { + "include": { + "_tier_preference": "data_hot" + } + } + } + } + ``` + + Updating allocation inclusion rules will trigger a shard re-allocation, moving the shards from the nodes to be removed. + + 2. Updating node attribute allocation requirement rules. + + Node attribute based ILM policies uses `index.routing.allocation.require` to allocate shards to the appropriate nodes. The indices that use this method have index routing settings that are similar to the following example: + + ```sh + { + ... + "routing": { + "allocation": { + "require": { + "data": "warm" + } + } + } + ... + } + ``` + + You must either remove or redefine the routing requirements. To remove the attribute requirements, use the following code: + + ```sh + PUT /my-index/_settings + { + "routing": { + "allocation": { + "require": { + "data": null + } + } + } + } + ``` + + Removing required attributes does not trigger a shard reallocation. These shards are moved when applying the plan to disable the data tier. Alternatively, you can use the [cluster re-route API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html) to manually re-allocate the shards before removing the nodes, or explicitly re-allocate shards to hot nodes by using the following code: + + ```sh + PUT /my-index/_settings + { + "routing": { + "allocation": { + "require": { + "data": "hot" + } + } + } + } + ``` + + 3. Removing custom allocation rules. + + If indices on nodes to be removed have shard allocation rules of other forms, they must be removed as shown in the following example: + + ```sh + PUT /my-index/_settings + { + "routing": { + "allocation": { + "require": null, + "include": null, + "exclude": null + } + } + } + ``` + +5. Edit the deployment, disabling the data tier. + + If autoscaling is enabled, set the maximum size to 0 for the data tier to ensure autoscaling does not re-enable the data tier. + + Any remaining shards on the tier being disabled are re-allocated across the remaining cluster nodes while applying the plan to disable the data tier. Monitor shard allocation during the data migration phase to ensure all allocation rules have been correctly updated. If the plan fails to migrate data away from the data tier, then re-examine the allocation rules for the indices remaining on that data tier. + +6. Once the plan change completes, confirm that there are no remaining nodes associated with the disabled tier and that `GET _cluster/health` reports `green`. If this is the case, re-enable ILM. + + ```sh + POST _ilm/start + ``` + +##### Searchable snapshot data tier [ece-disable-searchable-snapshot-data-tier] + +When data reaches the `cold` or `frozen` phases, it is automatically converted to a [searchable snapshot](https://www.elastic.co/guide/en/elasticsearch/reference/current/searchable-snapshots.html) by ILM. If you do not intend to delete this data, you should manually restore each of the searchable snapshot indices to a regular index before disabling the data tier, by following these steps: + +1. Stop ILM and check ILM status is `STOPPED` to prevent data from migrating to the phase you intend to disable while you are working through the next steps. + + ```sh + # stop ILM + POST _ilm/stop + + # check status + GET _ilm/status + ``` + +2. Capture a comprehensive list of index and searchable snapshot names. + + 1. The index name of the searchable snapshots may differ based on the data tier. If you intend to disable the cold tier, then perform the following request with the `restored-*` prefix. If the frozen tier is the one to be disabled, use the `partial-*` prefix. + + ```sh + GET /_settings?filter_path=**.index.store.snapshot.snapshot_name&expand_wildcards=all + ``` + + In the example we have a list of 4 indices, which need to be moved away from the frozen tier. + + :::{image} ../../images/cloud-enterprise-ec-ce-remove-tier-filter-snapshot-indices.png + :alt: A screenshot showing a snapshot indices list + ::: + +3. (Optional) Save the list of index and snapshot names in a text file, so you can access it throughout the rest of the process. +4. Remove the aliases that were applied to searchable snapshots indices. Use the index prefix from step 2. + + ```sh + POST _aliases + { + "actions": [ + { + "remove": { + "index": "-", + "alias": "" + } + } + ] + } + ``` + + ::::{note} + If you use data stream, you can skip this step. + :::: + + + In the example we are removing the alias for the `frozen-index-1` index. + + :::{image} ../../images/cloud-enterprise-ec-ce-remove-tier-remove-alias.png + :alt: A screenshot showing the process of removing a searchable snapshot indice alias + ::: + +5. Restore indices from the searchable snapshots. + + 1. Follow the steps to [specify the data tier based allocation inclusion rules](/manage-data/lifecycle/data-tiers.md#ece-update-data-tier-allocation-rules). + 2. Remove the associated ILM policy (set it to `null`). If you want to apply a different ILM policy, follow the steps to [Switch lifecycle policies](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html#switch-lifecycle-policies). + 3. If needed, specify the alias for rollover, otherwise set it to `null`. + 4. Optionally, specify the desired number of replica shards. + + ```sh + POST _snapshot/found-snapshots//_restore + { + "indices": "*", + "index_settings": { + "index.routing.allocation.include._tier_preference": "", + "number_of_replicas": X, + "index.lifecycle.name": "", + "index.lifecycle.rollover_alias": "" + } + } + ``` + + The `` refers to the above-mentioned step: "Capture a comprehensive list of index and searchable snapshot names". + + In the example we are restoring `frozen-index-1` from the snapshot in `found-snapshots` (default snapshot repository) and placing it in the warm tier. + + :::{image} ../../images/cloud-enterprise-ec-ce-remove-tier-restore-snapshot.png + :alt: A screenshot showing the process of restoring a searchable snapshot to a regular index + ::: + +6. Repeat steps 4 and 5 until all snapshots are restored to regular indices. +7. Once all snapshots are restored, use `GET _cat/indices/?v=true` to check that the restored indices are `green` and are correctly reflecting the expected `doc` and `store.size` counts. + + If you are using data stream, you may need to use `GET _data_stream/` to get the list of the backing indices, and then specify them by using `GET _cat/indices/?v=true` to check. + +8. Once your data has completed restoration from searchable snapshots to the target data tier, `DELETE` searchable snapshot indices using the prefix from step 2. + + ```sh + DELETE - + ``` + +9. Delete the searchable snapshots by following these steps: + + 1. Open Kibana and navigate to Management > Data > Snapshot and Restore > Snapshots (or go to `/app/management/data/snapshot_restore/snapshots`) + 2. Search for `**` + 3. Bulk select the snapshots and delete them + + In the example we are deleting the snapshots associated with the `policy_with_frozen_phase`. + + :::{image} ../../images/cloud-enterprise-ec-ce-remove-tier-remove-snapshots.png + :alt: A screenshot showing the process of deleting snapshots + ::: + +10. Confirm that no shards remain on the data nodes you wish to remove using `GET _cat/allocation?v=true&s=node`. +11. Edit your cluster from the console to disable the data tier. +12. Once the plan change completes, confirm that there are no remaining nodes associated with the disabled tier and that `GET _cluster/health` reports `green`. If this is the case, re-enable ILM. + + ```sh + POST _ilm/start + ``` + +### Self-managed deployments [configure-data-tiers-on-premise] + +For self-managed deployments, each node’s [data role](/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md#data-node-role) is configured in `elasticsearch.yml`. For example, the highest-performance nodes in a cluster might be assigned to both the hot and content tiers: + +```yaml +node.roles: ["data_hot", "data_content"] +``` + +::::{note} +We recommend you use [dedicated nodes](/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md#data-frozen-node) in the frozen tier. +:::: + +## Data tier index allocation [data-tier-allocation] + +The [`index.routing.allocation.include._tier_preference`](https://www.elastic.co/guide/en/elasticsearch/reference/current/data-tier-shard-filtering.html#tier-preference-allocation-filter) setting determines which tier the index should be allocated to. + +When you create an index, by default {{es}} sets the `_tier_preference` to `data_content` to automatically allocate the index shards to the content tier. + +When {{es}} creates an index as part of a [data stream](/manage-data/data-store/index-types/data-streams.md), by default {{es}} sets the `_tier_preference` to `data_hot` to automatically allocate the index shards to the hot tier. + +At the time of index creation, you can override the default setting by explicitly setting the preferred value in one of two ways: + +* Using an [index template](/manage-data/data-store/templates.md). Refer to [Automate rollover with ILM](/manage-data/lifecycle/index-lifecycle-management.md) for details. +* Within the [create index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html) request body. + +You can override this setting after index creation by [updating the index setting](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-update-settings.html) to the preferred value. + +This setting also accepts multiple tiers in order of preference. This prevents indices from remaining unallocated if no nodes are available in the preferred tier. For example, when {{ilm}} migrates an index to the cold phase, it sets the index `_tier_preference` to `data_cold,data_warm,data_hot`. + +To remove the data tier preference setting, set the `_tier_preference` value to `null`. This allows the index to allocate to any data node within the cluster. Setting the `_tier_preference` to `null` does not restore the default value. Note that, in the case of managed indices, a [migrate](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-migrate.html) action might apply a new value in its place. + +### Determine the current data tier preference [data-tier-allocation-value] -% Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): +You can check an existing index’s data tier preference by [polling its settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-settings.html) for `index.routing.allocation.include._tier_preference`: -$$$cold-tier$$$ +```console +GET /my-index-000001/_settings?filter_path=*.settings.index.routing.allocation.include._tier_preference +``` -$$$data-tier-allocation$$$ +### Troubleshooting [data-tier-allocation-troubleshooting] -$$$frozen-tier$$$ +The `_tier_preference` setting might conflict with other allocation settings. This conflict might prevent the shard from allocating. A conflict might occur when a cluster has not yet been completely [migrated to data tiers](/troubleshoot/elasticsearch/troubleshoot-migrate-to-tiers.md). -$$$ece-update-data-tier-allocation-rules$$$ +This setting will not unallocate a currently allocated shard, but might prevent it from migrating from its current location to its designated data tier. To troubleshoot, call the [cluster allocation explain API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html) and specify the suspected problematic shard. -$$$content-tier$$$ -$$$hot-tier$$$ +### Automatic data tier migration [data-tier-migration] -$$$warm-tier$$$ \ No newline at end of file +{{ilm-init}} automatically transitions managed indices through the available data tiers using the [migrate](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-migrate.html) action. By default, this action is automatically injected in every phase. You can explicitly specify the migrate action with `"enabled": false` to [disable automatic migration](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-migrate.html#ilm-disable-migrate-ex), for example, if you’re using the [allocate action](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-allocate.html) to manually specify allocation rules. From 3dc1ccea22c12c150e895fb934345e74439b720d Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Wed, 12 Feb 2025 14:41:17 -0600 Subject: [PATCH 2/7] remove raw-migrated-files that are no longer needed --- .../cloud-enterprise/ece-disable-data-tier.md | 295 ------------------ .../elasticsearch-reference/data-tiers.md | 164 ---------- raw-migrated-files/toc.yml | 4 +- 3 files changed, 1 insertion(+), 462 deletions(-) delete mode 100644 raw-migrated-files/cloud/cloud-enterprise/ece-disable-data-tier.md delete mode 100644 raw-migrated-files/elasticsearch/elasticsearch-reference/data-tiers.md diff --git a/raw-migrated-files/cloud/cloud-enterprise/ece-disable-data-tier.md b/raw-migrated-files/cloud/cloud-enterprise/ece-disable-data-tier.md deleted file mode 100644 index 907d43063c..0000000000 --- a/raw-migrated-files/cloud/cloud-enterprise/ece-disable-data-tier.md +++ /dev/null @@ -1,295 +0,0 @@ -# Disable an Elasticsearch data tier [ece-disable-data-tier] - -The process of disabling a data tier depends on whether we are dealing with searchable snapshots or regular indices. - -The hot and warm tiers store regular indices, while the frozen tier stores searchable snapshots. However, the cold tier can store either regular indices or searchable snapshots. To check if a cold tier contains searchable snapshots perform the following request: - -```sh -GET /_cat/indices/restored-* -``` - - -## Non-searchable snapshot data tier [ece-disable-non-searchable-snapshot-data-tier] - -Elastic Cloud Enterprise tries to move all data from the nodes that are removed during plan changes. To disable a non-searchable snapshot data tier (e.g., hot, warm, or cold tier), make sure that all data on that tier can be re-allocated by reconfiguring the relevant shard allocation filters. You’ll also need to temporarily stop your index lifecycle management (ILM) policies to prevent new indices from being moved to the data tier you want to disable. - -To learn more about ILM for Elastic Cloud Enterprise, or shard allocation filtering, check the following documentation: - -* [Configure index management](https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-index-management.html) -* [Create your index lifecyle policy](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html) -* [Managing the index lifecycle](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html) -* [Shard allocation filters](https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-allocation-filtering.html) - -To make sure that all data can be migrated from the data tier you want to disable, follow these steps: - -1. Determine which nodes will be removed from the cluster. - - 1. [Log into the Cloud UI](../../../deploy-manage/deploy/cloud-enterprise/log-into-cloud-ui.md). - 2. From the **Deployments** page, select your deployment. - - Narrow the list by name, ID, or choose from several other filters. To further define the list, use a combination of filters. - - 3. Filter the list of instances by the Data tier you want to disable. - - :::{image} ../../../images/cloud-enterprise-ec-ce-remove-tier-filter-instances.png - :alt: A screenshot showing a filtered instance list - ::: - - Note the listed instance IDs. In this example, it would be Instance 2 and Instance 3. - -2. Stop ILM. - - ```sh - POST /_ilm/stop - ``` - -3. Determine which shards need to be moved. - - ```sh - GET /_cat/shards - ``` - - Parse the output, looking for shards allocated to the nodes to be removed from the cluster. Note that `Instance #2` is shown as `instance-0000000002` in the output. - - :::{image} ../../../images/cloud-enterprise-ec-ce-remove-tier-filtered-cat-shards.png - :alt: A screenshot showing a filtered shard list - ::: - -4. Move shards off the nodes to be removed from the cluster. - - You must remove any [index-level shard allocation filters](https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-allocation-filtering.html) from the indices on the nodes to be removed. ILM uses different rules depending on the policy and version of Elasticsearch. Check the index settings to determine which rule to use: - - ```sh - GET /my-index/_settings - ``` - - 1. Updating data tier based allocation inclusion rules. - - Data tier based ILM policies use `index.routing.allocation.include` to allocate shards to the appropriate tier. The indices that use this method have index routing settings similar to the following example: - - ```sh - { - ... - "routing": { - "allocation": { - "include": { - "_tier_preference": "data_warm,data_hot" - } - } - } - ... - } - ``` - - You must remove the relevant tier from the inclusion rules. For example, to disable the warm tier, the `data_warm` tier preference should be removed: - - ```sh - PUT /my-index/_settings - { - "routing": { - "allocation": { - "include": { - "_tier_preference": "data_hot" - } - } - } - } - ``` - - Updating allocation inclusion rules will trigger a shard re-allocation, moving the shards from the nodes to be removed. - - 2. Updating node attribute allocation requirement rules. - - Node attribute based ILM policies uses `index.routing.allocation.require` to allocate shards to the appropriate nodes. The indices that use this method have index routing settings that are similar to the following example: - - ```sh - { - ... - "routing": { - "allocation": { - "require": { - "data": "warm" - } - } - } - ... - } - ``` - - You must either remove or redefine the routing requirements. To remove the attribute requirements, use the following code: - - ```sh - PUT /my-index/_settings - { - "routing": { - "allocation": { - "require": { - "data": null - } - } - } - } - ``` - - Removing required attributes does not trigger a shard reallocation. These shards are moved when applying the plan to disable the data tier. Alternatively, you can use the [cluster re-route API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html) to manually re-allocate the shards before removing the nodes, or explicitly re-allocate shards to hot nodes by using the following code: - - ```sh - PUT /my-index/_settings - { - "routing": { - "allocation": { - "require": { - "data": "hot" - } - } - } - } - ``` - - 3. Removing custom allocation rules. - - If indices on nodes to be removed have shard allocation rules of other forms, they must be removed as shown in the following example: - - ```sh - PUT /my-index/_settings - { - "routing": { - "allocation": { - "require": null, - "include": null, - "exclude": null - } - } - } - ``` - -5. Edit the deployment, disabling the data tier. - - If autoscaling is enabled, set the maximum size to 0 for the data tier to ensure autoscaling does not re-enable the data tier. - - Any remaining shards on the tier being disabled are re-allocated across the remaining cluster nodes while applying the plan to disable the data tier. Monitor shard allocation during the data migration phase to ensure all allocation rules have been correctly updated. If the plan fails to migrate data away from the data tier, then re-examine the allocation rules for the indices remaining on that data tier. - -6. Once the plan change completes, confirm that there are no remaining nodes associated with the disabled tier and that `GET _cluster/health` reports `green`. If this is the case, re-enable ILM. - - ```sh - POST _ilm/start - ``` - - - -## Searchable snapshot data tier [ece-disable-searchable-snapshot-data-tier] - -When data reaches the `cold` or `frozen` phases, it is automatically converted to a [searchable snapshot](https://www.elastic.co/guide/en/elasticsearch/reference/current/searchable-snapshots.html) by ILM. If you do not intend to delete this data, you should manually restore each of the searchable snapshot indices to a regular index before disabling the data tier, by following these steps: - -1. Stop ILM and check ILM status is `STOPPED` to prevent data from migrating to the phase you intend to disable while you are working through the next steps. - - ```sh - # stop ILM - POST _ilm/stop - - # check status - GET _ilm/status - ``` - -2. Capture a comprehensive list of index and searchable snapshot names. - - 1. The index name of the searchable snapshots may differ based on the data tier. If you intend to disable the cold tier, then perform the following request with the `restored-*` prefix. If the frozen tier is the one to be disabled, use the `partial-*` prefix. - - ```sh - GET /_settings?filter_path=**.index.store.snapshot.snapshot_name&expand_wildcards=all - ``` - - In the example we have a list of 4 indices, which need to be moved away from the frozen tier. - - :::{image} ../../../images/cloud-enterprise-ec-ce-remove-tier-filter-snapshot-indices.png - :alt: A screenshot showing a snapshot indices list - ::: - -3. (Optional) Save the list of index and snapshot names in a text file, so you can access it throughout the rest of the process. -4. Remove the aliases that were applied to searchable snapshots indices. Use the index prefix from step 2. - - ```sh - POST _aliases - { - "actions": [ - { - "remove": { - "index": "-", - "alias": "" - } - } - ] - } - ``` - - ::::{note} - If you use data stream, you can skip this step. - :::: - - - In the example we are removing the alias for the `frozen-index-1` index. - - :::{image} ../../../images/cloud-enterprise-ec-ce-remove-tier-remove-alias.png - :alt: A screenshot showing the process of removing a searchable snapshot indice alias - ::: - -5. Restore indices from the searchable snapshots. - - 1. Follow the steps to [specify the data tier based allocation inclusion rules](../../../manage-data/lifecycle/data-tiers.md#ece-update-data-tier-allocation-rules). - 2. Remove the associated ILM policy (set it to `null`). If you want to apply a different ILM policy, follow the steps to [Switch lifecycle policies](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html#switch-lifecycle-policies). - 3. If needed, specify the alias for rollover, otherwise set it to `null`. - 4. Optionally, specify the desired number of replica shards. - - ```sh - POST _snapshot/found-snapshots//_restore - { - "indices": "*", - "index_settings": { - "index.routing.allocation.include._tier_preference": "", - "number_of_replicas": X, - "index.lifecycle.name": "", - "index.lifecycle.rollover_alias": "" - } - } - ``` - - The `` refers to the above-mentioned step: "Capture a comprehensive list of index and searchable snapshot names". - - In the example we are restoring `frozen-index-1` from the snapshot in `found-snapshots` (default snapshot repository) and placing it in the warm tier. - - :::{image} ../../../images/cloud-enterprise-ec-ce-remove-tier-restore-snapshot.png - :alt: A screenshot showing the process of restoring a searchable snapshot to a regular index - ::: - -6. Repeat steps 4 and 5 until all snapshots are restored to regular indices. -7. Once all snapshots are restored, use `GET _cat/indices/?v=true` to check that the restored indices are `green` and are correctly reflecting the expected `doc` and `store.size` counts. - - If you are using data stream, you may need to use `GET _data_stream/` to get the list of the backing indices, and then specify them by using `GET _cat/indices/?v=true` to check. - -8. Once your data has completed restoration from searchable snapshots to the target data tier, `DELETE` searchable snapshot indices using the prefix from step 2. - - ```sh - DELETE - - ``` - -9. Delete the searchable snapshots by following these steps: - - 1. Open Kibana and navigate to Management > Data > Snapshot and Restore > Snapshots (or go to `/app/management/data/snapshot_restore/snapshots`) - 2. Search for `**` - 3. Bulk select the snapshots and delete them - - In the example we are deleting the snapshots associated with the `policy_with_frozen_phase`. - - :::{image} ../../../images/cloud-enterprise-ec-ce-remove-tier-remove-snapshots.png - :alt: A screenshot showing the process of deleting snapshots - ::: - -10. Confirm that no shards remain on the data nodes you wish to remove using `GET _cat/allocation?v=true&s=node`. -11. Edit your cluster from the console to disable the data tier. -12. Once the plan change completes, confirm that there are no remaining nodes associated with the disabled tier and that `GET _cluster/health` reports `green`. If this is the case, re-enable ILM. - - ```sh - POST _ilm/start - ``` - - diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/data-tiers.md b/raw-migrated-files/elasticsearch/elasticsearch-reference/data-tiers.md deleted file mode 100644 index e874e8deea..0000000000 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/data-tiers.md +++ /dev/null @@ -1,164 +0,0 @@ -# Data tiers [data-tiers] - -A *data tier* is a collection of [nodes](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html) within a cluster that share the same [data node role](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#node-roles), and a hardware profile that’s appropriately sized for the role. Elastic recommends that nodes in the same tier share the same hardware profile to avoid [hot spotting](../../../troubleshoot/elasticsearch/hotspotting.md). - -The data tiers that you use, and the way that you use them, depends on the data’s [category](../../../manage-data/lifecycle.md). - -The following data tiers are can be used with each data category: - -Content data: - -* [Content tier](../../../manage-data/lifecycle/data-tiers.md#content-tier) nodes handle the indexing and query load for non-timeseries indices, such as a product catalog. - -Time series data: - -* [Hot tier](../../../manage-data/lifecycle/data-tiers.md#hot-tier) nodes handle the indexing load for time series data, such as logs or metrics. They hold your most recent, most-frequently-accessed data. -* [Warm tier](../../../manage-data/lifecycle/data-tiers.md#warm-tier) nodes hold time series data that is accessed less-frequently and rarely needs to be updated. -* [Cold tier](../../../manage-data/lifecycle/data-tiers.md#cold-tier) nodes hold time series data that is accessed infrequently and not normally updated. To save space, you can keep [fully mounted indices](../../../deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#fully-mounted) of [{{search-snaps}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-searchable-snapshot.html) on the cold tier. These fully mounted indices eliminate the need for replicas, reducing required disk space by approximately 50% compared to the regular indices. -* [Frozen tier](../../../manage-data/lifecycle/data-tiers.md#frozen-tier) nodes hold time series data that is accessed rarely and never updated. The frozen tier stores [partially mounted indices](../../../deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#partially-mounted) of [{{search-snaps}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-searchable-snapshot.html) exclusively. This extends the storage capacity even further — by up to 20 times compared to the warm tier. - -::::{tip} -The performance of an {{es}} node is often limited by the performance of the underlying storage and hardware profile. For example hardware profiles, refer to Elastic Cloud’s [instance configurations](https://www.elastic.co/guide/en/cloud/current/ec-reference-hardware.html). Review our recommendations for optimizing your storage for [indexing](../../../deploy-manage/production-guidance/optimize-performance/indexing-speed.md#indexing-use-faster-hardware) and [search](../../../deploy-manage/production-guidance/optimize-performance/search-speed.md#search-use-faster-hardware). -:::: - - -::::{important} -{{es}} assumes nodes within a data tier share the same hardware profile (such as CPU, RAM, disk capacity). Data tiers with unequally resourced nodes have a higher risk of [hot spotting](../../../troubleshoot/elasticsearch/hotspotting.md). -:::: - - -The way data tiers are used often depends on the data’s category: - -* Content data remains on the [content tier](../../../manage-data/lifecycle/data-tiers.md#content-tier) for its entire data lifecycle. -* Time series data may progress through the descending temperature data tiers (hot, warm, cold, and frozen) according to your performance, resiliency, and data retention requirements. - - You can automate these lifecycle transitions using the [data stream lifecycle](../../../manage-data/data-store/index-types/data-streams.md), or custom [{{ilm}}](../../../manage-data/lifecycle/index-lifecycle-management.md). - - - -## Available data tiers [available-tier] - -Learn more about each data tier, including when and how it should be used. - - -### Content tier [content-tier] - -Data stored in the content tier is generally a collection of items such as a product catalog or article archive. Unlike time series data, the value of the content remains relatively constant over time, so it doesn’t make sense to move it to a tier with different performance characteristics as it ages. Content data typically has long data retention requirements, and you want to be able to retrieve items quickly regardless of how old they are. - -Content tier nodes are usually optimized for query performance—​they prioritize processing power over IO throughput so they can process complex searches and aggregations and return results quickly. While they are also responsible for indexing, content data is generally not ingested at as high a rate as time series data such as logs and metrics. From a resiliency perspective the indices in this tier should be configured to use one or more replicas. - -The content tier is required and is often deployed within the same node grouping as the hot tier. System indices and other indices that aren’t part of a data stream are automatically allocated to the content tier. - - -### Hot tier [hot-tier] - -The hot tier is the {{es}} entry point for time series data and holds your most-recent, most-frequently-searched time series data. Nodes in the hot tier need to be fast for both reads and writes, which requires more hardware resources and faster storage (SSDs). For resiliency, indices in the hot tier should be configured to use one or more replicas. - -The hot tier is required. New indices that are part of a [data stream](../../../manage-data/data-store/index-types/data-streams.md) are automatically allocated to the hot tier. - - -### Warm tier [warm-tier] - -Time series data can move to the warm tier once it is being queried less frequently than the recently-indexed data in the hot tier. The warm tier typically holds data from recent weeks. Updates are still allowed, but likely infrequent. Nodes in the warm tier generally don’t need to be as fast as those in the hot tier. For resiliency, indices in the warm tier should be configured to use one or more replicas. - - -### Cold tier [cold-tier] - -When you no longer need to search time series data regularly, it can move from the warm tier to the cold tier. While still searchable, this tier is typically optimized for lower storage costs rather than search speed. - -For better storage savings, you can keep [fully mounted indices](../../../deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#fully-mounted) of [{{search-snaps}}](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-searchable-snapshot.html) on the cold tier. Unlike regular indices, these fully mounted indices don’t require replicas for reliability. In the event of a failure, they can recover data from the underlying snapshot instead. This potentially halves the local storage needed for the data. A snapshot repository is required to use fully mounted indices in the cold tier. Fully mounted indices are read-only. - -Alternatively, you can use the cold tier to store regular indices with replicas instead of using {{search-snaps}}. This lets you store older data on less expensive hardware but doesn’t reduce required disk space compared to the warm tier. - - -### Frozen tier [frozen-tier] - -Once data is no longer being queried, or being queried rarely, it may move from the cold tier to the frozen tier where it stays for the rest of its life. - -The frozen tier requires a snapshot repository. The frozen tier uses [partially mounted indices](../../../deploy-manage/tools/snapshot-and-restore/searchable-snapshots.md#partially-mounted) to store and load data from a snapshot repository. This reduces local storage and operating costs while still letting you search frozen data. Because {{es}} must sometimes fetch frozen data from the snapshot repository, searches on the frozen tier are typically slower than on the cold tier. - - -## Configure data tiers [configure-data-tiers] - -Follow the instructions for your deployment type to configure data tiers. - - -### {{ess}} or {{ece}} [configure-data-tiers-cloud] - -The default configuration for an {{ecloud}} deployment includes a shared tier for hot and content data. This tier is required and can’t be removed. - -To add a warm, cold, or frozen tier when you create a deployment: - -1. On the **Create deployment** page, click **Advanced Settings**. -2. Click **+ Add capacity** for any data tiers to add. -3. Click **Create deployment** at the bottom of the page to save your changes. - -:::{image} ../../../images/elasticsearch-reference-ess-advanced-config-data-tiers.png -:alt: {{ecloud}}'s deployment Advanced configuration page -:class: screenshot -::: - -To add a data tier to an existing deployment: - -1. Log in to the [{{ecloud}} console](https://cloud.elastic.co?page=docs&placement=docs-body). -2. On the **Deployments** page, select your deployment. -3. In your deployment menu, select **Edit**. -4. Click **+ Add capacity** for any data tiers to add. -5. Click **Save** at the bottom of the page to save your changes. - -To remove a data tier, refer to [Disable a data tier](../../../manage-data/lifecycle/index-lifecycle-management.md). - - -### Self-managed deployments [configure-data-tiers-on-premise] - -For self-managed deployments, each node’s [data role](../../../deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md#data-node-role) is configured in `elasticsearch.yml`. For example, the highest-performance nodes in a cluster might be assigned to both the hot and content tiers: - -```yaml -node.roles: ["data_hot", "data_content"] -``` - -::::{note} -We recommend you use [dedicated nodes](../../../deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md#data-frozen-node) in the frozen tier. -:::: - - - -## Data tier index allocation [data-tier-allocation] - -The [`index.routing.allocation.include._tier_preference`](https://www.elastic.co/guide/en/elasticsearch/reference/current/data-tier-shard-filtering.html#tier-preference-allocation-filter) setting determines which tier the index should be allocated to. - -When you create an index, by default {{es}} sets the `_tier_preference` to `data_content` to automatically allocate the index shards to the content tier. - -When {{es}} creates an index as part of a [data stream](../../../manage-data/data-store/index-types/data-streams.md), by default {{es}} sets the `_tier_preference` to `data_hot` to automatically allocate the index shards to the hot tier. - -At the time of index creation, you can override the default setting by explicitly setting the preferred value in one of two ways: - -* Using an [index template](../../../manage-data/data-store/templates.md). Refer to [Automate rollover with ILM](../../../manage-data/lifecycle/index-lifecycle-management.md) for details. -* Within the [create index](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html) request body. - -You can override this setting after index creation by [updating the index setting](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-update-settings.html) to the preferred value. - -This setting also accepts multiple tiers in order of preference. This prevents indices from remaining unallocated if no nodes are available in the preferred tier. For example, when {{ilm}} migrates an index to the cold phase, it sets the index `_tier_preference` to `data_cold,data_warm,data_hot`. - -To remove the data tier preference setting, set the `_tier_preference` value to `null`. This allows the index to allocate to any data node within the cluster. Setting the `_tier_preference` to `null` does not restore the default value. Note that, in the case of managed indices, a [migrate](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-migrate.html) action might apply a new value in its place. - - -### Determine the current data tier preference [data-tier-allocation-value] - -You can check an existing index’s data tier preference by [polling its settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-settings.html) for `index.routing.allocation.include._tier_preference`: - -```console -GET /my-index-000001/_settings?filter_path=*.settings.index.routing.allocation.include._tier_preference -``` - - -### Troubleshooting [data-tier-allocation-troubleshooting] - -The `_tier_preference` setting might conflict with other allocation settings. This conflict might prevent the shard from allocating. A conflict might occur when a cluster has not yet been completely [migrated to data tiers](../../../troubleshoot/elasticsearch/troubleshoot-migrate-to-tiers.md). - -This setting will not unallocate a currently allocated shard, but might prevent it from migrating from its current location to its designated data tier. To troubleshoot, call the [cluster allocation explain API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html) and specify the suspected problematic shard. - - -### Automatic data tier migration [data-tier-migration] - -{{ilm-init}} automatically transitions managed indices through the available data tiers using the [migrate](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-migrate.html) action. By default, this action is automatically injected in every phase. You can explicitly specify the migrate action with `"enabled": false` to [disable automatic migration](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-migrate.html#ilm-disable-migrate-ex), for example, if you’re using the [allocate action](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-allocate.html) to manually specify allocation rules. diff --git a/raw-migrated-files/toc.yml b/raw-migrated-files/toc.yml index 60707228e9..29c1b39d2d 100644 --- a/raw-migrated-files/toc.yml +++ b/raw-migrated-files/toc.yml @@ -55,7 +55,6 @@ toc: - file: cloud/cloud-enterprise/ece-create-deployment.md - file: cloud/cloud-enterprise/ece-delete-deployment.md - file: cloud/cloud-enterprise/ece-deployment-no-op.md - - file: cloud/cloud-enterprise/ece-disable-data-tier.md - file: cloud/cloud-enterprise/ece-enable-auditing.md - file: cloud/cloud-enterprise/ece-find.md - file: cloud/cloud-enterprise/ece-generate-roles-token.md @@ -400,7 +399,7 @@ toc: - file: docs-content/serverless/security-about-rules.md - file: docs-content/serverless/security-add-exceptions.md - file: docs-content/serverless/security-add-manage-notes.md - - file: docs-content/serverless/security-advanced-behavioral-detections.md + - file: docs-content/serverless/security-advanced-behavioral-detections.md - file: docs-content/serverless/security-advanced-settings.md - file: docs-content/serverless/security-agent-tamper-protection.md - file: docs-content/serverless/security-ai-assistant-esql-queries.md @@ -560,7 +559,6 @@ toc: - file: elasticsearch/elasticsearch-reference/configuring-stack-security.md - file: elasticsearch/elasticsearch-reference/data-management.md - file: elasticsearch/elasticsearch-reference/data-streams.md - - file: elasticsearch/elasticsearch-reference/data-tiers.md - file: elasticsearch/elasticsearch-reference/defining-roles.md - file: elasticsearch/elasticsearch-reference/document-level-security.md - file: elasticsearch/elasticsearch-reference/documents-indices.md From 7a07225612179e52329d774df7dece2f49fb443d Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Wed, 12 Feb 2025 14:43:37 -0600 Subject: [PATCH 3/7] remove code comments --- manage-data/lifecycle/data-tiers.md | 9 --------- 1 file changed, 9 deletions(-) diff --git a/manage-data/lifecycle/data-tiers.md b/manage-data/lifecycle/data-tiers.md index 52fa0130d6..d940888d5d 100644 --- a/manage-data/lifecycle/data-tiers.md +++ b/manage-data/lifecycle/data-tiers.md @@ -4,15 +4,6 @@ mapped_urls: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-disable-data-tier.html --- -% What needs to be done: Refine - -% GitHub issue: docs-projects#375 - -% Use migrated content from existing pages that map to this page: - -% - [x] ./raw-migrated-files/elasticsearch/elasticsearch-reference/data-tiers.md -% - [x] ./raw-migrated-files/cloud/cloud-enterprise/ece-disable-data-tier.md - # Data tiers A *data tier* is a collection of [nodes](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html) within a cluster that share the same [data node role](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#node-roles), and a hardware profile that’s appropriately sized for the role. Elastic recommends that nodes in the same tier share the same hardware profile to avoid [hot spotting](/troubleshoot/elasticsearch/hotspotting.md). From a0339dd8325d0166acc8e8ea711011920793f2c8 Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Wed, 12 Feb 2025 15:04:46 -0600 Subject: [PATCH 4/7] clean up mapped pages --- manage-data/lifecycle/data-tiers.md | 7 +- .../lifecycle/index-lifecycle-management.md | 4 - .../cloud/cloud/ec-disable-data-tier.md | 295 ------------------ raw-migrated-files/toc.yml | 1 - 4 files changed, 4 insertions(+), 303 deletions(-) delete mode 100644 raw-migrated-files/cloud/cloud/ec-disable-data-tier.md diff --git a/manage-data/lifecycle/data-tiers.md b/manage-data/lifecycle/data-tiers.md index d940888d5d..f2cbdee87f 100644 --- a/manage-data/lifecycle/data-tiers.md +++ b/manage-data/lifecycle/data-tiers.md @@ -2,6 +2,7 @@ mapped_urls: - https://www.elastic.co/guide/en/elasticsearch/reference/current/data-tiers.html - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-disable-data-tier.html + - https://www.elastic.co/guide/en/cloud/current/ec-disable-data-tier.html --- # Data tiers @@ -105,7 +106,7 @@ To add a data tier to an existing deployment: 4. Click **+ Add capacity** for any data tiers to add. 5. Click **Save** at the bottom of the page to save your changes. -#### Disable a data tier [ece-update-data-tier-allocation-rules] +#### Disable a data tier [disable-a-data-tier] The process of disabling a data tier depends on whether we are dealing with [searchable snapshots](#ece-disable-searchable-snapshot-data-tier) or [regular indices](#ece-disable-non-searchable-snapshot-data-tier). @@ -169,7 +170,7 @@ To make sure that all data can be migrated from the data tier you want to disabl GET /my-index/_settings ``` - 1. Updating data tier based allocation inclusion rules. + 1. $$$update-data-tier-allocation-rules$$$ Updating data tier based allocation inclusion rules. Data tier based ILM policies use `index.routing.allocation.include` to allocate shards to the appropriate tier. The indices that use this method have index routing settings similar to the following example: @@ -339,7 +340,7 @@ When data reaches the `cold` or `frozen` phases, it is automatically converted t 5. Restore indices from the searchable snapshots. - 1. Follow the steps to [specify the data tier based allocation inclusion rules](/manage-data/lifecycle/data-tiers.md#ece-update-data-tier-allocation-rules). + 1. Follow the steps to [specify the data tier based allocation inclusion rules](/manage-data/lifecycle/data-tiers.md#update-data-tier-allocation-rules). 2. Remove the associated ILM policy (set it to `null`). If you want to apply a different ILM policy, follow the steps to [Switch lifecycle policies](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html#switch-lifecycle-policies). 3. If needed, specify the alias for rollover, otherwise set it to `null`. 4. Optionally, specify the desired number of replica shards. diff --git a/manage-data/lifecycle/index-lifecycle-management.md b/manage-data/lifecycle/index-lifecycle-management.md index 8e8ea13bc9..5fb70203c0 100644 --- a/manage-data/lifecycle/index-lifecycle-management.md +++ b/manage-data/lifecycle/index-lifecycle-management.md @@ -6,7 +6,6 @@ mapped_urls: - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-migrate-index-management.html - https://www.elastic.co/guide/en/cloud/current/ec-configure-index-management.html - https://www.elastic.co/guide/en/cloud/current/ec-migrate-index-management.html - - https://www.elastic.co/guide/en/cloud/current/ec-disable-data-tier.html --- # Index lifecycle management @@ -25,7 +24,6 @@ mapped_urls: % - [ ] ./raw-migrated-files/cloud/cloud-enterprise/ece-migrate-index-management.md % - [ ] ./raw-migrated-files/cloud/cloud/ec-configure-index-management.md % - [ ] ./raw-migrated-files/cloud/cloud/ec-migrate-index-management.md -% - [ ] ./raw-migrated-files/cloud/cloud/ec-disable-data-tier.md % Internal links rely on the following IDs being on this page (e.g. as a heading ID, paragraph ID, etc): @@ -44,5 +42,3 @@ $$$ilm-gs-check-progress$$$ $$$ilm-gs-alias-apply-policy$$$ $$$ilm-gs-alias-check-progress$$$ - -$$$ec-update-data-tier-allocation-rules$$$ \ No newline at end of file diff --git a/raw-migrated-files/cloud/cloud/ec-disable-data-tier.md b/raw-migrated-files/cloud/cloud/ec-disable-data-tier.md deleted file mode 100644 index bab5da2218..0000000000 --- a/raw-migrated-files/cloud/cloud/ec-disable-data-tier.md +++ /dev/null @@ -1,295 +0,0 @@ -# Disable an Elasticsearch data tier [ec-disable-data-tier] - -The process of disabling a data tier depends on whether we are dealing with searchable snapshots or regular indices. - -The hot and warm tiers store regular indices, while the frozen tier stores searchable snapshots. However, the cold tier can store either regular indices or searchable snapshots. To check if a cold tier contains searchable snapshots perform the following request: - -```sh -GET /_cat/indices/restored-* -``` - - -## Non-searchable snapshot data tier [ec-disable-non-searchable-snapshot-data-tier] - -Elasticsearch Service tries to move all data from the nodes that are removed during plan changes. To disable a non-searchable snapshot data tier (e.g., hot, warm, or cold tier), make sure that all data on that tier can be re-allocated by reconfiguring the relevant shard allocation filters. You’ll also need to temporarily stop your index lifecycle management (ILM) policies to prevent new indices from being moved to the data tier you want to disable. - -To learn more about ILM for Elasticsearch Service, or shard allocation filtering, check the following documentation: - -* [Configure index management](../../../manage-data/lifecycle/index-lifecycle-management.md) -* [Create your index lifecyle policy](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html) -* [Managing the index lifecycle](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html) -* [Shard allocation filters](https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-allocation-filtering.html) - -To make sure that all data can be migrated from the data tier you want to disable, follow these steps: - -1. Determine which nodes will be removed from the cluster. - - 1. Log in to the [Elasticsearch Service Console](https://cloud.elastic.co?page=docs&placement=docs-body). - 2. From the **Deployments** page, select your deployment. - - On the deployments page you can narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list. - - 3. Filter the list of instances by the Data tier you want to disable. - - :::{image} ../../../images/cloud-ec-ce-remove-tier-filter-instances.png - :alt: A screenshot showing a filtered instance list - ::: - - Note the listed instance IDs. In this example, it would be Instance 2 and Instance 3. - -2. Stop ILM. - - ```sh - POST /_ilm/stop - ``` - -3. Determine which shards need to be moved. - - ```sh - GET /_cat/shards - ``` - - Parse the output, looking for shards allocated to the nodes to be removed from the cluster. Note that `Instance #2` is shown as `instance-0000000002` in the output. - - :::{image} ../../../images/cloud-ec-ce-remove-tier-filtered-cat-shards.png - :alt: A screenshot showing a filtered shard list - ::: - -4. Move shards off the nodes to be removed from the cluster. - - You must remove any [index-level shard allocation filters](https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-allocation-filtering.html) from the indices on the nodes to be removed. ILM uses different rules depending on the policy and version of Elasticsearch. Check the index settings to determine which rule to use: - - ```sh - GET /my-index/_settings - ``` - - 1. Updating data tier based allocation inclusion rules. - - Data tier based ILM policies use `index.routing.allocation.include` to allocate shards to the appropriate tier. The indices that use this method have index routing settings similar to the following example: - - ```sh - { - ... - "routing": { - "allocation": { - "include": { - "_tier_preference": "data_warm,data_hot" - } - } - } - ... - } - ``` - - You must remove the relevant tier from the inclusion rules. For example, to disable the warm tier, the `data_warm` tier preference should be removed: - - ```sh - PUT /my-index/_settings - { - "routing": { - "allocation": { - "include": { - "_tier_preference": "data_hot" - } - } - } - } - ``` - - Updating allocation inclusion rules will trigger a shard re-allocation, moving the shards from the nodes to be removed. - - 2. Updating node attribute allocation requirement rules. - - Node attribute based ILM policies uses `index.routing.allocation.require` to allocate shards to the appropriate nodes. The indices that use this method have index routing settings that are similar to the following example: - - ```sh - { - ... - "routing": { - "allocation": { - "require": { - "data": "warm" - } - } - } - ... - } - ``` - - You must either remove or redefine the routing requirements. To remove the attribute requirements, use the following code: - - ```sh - PUT /my-index/_settings - { - "routing": { - "allocation": { - "require": { - "data": null - } - } - } - } - ``` - - Removing required attributes does not trigger a shard reallocation. These shards are moved when applying the plan to disable the data tier. Alternatively, you can use the [cluster re-route API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html) to manually re-allocate the shards before removing the nodes, or explicitly re-allocate shards to hot nodes by using the following code: - - ```sh - PUT /my-index/_settings - { - "routing": { - "allocation": { - "require": { - "data": "hot" - } - } - } - } - ``` - - 3. Removing custom allocation rules. - - If indices on nodes to be removed have shard allocation rules of other forms, they must be removed as shown in the following example: - - ```sh - PUT /my-index/_settings - { - "routing": { - "allocation": { - "require": null, - "include": null, - "exclude": null - } - } - } - ``` - -5. Edit the deployment, disabling the data tier. - - If autoscaling is enabled, set the maximum size to 0 for the data tier to ensure autoscaling does not re-enable the data tier. - - Any remaining shards on the tier being disabled are re-allocated across the remaining cluster nodes while applying the plan to disable the data tier. Monitor shard allocation during the data migration phase to ensure all allocation rules have been correctly updated. If the plan fails to migrate data away from the data tier, then re-examine the allocation rules for the indices remaining on that data tier. - -6. Once the plan change completes, confirm that there are no remaining nodes associated with the disabled tier and that `GET _cluster/health` reports `green`. If this is the case, re-enable ILM. - - ```sh - POST _ilm/start - ``` - - - -## Searchable snapshot data tier [ec-disable-searchable-snapshot-data-tier] - -When data reaches the `cold` or `frozen` phases, it is automatically converted to a [searchable snapshot](https://www.elastic.co/guide/en/elasticsearch/reference/current/searchable-snapshots.html) by ILM. If you do not intend to delete this data, you should manually restore each of the searchable snapshot indices to a regular index before disabling the data tier, by following these steps: - -1. Stop ILM and check ILM status is `STOPPED` to prevent data from migrating to the phase you intend to disable while you are working through the next steps. - - ```sh - # stop ILM - POST _ilm/stop - - # check status - GET _ilm/status - ``` - -2. Capture a comprehensive list of index and searchable snapshot names. - - 1. The index name of the searchable snapshots may differ based on the data tier. If you intend to disable the cold tier, then perform the following request with the `restored-*` prefix. If the frozen tier is the one to be disabled, use the `partial-*` prefix. - - ```sh - GET /_settings?filter_path=**.index.store.snapshot.snapshot_name&expand_wildcards=all - ``` - - In the example we have a list of 4 indices, which need to be moved away from the frozen tier. - - :::{image} ../../../images/cloud-ec-ce-remove-tier-filter-snapshot-indices.png - :alt: A screenshot showing a snapshot indices list - ::: - -3. (Optional) Save the list of index and snapshot names in a text file, so you can access it throughout the rest of the process. -4. Remove the aliases that were applied to searchable snapshots indices. Use the index prefix from step 2. - - ```sh - POST _aliases - { - "actions": [ - { - "remove": { - "index": "-", - "alias": "" - } - } - ] - } - ``` - - ::::{note} - If you use data stream, you can skip this step. - :::: - - - In the example we are removing the alias for the `frozen-index-1` index. - - :::{image} ../../../images/cloud-ec-ce-remove-tier-remove-alias.png - :alt: A screenshot showing the process of removing a searchable snapshot indice alias - ::: - -5. Restore indices from the searchable snapshots. - - 1. Follow the steps to [specify the data tier based allocation inclusion rules](../../../manage-data/lifecycle/index-lifecycle-management.md#ec-update-data-tier-allocation-rules). - 2. Remove the associated ILM policy (set it to `null`). If you want to apply a different ILM policy, follow the steps to [Switch lifecycle policies](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html#switch-lifecycle-policies). - 3. If needed, specify the alias for rollover, otherwise set it to `null`. - 4. Optionally, specify the desired number of replica shards. - - ```sh - POST _snapshot/found-snapshots//_restore - { - "indices": "*", - "index_settings": { - "index.routing.allocation.include._tier_preference": "", - "number_of_replicas": X, - "index.lifecycle.name": "", - "index.lifecycle.rollover_alias": "" - } - } - ``` - - The `` refers to the above-mentioned step: "Capture a comprehensive list of index and searchable snapshot names". - - In the example we are restoring `frozen-index-1` from the snapshot in `found-snapshots` (default snapshot repository) and placing it in the warm tier. - - :::{image} ../../../images/cloud-ec-ce-remove-tier-restore-snapshot.png - :alt: A screenshot showing the process of restoring a searchable snapshot to a regular index - ::: - -6. Repeat steps 4 and 5 until all snapshots are restored to regular indices. -7. Once all snapshots are restored, use `GET _cat/indices/?v=true` to check that the restored indices are `green` and are correctly reflecting the expected `doc` and `store.size` counts. - - If you are using data stream, you may need to use `GET _data_stream/` to get the list of the backing indices, and then specify them by using `GET _cat/indices/?v=true` to check. - -8. Once your data has completed restoration from searchable snapshots to the target data tier, `DELETE` searchable snapshot indices using the prefix from step 2. - - ```sh - DELETE - - ``` - -9. Delete the searchable snapshots by following these steps: - - 1. Open Kibana and navigate to Management > Data > Snapshot and Restore > Snapshots (or go to `/app/management/data/snapshot_restore/snapshots`) - 2. Search for `**` - 3. Bulk select the snapshots and delete them - - In the example we are deleting the snapshots associated with the `policy_with_frozen_phase`. - - :::{image} ../../../images/cloud-ec-ce-remove-tier-remove-snapshots.png - :alt: A screenshot showing the process of deleting snapshots - ::: - -10. Confirm that no shards remain on the data nodes you wish to remove using `GET _cat/allocation?v=true&s=node`. -11. Edit your cluster from the console to disable the data tier. -12. Once the plan change completes, confirm that there are no remaining nodes associated with the disabled tier and that `GET _cluster/health` reports `green`. If this is the case, re-enable ILM. - - ```sh - POST _ilm/start - ``` - - diff --git a/raw-migrated-files/toc.yml b/raw-migrated-files/toc.yml index 29c1b39d2d..d99d0c0379 100644 --- a/raw-migrated-files/toc.yml +++ b/raw-migrated-files/toc.yml @@ -182,7 +182,6 @@ toc: - file: cloud/cloud/ec-customize-deployment.md - file: cloud/cloud/ec-delete-deployment.md - file: cloud/cloud/ec-deployment-no-op.md - - file: cloud/cloud/ec-disable-data-tier.md - file: cloud/cloud/ec-editing-user-settings.md - file: cloud/cloud/ec-enable-logging-and-monitoring.md - file: cloud/cloud/ec-faq-getting-started.md From c53ac1a7a90460f93896de36117af92e2dca6b65 Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Thu, 13 Feb 2025 11:12:09 -0600 Subject: [PATCH 5/7] address feeddback from @kilfoyle --- manage-data/lifecycle/data-tiers.md | 34 ++++++++++++++++++++++++----- 1 file changed, 28 insertions(+), 6 deletions(-) diff --git a/manage-data/lifecycle/data-tiers.md b/manage-data/lifecycle/data-tiers.md index f2cbdee87f..65716741ea 100644 --- a/manage-data/lifecycle/data-tiers.md +++ b/manage-data/lifecycle/data-tiers.md @@ -118,19 +118,38 @@ GET /_cat/indices/restored-* ##### Non-searchable snapshot data tier [ece-disable-non-searchable-snapshot-data-tier] -Elastic Cloud Enterprise tries to move all data from the nodes that are removed during plan changes. To disable a non-searchable snapshot data tier (e.g., hot, warm, or cold tier), make sure that all data on that tier can be re-allocated by reconfiguring the relevant shard allocation filters. You’ll also need to temporarily stop your index lifecycle management (ILM) policies to prevent new indices from being moved to the data tier you want to disable. +{{ess}} and {{ece}} try to move all data from the nodes that are removed during plan changes. To disable a non-searchable snapshot data tier (e.g., hot, warm, or cold tier), make sure that all data on that tier can be re-allocated by reconfiguring the relevant shard allocation filters. You’ll also need to temporarily stop your index lifecycle management (ILM) policies to prevent new indices from being moved to the data tier you want to disable. -To learn more about ILM for Elastic Cloud Enterprise, or shard allocation filtering, check the following documentation: +To learn more about ILM, or shard allocation filtering, check the following documentation: -* [Configure index management](https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-index-management.html) -* [Create your index lifecyle policy](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html) -* [Managing the index lifecycle](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html) -* [Shard allocation filters](https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-allocation-filtering.html) +* [Create your index lifecyle policy](/manage-data/lifecycle/index-lifecycle-management/configure-lifecycle-policy.md) +* [Managing the index lifecycle](/manage-data/lifecycle/index-lifecycle-management.md) +* [Shard allocation filters](/deploy-manage/distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md) To make sure that all data can be migrated from the data tier you want to disable, follow these steps: 1. Determine which nodes will be removed from the cluster. + :::::{tab-set} + + ::::{tab-item} {{ess}} + + 1. Log in to the [Elasticsearch Service Console](https://cloud.elastic.co?page=docs&placement=docs-body). + 2. From the **Deployments** page, select your deployment. + + On the deployments page you can narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list. + + 3. Filter the list of instances by the Data tier you want to disable. + + :::{image} ../../images/cloud-ec-ce-remove-tier-filter-instances.png + :alt: A screenshot showing a filtered instance list + ::: + + Note the listed instance IDs. In this example, it would be Instance 2 and Instance 3. + + :::: + + ::::{tab-item} Elastic Cloud Enterprise 1. [Log into the Cloud UI](/deploy-manage/deploy/cloud-enterprise/log-into-cloud-ui.md). 2. From the **Deployments** page, select your deployment. @@ -143,6 +162,9 @@ To make sure that all data can be migrated from the data tier you want to disabl ::: Note the listed instance IDs. In this example, it would be Instance 2 and Instance 3. + :::: + + ::::: 2. Stop ILM. From a26a560dd075f772f983eb5f1c0a96db03e0d4d6 Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Thu, 13 Feb 2025 11:16:40 -0600 Subject: [PATCH 6/7] Update manage-data/lifecycle/data-tiers.md Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com> --- manage-data/lifecycle/data-tiers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/manage-data/lifecycle/data-tiers.md b/manage-data/lifecycle/data-tiers.md index 65716741ea..9cc4f7002c 100644 --- a/manage-data/lifecycle/data-tiers.md +++ b/manage-data/lifecycle/data-tiers.md @@ -100,7 +100,7 @@ To add a warm, cold, or frozen tier when you create a deployment: To add a data tier to an existing deployment: -1. Log in to the [{{ecloud}} console](https://cloud.elastic.co?page=docs&placement=docs-body). +1. Log in to the [{{ecloud}} console](https://cloud.elastic.co?page=docs&placement=docs-body) or the {{ece}} admin console. 2. On the **Deployments** page, select your deployment. 3. In your deployment menu, select **Edit**. 4. Click **+ Add capacity** for any data tiers to add. From 4cd3377d16a192896f669112446eac3658c1af94 Mon Sep 17 00:00:00 2001 From: Colleen McGinnis Date: Thu, 13 Feb 2025 11:29:56 -0600 Subject: [PATCH 7/7] ess to ech --- manage-data/lifecycle/data-tiers.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/manage-data/lifecycle/data-tiers.md b/manage-data/lifecycle/data-tiers.md index 9cc4f7002c..1ea3ebe769 100644 --- a/manage-data/lifecycle/data-tiers.md +++ b/manage-data/lifecycle/data-tiers.md @@ -81,7 +81,7 @@ The frozen tier requires a snapshot repository. The frozen tier uses [partially Follow the instructions for your deployment type to configure data tiers. -### {{ess}} or {{ece}} [configure-data-tiers-cloud] +### {{ech}} or {{ece}} [configure-data-tiers-cloud] The default configuration for an {{ecloud}} deployment includes a shared tier for hot and content data. This tier is required and can’t be removed. @@ -118,7 +118,7 @@ GET /_cat/indices/restored-* ##### Non-searchable snapshot data tier [ece-disable-non-searchable-snapshot-data-tier] -{{ess}} and {{ece}} try to move all data from the nodes that are removed during plan changes. To disable a non-searchable snapshot data tier (e.g., hot, warm, or cold tier), make sure that all data on that tier can be re-allocated by reconfiguring the relevant shard allocation filters. You’ll also need to temporarily stop your index lifecycle management (ILM) policies to prevent new indices from being moved to the data tier you want to disable. +{{ech}} and {{ece}} try to move all data from the nodes that are removed during plan changes. To disable a non-searchable snapshot data tier (e.g., hot, warm, or cold tier), make sure that all data on that tier can be re-allocated by reconfiguring the relevant shard allocation filters. You’ll also need to temporarily stop your index lifecycle management (ILM) policies to prevent new indices from being moved to the data tier you want to disable. To learn more about ILM, or shard allocation filtering, check the following documentation: @@ -132,7 +132,7 @@ To make sure that all data can be migrated from the data tier you want to disabl :::::{tab-set} - ::::{tab-item} {{ess}} + ::::{tab-item} {{ech}} 1. Log in to the [Elasticsearch Service Console](https://cloud.elastic.co?page=docs&placement=docs-body). 2. From the **Deployments** page, select your deployment.