From fcd05b8a2cce5892c3e076929332b7f5ac652da4 Mon Sep 17 00:00:00 2001 From: kosabogi Date: Tue, 4 Feb 2025 11:08:47 +0100 Subject: [PATCH 01/12] Adds Backup and high availability landing page --- deploy-manage/tools.md | 46 ++++++++++++++++++++++++++++++------------ 1 file changed, 33 insertions(+), 13 deletions(-) diff --git a/deploy-manage/tools.md b/deploy-manage/tools.md index dbbe87b9f4..6c69842cc2 100644 --- a/deploy-manage/tools.md +++ b/deploy-manage/tools.md @@ -5,28 +5,48 @@ mapped_pages: # Backup, high availability, and resilience tools [high-availability] -Your data is important to you. Keeping it safe and available is important to Elastic. Sometimes your cluster may experience hardware failure or a power loss. To help you plan for this, {{es}} offers a number of features to achieve high availability despite failures. Depending on your deployment type, you might need to provision servers in different zones or configure external repositories to meet your organization’s availability needs. +Elasticsearch provides comprehensive tools to safeguard data, ensure continuous availability, and maintain resilience. These tools are designed to support disaster recovery strategies, enabling businesses to protect critical information and minimize downtime, and maintain high availability in case of unexpected failures. -* **[Design for resilience](production-guidance/availability-and-resilience.md)** +For strategies to design resilient clusters, see **[Availability and resilience](production-guidance/availability-and-resilience.md)**. - Distributed systems like Elasticsearch are designed to keep working even if some of their components have failed. An Elasticsearch cluster can continue operating normally if some of its nodes are unavailable or disconnected, as long as there are enough well-connected nodes to take over the unavailable node’s responsibilities. +::::{note} +Snapshot and restore, as well as cross-cluster replication, are not currently available in serverless deployments, but they are planned for future releases. For more information, see **[LINK-TODO]**. +:::: - If you’re designing a smaller cluster, you might focus on making your cluster resilient to single-node failures. Designers of larger clusters must also consider cases where multiple nodes fail at the same time. +## Snapshots -* **[Cross-cluster replication](tools/cross-cluster-replication.md)** +Snapshots in Elasticsearch are point-in-time backups that include your cluster's data, settings, and overall state. They capture all the information necessary to restore your cluster to a specific moment in time, making them essential for protecting data, recovering from unexpected issues, and transferring data between clusters. Snapshots are a reliable way to ensure the safety of your data and maintain the continuity of your operations. - To effectively distribute read and write operations across nodes, the nodes in a cluster need good, reliable connections to each other. To provide better connections, you typically co-locate the nodes in the same data center or nearby data centers. +You can perform the following tasks to manage snapshots and snapshot repositories: - Co-locating nodes in a single location exposes you to the risk of a single outage taking your entire cluster offline. To maintain high availability, you can prepare a second cluster that can take over in case of disaster by implementing {{ccr}} (CCR). +- **[Register a repository](tools/snapshot-and-restore/manage-snapshot-repositories.md):** Configure storage repositories (for example, S3, Azure, Google Cloud) to store snapshots. The way that you register repositories differs depending on your deployment method: + - **[Elastic Cloud Hosted](tools/snapshot-and-restore/elastic-cloud-hosted.md):** Deployments come with a preconfigured S3 repository for automatic backups, simplifying the setup process. You can also register external repositories, such as Azure, and Google Cloud, for more flexibility. + - **[Elastic Cloud Enterprise](tools/snapshot-and-restore/cloud-enterprise.md):** Repository configuration is managed through the Elastic Cloud Enterprise user interface and automatically linked to deployments. + - **[Elastic Cloud on Kubernetes](tools/snapshot-and-restore/cloud-on-k8s.md) and [self-managed](tools/snapshot-and-restore/self-managed.md) deployments:** Repositories must be configured manually. - CCR provides a way to automatically synchronize indices from a leader cluster to a follower cluster. This cluster could be in a different data center or even a different content from the leader cluster. If the primary cluster fails, the secondary cluster can take over. +- **[Create snapshots](tools/snapshot-and-restore/create-snapshots.md):** Manually or automatically create backups of your cluster. +- **[Restore a snapshot](tools/snapshot-and-restore/restore-snapshot.md):** Recover indices, data streams, or the entire cluster to revert to a previous state. You can choose to restore specific parts of a snapshot, such as a single index, or perform a full restore. - ::::{tip} - You can also use CCR to create secondary clusters to serve read requests in geo-proximity to your users. - :::: +To reduce storage costs for infrequently accessed data while maintaining access, you can also create **[searchable snapshots](tools/snapshot-and-restore/searchable-snapshots.md)**. -* **[Snapshots](tools/snapshot-and-restore.md)** +::::{note} +Snapshot configurations vary across Elastic Cloud Hosted, Elastic Cloud Enterprise (ECE), Elastic Cloud on Kubernetes (ECK), and self-managed deployments. +:::: - Take snapshots of your cluster that can be restored in case of failure. +## Cross-cluster replication (CCR) +**[Cross-cluster replication (CCR)](tools/cross-cluster-replication.md)** is a feature in Elasticsearch that allows you to replicate data in real time from a leader cluster to one or more follower clusters. This replication ensures that data is synchronized across clusters, providing continuity, redundancy, and enhanced data accessibility. +CCR provides a way to automatically synchronize indices from a leader cluster to a follower cluster. This cluster could be in a different data center or even a different continent from the leader cluster. If the primary cluster fails, the secondary cluster can take over. + +::::{note} +CCR relies on **[Remote clusters](remote-clusters.md)** functionality to establish and manage connections between the leader and the follower clusters. +:::: + +You can perform the following tasks to manage cross-cluster replication: + +- **[Set up CCR](tools/cross-cluster-replication/set-up-cross-cluster-replication.md):** Configure leader and follower clusters for data replication. To enable CCR, you need to configure **[Remote clusters](LINK)**. +- **[Manage CCR](tools/cross-cluster-replication/manage-cross-cluster-replication.md):** Monitor and manage replicated indices. +- **[Automate replication](tools/cross-cluster-replication/manage-auto-follow-patterns.md):** Use auto-follow patterns to automatically replicate newly created indices. +- **Failover clusters:** Configure **[uni-directional](tools/cross-cluster-replication/uni-directional-disaster-recovery.md)** or **[bi-directional](tools/cross-cluster-replication/bi-directional-disaster-recovery.md)** CCR for redundancy and disaster recovery. +- **[Cluster upgrade considerations when using CCR](upgrade.md):** Maintain cluster performance and security by upgrading your Elasticsearch clusters. Use the tools and procedures suited to your deployment type: Elastic Cloud Hosted, ECE, ECK, or self-managed. \ No newline at end of file From 1893e57019c37c94373ecf90e69510fc82dc2480 Mon Sep 17 00:00:00 2001 From: kosabogi Date: Tue, 4 Feb 2025 11:39:18 +0100 Subject: [PATCH 02/12] Trigger new deployment --- deploy-manage/tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deploy-manage/tools.md b/deploy-manage/tools.md index 6c69842cc2..718d65e81f 100644 --- a/deploy-manage/tools.md +++ b/deploy-manage/tools.md @@ -5,7 +5,7 @@ mapped_pages: # Backup, high availability, and resilience tools [high-availability] -Elasticsearch provides comprehensive tools to safeguard data, ensure continuous availability, and maintain resilience. These tools are designed to support disaster recovery strategies, enabling businesses to protect critical information and minimize downtime, and maintain high availability in case of unexpected failures. +Elasticsearch provides comprehensive tools to safeguard data, ensure continuous availability, and maintain resilience. These tools are designed to support disaster recovery strategies, enabling businesses to protect critical information and minimize downtime, and maintain high availability in case of unexpected failures. For strategies to design resilient clusters, see **[Availability and resilience](production-guidance/availability-and-resilience.md)**. From 2388fbd171da50fb2a2e331b96ebc507715b8d15 Mon Sep 17 00:00:00 2001 From: kosabogi Date: Tue, 4 Feb 2025 11:43:18 +0100 Subject: [PATCH 03/12] Fixes link --- deploy-manage/tools.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/deploy-manage/tools.md b/deploy-manage/tools.md index 718d65e81f..cabf5953f7 100644 --- a/deploy-manage/tools.md +++ b/deploy-manage/tools.md @@ -5,7 +5,7 @@ mapped_pages: # Backup, high availability, and resilience tools [high-availability] -Elasticsearch provides comprehensive tools to safeguard data, ensure continuous availability, and maintain resilience. These tools are designed to support disaster recovery strategies, enabling businesses to protect critical information and minimize downtime, and maintain high availability in case of unexpected failures. +Elasticsearch provides comprehensive tools to safeguard data, ensure continuous availability, and maintain resilience. These tools are designed to support disaster recovery strategies, enabling businesses to protect critical information and minimize downtime, and maintain high availability in case of unexpected failures. For strategies to design resilient clusters, see **[Availability and resilience](production-guidance/availability-and-resilience.md)**. @@ -45,7 +45,7 @@ CCR relies on **[Remote clusters](remote-clusters.md)** functionality to establi You can perform the following tasks to manage cross-cluster replication: -- **[Set up CCR](tools/cross-cluster-replication/set-up-cross-cluster-replication.md):** Configure leader and follower clusters for data replication. To enable CCR, you need to configure **[Remote clusters](LINK)**. +- **[Set up CCR](tools/cross-cluster-replication/set-up-cross-cluster-replication.md):** Configure leader and follower clusters for data replication. To enable CCR, you need to configure **[Remote clusters](remote-clusters.md)**. - **[Manage CCR](tools/cross-cluster-replication/manage-cross-cluster-replication.md):** Monitor and manage replicated indices. - **[Automate replication](tools/cross-cluster-replication/manage-auto-follow-patterns.md):** Use auto-follow patterns to automatically replicate newly created indices. - **Failover clusters:** Configure **[uni-directional](tools/cross-cluster-replication/uni-directional-disaster-recovery.md)** or **[bi-directional](tools/cross-cluster-replication/bi-directional-disaster-recovery.md)** CCR for redundancy and disaster recovery. From 4fd2befb4fdc8ff82006c749cf941612000f2020 Mon Sep 17 00:00:00 2001 From: kosabogi <105062005+kosabogi@users.noreply.github.com> Date: Wed, 5 Feb 2025 08:35:50 +0100 Subject: [PATCH 04/12] Update deploy-manage/tools.md Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> --- deploy-manage/tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deploy-manage/tools.md b/deploy-manage/tools.md index cabf5953f7..cf28b869df 100644 --- a/deploy-manage/tools.md +++ b/deploy-manage/tools.md @@ -5,7 +5,7 @@ mapped_pages: # Backup, high availability, and resilience tools [high-availability] -Elasticsearch provides comprehensive tools to safeguard data, ensure continuous availability, and maintain resilience. These tools are designed to support disaster recovery strategies, enabling businesses to protect critical information and minimize downtime, and maintain high availability in case of unexpected failures. +Elastic provides comprehensive tools to safeguard data, ensure continuous availability, and maintain resilience. These tools are designed to support disaster recovery strategies, enabling businesses to protect critical information and minimize downtime, and maintain high availability in case of unexpected failures. In this section, you'll learn about these tools and how to implement them in your environment. For strategies to design resilient clusters, see **[Availability and resilience](production-guidance/availability-and-resilience.md)**. From 2733cf3ef20e827d3e3bd20b3bddecb153667c59 Mon Sep 17 00:00:00 2001 From: kosabogi <105062005+kosabogi@users.noreply.github.com> Date: Wed, 5 Feb 2025 08:37:15 +0100 Subject: [PATCH 05/12] Update deploy-manage/tools.md Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> --- deploy-manage/tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deploy-manage/tools.md b/deploy-manage/tools.md index cf28b869df..df51d577ae 100644 --- a/deploy-manage/tools.md +++ b/deploy-manage/tools.md @@ -10,7 +10,7 @@ Elastic provides comprehensive tools to safeguard data, ensure continuous availa For strategies to design resilient clusters, see **[Availability and resilience](production-guidance/availability-and-resilience.md)**. ::::{note} -Snapshot and restore, as well as cross-cluster replication, are not currently available in serverless deployments, but they are planned for future releases. For more information, see **[LINK-TODO]**. +The snapshot and restore and cross-cluster replication features are currently not available for Elastic Cloud Serverless projects. These features will be introduced in the future. For more information, refer to [Serverless differences](/deploy-manage/deploy/elastic-cloud/differences-from-other-elasticsearch-offerings.md#elasticsearch-differences-serverless-feature-categories). :::: ## Snapshots From d264df23aaaed097c8ad410965fb55c783188bb4 Mon Sep 17 00:00:00 2001 From: kosabogi <105062005+kosabogi@users.noreply.github.com> Date: Wed, 5 Feb 2025 08:37:33 +0100 Subject: [PATCH 06/12] Update deploy-manage/tools.md Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> --- deploy-manage/tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deploy-manage/tools.md b/deploy-manage/tools.md index df51d577ae..1ff91ef9b5 100644 --- a/deploy-manage/tools.md +++ b/deploy-manage/tools.md @@ -13,7 +13,7 @@ For strategies to design resilient clusters, see **[Availability and resilience] The snapshot and restore and cross-cluster replication features are currently not available for Elastic Cloud Serverless projects. These features will be introduced in the future. For more information, refer to [Serverless differences](/deploy-manage/deploy/elastic-cloud/differences-from-other-elasticsearch-offerings.md#elasticsearch-differences-serverless-feature-categories). :::: -## Snapshots +## Snapshot and restore Snapshots in Elasticsearch are point-in-time backups that include your cluster's data, settings, and overall state. They capture all the information necessary to restore your cluster to a specific moment in time, making them essential for protecting data, recovering from unexpected issues, and transferring data between clusters. Snapshots are a reliable way to ensure the safety of your data and maintain the continuity of your operations. From 88c3e815ce65d583057fea93c67b5e9795c9c0c7 Mon Sep 17 00:00:00 2001 From: kosabogi <105062005+kosabogi@users.noreply.github.com> Date: Wed, 5 Feb 2025 08:38:04 +0100 Subject: [PATCH 07/12] Update deploy-manage/tools.md Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> --- deploy-manage/tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deploy-manage/tools.md b/deploy-manage/tools.md index 1ff91ef9b5..64af4eb5dc 100644 --- a/deploy-manage/tools.md +++ b/deploy-manage/tools.md @@ -40,7 +40,7 @@ Snapshot configurations vary across Elastic Cloud Hosted, Elastic Cloud Enterpri CCR provides a way to automatically synchronize indices from a leader cluster to a follower cluster. This cluster could be in a different data center or even a different continent from the leader cluster. If the primary cluster fails, the secondary cluster can take over. ::::{note} -CCR relies on **[Remote clusters](remote-clusters.md)** functionality to establish and manage connections between the leader and the follower clusters. +CCR relies on **[remote clusters](remote-clusters.md)** functionality to establish and manage connections between the leader and the follower clusters. :::: You can perform the following tasks to manage cross-cluster replication: From b2b65a3a0dbe44ecf8ee00f215cd5739f6dd21e9 Mon Sep 17 00:00:00 2001 From: kosabogi <105062005+kosabogi@users.noreply.github.com> Date: Wed, 5 Feb 2025 08:42:01 +0100 Subject: [PATCH 08/12] Update deploy-manage/tools.md Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> --- deploy-manage/tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deploy-manage/tools.md b/deploy-manage/tools.md index 64af4eb5dc..16790fd50e 100644 --- a/deploy-manage/tools.md +++ b/deploy-manage/tools.md @@ -45,7 +45,7 @@ CCR relies on **[remote clusters](remote-clusters.md)** functionality to establi You can perform the following tasks to manage cross-cluster replication: -- **[Set up CCR](tools/cross-cluster-replication/set-up-cross-cluster-replication.md):** Configure leader and follower clusters for data replication. To enable CCR, you need to configure **[Remote clusters](remote-clusters.md)**. +- **[Set up CCR](tools/cross-cluster-replication/set-up-cross-cluster-replication.md):** Configure leader and follower clusters for data replication. - **[Manage CCR](tools/cross-cluster-replication/manage-cross-cluster-replication.md):** Monitor and manage replicated indices. - **[Automate replication](tools/cross-cluster-replication/manage-auto-follow-patterns.md):** Use auto-follow patterns to automatically replicate newly created indices. - **Failover clusters:** Configure **[uni-directional](tools/cross-cluster-replication/uni-directional-disaster-recovery.md)** or **[bi-directional](tools/cross-cluster-replication/bi-directional-disaster-recovery.md)** CCR for redundancy and disaster recovery. From 85540918d7318bc63bbf96b69a8cb945fe36633a Mon Sep 17 00:00:00 2001 From: kosabogi <105062005+kosabogi@users.noreply.github.com> Date: Wed, 5 Feb 2025 08:42:17 +0100 Subject: [PATCH 09/12] Update deploy-manage/tools.md Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> --- deploy-manage/tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deploy-manage/tools.md b/deploy-manage/tools.md index 16790fd50e..3697e31e24 100644 --- a/deploy-manage/tools.md +++ b/deploy-manage/tools.md @@ -48,5 +48,5 @@ You can perform the following tasks to manage cross-cluster replication: - **[Set up CCR](tools/cross-cluster-replication/set-up-cross-cluster-replication.md):** Configure leader and follower clusters for data replication. - **[Manage CCR](tools/cross-cluster-replication/manage-cross-cluster-replication.md):** Monitor and manage replicated indices. - **[Automate replication](tools/cross-cluster-replication/manage-auto-follow-patterns.md):** Use auto-follow patterns to automatically replicate newly created indices. -- **Failover clusters:** Configure **[uni-directional](tools/cross-cluster-replication/uni-directional-disaster-recovery.md)** or **[bi-directional](tools/cross-cluster-replication/bi-directional-disaster-recovery.md)** CCR for redundancy and disaster recovery. +- **Set up failover clusters:** Configure **[uni-directional](tools/cross-cluster-replication/uni-directional-disaster-recovery.md)** or **[bi-directional](tools/cross-cluster-replication/bi-directional-disaster-recovery.md)** CCR for redundancy and disaster recovery. - **[Cluster upgrade considerations when using CCR](upgrade.md):** Maintain cluster performance and security by upgrading your Elasticsearch clusters. Use the tools and procedures suited to your deployment type: Elastic Cloud Hosted, ECE, ECK, or self-managed. \ No newline at end of file From 6da166cfe9c00a87759ee285e4c779b789690f97 Mon Sep 17 00:00:00 2001 From: kosabogi <105062005+kosabogi@users.noreply.github.com> Date: Wed, 5 Feb 2025 08:43:07 +0100 Subject: [PATCH 10/12] Update deploy-manage/tools.md Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> --- deploy-manage/tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deploy-manage/tools.md b/deploy-manage/tools.md index 3697e31e24..3970107116 100644 --- a/deploy-manage/tools.md +++ b/deploy-manage/tools.md @@ -49,4 +49,4 @@ You can perform the following tasks to manage cross-cluster replication: - **[Manage CCR](tools/cross-cluster-replication/manage-cross-cluster-replication.md):** Monitor and manage replicated indices. - **[Automate replication](tools/cross-cluster-replication/manage-auto-follow-patterns.md):** Use auto-follow patterns to automatically replicate newly created indices. - **Set up failover clusters:** Configure **[uni-directional](tools/cross-cluster-replication/uni-directional-disaster-recovery.md)** or **[bi-directional](tools/cross-cluster-replication/bi-directional-disaster-recovery.md)** CCR for redundancy and disaster recovery. -- **[Cluster upgrade considerations when using CCR](upgrade.md):** Maintain cluster performance and security by upgrading your Elasticsearch clusters. Use the tools and procedures suited to your deployment type: Elastic Cloud Hosted, ECE, ECK, or self-managed. \ No newline at end of file +- **[Review cluster upgrade considerations when using CCR](upgrade.md):** If you're using CCR, then you might need to upgrade your clusters in a specific order to prevent errors. Review the considerations and recommended procedures for performing upgrades on CCR leaders and followers. \ No newline at end of file From 33b8d05261e7231932198373c63c0f0ea0fac426 Mon Sep 17 00:00:00 2001 From: kosabogi Date: Wed, 5 Feb 2025 09:09:41 +0100 Subject: [PATCH 11/12] Adds applies-to labels --- deploy-manage/tools.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/deploy-manage/tools.md b/deploy-manage/tools.md index 3970107116..f3e5a5324e 100644 --- a/deploy-manage/tools.md +++ b/deploy-manage/tools.md @@ -1,8 +1,16 @@ --- mapped_pages: - https://www.elastic.co/guide/en/elasticsearch/reference/current/high-availability.html + +applies: + stack: all + serverless: all + hosted: all + ece: all + eck: all --- + # Backup, high availability, and resilience tools [high-availability] Elastic provides comprehensive tools to safeguard data, ensure continuous availability, and maintain resilience. These tools are designed to support disaster recovery strategies, enabling businesses to protect critical information and minimize downtime, and maintain high availability in case of unexpected failures. In this section, you'll learn about these tools and how to implement them in your environment. From 3238b15e846a26bf95dd88a773257cda61a1561d Mon Sep 17 00:00:00 2001 From: kosabogi <105062005+kosabogi@users.noreply.github.com> Date: Wed, 5 Feb 2025 15:27:51 +0100 Subject: [PATCH 12/12] Update deploy-manage/tools.md Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> --- deploy-manage/tools.md | 1 - 1 file changed, 1 deletion(-) diff --git a/deploy-manage/tools.md b/deploy-manage/tools.md index f3e5a5324e..85342ade85 100644 --- a/deploy-manage/tools.md +++ b/deploy-manage/tools.md @@ -4,7 +4,6 @@ mapped_pages: applies: stack: all - serverless: all hosted: all ece: all eck: all