From 1447c7307d345b2d5fa6d67375593d96229dc9bd Mon Sep 17 00:00:00 2001 From: scalar-boney Date: Mon, 14 Sep 2020 14:02:19 +0900 Subject: [PATCH 01/10] troubleshoot doc [skip ci] --- docs/TroubleshootingGuide.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 docs/TroubleshootingGuide.md diff --git a/docs/TroubleshootingGuide.md b/docs/TroubleshootingGuide.md new file mode 100644 index 000000000..84f157095 --- /dev/null +++ b/docs/TroubleshootingGuide.md @@ -0,0 +1,34 @@ +# Troubleshoot Guide + +This guide explains how to replace a node when the node cannot be replaced with normal procedures. This is especially useful when the node or os-disk is accidentally terminated in the **Azure** environment. + +Use this Troubleshooting Guide to: +- Replace accidentally removed node or os-disk +- Replace accidentally removed cassandra node or os-disk with existing data disk + +## Replace accidentally removed node or os-disk +The following process helps to replace accidentally terminated node or os-disk, but that node should not have an additional data disk. + +Note: Mainly applicable for scalardl, envoy, cassy, reaper, monitor and ca nodes. + +Please try the following +- Remove the os-disk If the node is not available in the resource group. +- Remove the node If the os-disk is not available in the resource group. +- Then you can follow [Node Replacement](NodeReplacement.md) + +## Replace accidentally removed cassandra node or os-disk with existing data disk +This documentation helps to replace accidentally terminated node or os-disk with existing data disk (taint volume attachment). + +Please try the following +* Remove the os-disk If the node is not available in the resource group. +* Remove the node If the os-disk is not available in the resource group. +* Do `terraform state rm` as follows. + +```console +terraform taint "module.cassandra.module.cassandra_cluster.azurerm_virtual_machine.vm-linux[0]" +terraform state rm "module.cassandra.azurerm_virtual_machine_data_disk_attachment.cassandra_data_volume_attachment[0]" + +terraform apply +``` + +Then you can follow [Cassandra Post Recovery Steps](CassandraOperation.md#post-recovery-steps) \ No newline at end of file From 62cf82386901bdfbfa531426d9f31a1d169b0e17 Mon Sep 17 00:00:00 2001 From: scalar-boney Date: Mon, 14 Sep 2020 14:56:49 +0900 Subject: [PATCH 02/10] troubleshoot doc modified [skip ci] --- docs/TroubleshootingGuide.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/TroubleshootingGuide.md b/docs/TroubleshootingGuide.md index 84f157095..d65d1ca44 100644 --- a/docs/TroubleshootingGuide.md +++ b/docs/TroubleshootingGuide.md @@ -7,21 +7,21 @@ Use this Troubleshooting Guide to: - Replace accidentally removed cassandra node or os-disk with existing data disk ## Replace accidentally removed node or os-disk -The following process helps to replace accidentally terminated node or os-disk, but that node should not have an additional data disk. +The following process helps to replace accidentally terminated node or os-disk, but the node should not have an additional data disk. Note: Mainly applicable for scalardl, envoy, cassy, reaper, monitor and ca nodes. Please try the following -- Remove the os-disk If the node is not available in the resource group. -- Remove the node If the os-disk is not available in the resource group. -- Then you can follow [Node Replacement](NodeReplacement.md) +* Delete the os-disk If the node is terminated. +* Terminate the node If the os-disk is deleted. +* Follow [Node Replacement](NodeReplacement.md) ## Replace accidentally removed cassandra node or os-disk with existing data disk This documentation helps to replace accidentally terminated node or os-disk with existing data disk (taint volume attachment). Please try the following -* Remove the os-disk If the node is not available in the resource group. -* Remove the node If the os-disk is not available in the resource group. +* Delete the os-disk If the node is terminated. +* Terminate the node If the os-disk is deleted. * Do `terraform state rm` as follows. ```console @@ -31,4 +31,4 @@ terraform state rm "module.cassandra.azurerm_virtual_machine_data_disk_attachmen terraform apply ``` -Then you can follow [Cassandra Post Recovery Steps](CassandraOperation.md#post-recovery-steps) \ No newline at end of file +Follow [Cassandra Post Recovery Steps](CassandraOperation.md#post-recovery-steps) \ No newline at end of file From 7e40ef5869fcb8137abd04608056b8d70a22f40c Mon Sep 17 00:00:00 2001 From: scalar-boney Date: Mon, 14 Sep 2020 14:59:30 +0900 Subject: [PATCH 03/10] troubleshoot doc modified [skip ci] --- docs/TroubleshootingGuide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/TroubleshootingGuide.md b/docs/TroubleshootingGuide.md index d65d1ca44..065e17474 100644 --- a/docs/TroubleshootingGuide.md +++ b/docs/TroubleshootingGuide.md @@ -25,7 +25,7 @@ Please try the following * Do `terraform state rm` as follows. ```console -terraform taint "module.cassandra.module.cassandra_cluster.azurerm_virtual_machine.vm-linux[0]" +terraform state rm "module.cassandra.module.cassandra_cluster.azurerm_virtual_machine.vm-linux[0]" terraform state rm "module.cassandra.azurerm_virtual_machine_data_disk_attachment.cassandra_data_volume_attachment[0]" terraform apply From 3121a72ec1087645c4c24c09dc19285e5065171a Mon Sep 17 00:00:00 2001 From: scalar-boney <58680313+scalar-boney@users.noreply.github.com> Date: Tue, 15 Sep 2020 10:02:12 +0900 Subject: [PATCH 04/10] Update docs/TroubleshootingGuide.md Co-authored-by: Hiroyuki Yamada --- docs/TroubleshootingGuide.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/TroubleshootingGuide.md b/docs/TroubleshootingGuide.md index 065e17474..b77c16aeb 100644 --- a/docs/TroubleshootingGuide.md +++ b/docs/TroubleshootingGuide.md @@ -1,4 +1,4 @@ -# Troubleshoot Guide +# Troubleshooting Guide This guide explains how to replace a node when the node cannot be replaced with normal procedures. This is especially useful when the node or os-disk is accidentally terminated in the **Azure** environment. @@ -31,4 +31,4 @@ terraform state rm "module.cassandra.azurerm_virtual_machine_data_disk_attachmen terraform apply ``` -Follow [Cassandra Post Recovery Steps](CassandraOperation.md#post-recovery-steps) \ No newline at end of file +Follow [Cassandra Post Recovery Steps](CassandraOperation.md#post-recovery-steps) From c2c74d88e0b550de12782eb356dc3ddd338ac9a9 Mon Sep 17 00:00:00 2001 From: scalar-boney Date: Tue, 15 Sep 2020 20:35:22 +0900 Subject: [PATCH 05/10] review modification [skip ci] --- docs/TroubleshootingGuide.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/TroubleshootingGuide.md b/docs/TroubleshootingGuide.md index b77c16aeb..39d7ab36f 100644 --- a/docs/TroubleshootingGuide.md +++ b/docs/TroubleshootingGuide.md @@ -1,12 +1,16 @@ # Troubleshooting Guide -This guide explains how to replace a node when the node cannot be replaced with normal procedures. This is especially useful when the node or os-disk is accidentally terminated in the **Azure** environment. +This is a guide for troubleshooting scalar-terraform environment. Use this Troubleshooting Guide to: +- Node Replacement + +## Node Replacement +These troubleshooting steps can be used when the node cannot be replaced with normal procedures. This is especially useful when the node or os-disk is accidentally terminated in the **Azure** environment. - Replace accidentally removed node or os-disk - Replace accidentally removed cassandra node or os-disk with existing data disk -## Replace accidentally removed node or os-disk +### Replace accidentally removed node or os-disk The following process helps to replace accidentally terminated node or os-disk, but the node should not have an additional data disk. Note: Mainly applicable for scalardl, envoy, cassy, reaper, monitor and ca nodes. @@ -16,7 +20,7 @@ Please try the following * Terminate the node If the os-disk is deleted. * Follow [Node Replacement](NodeReplacement.md) -## Replace accidentally removed cassandra node or os-disk with existing data disk +### Replace accidentally removed cassandra node or os-disk with existing data disk This documentation helps to replace accidentally terminated node or os-disk with existing data disk (taint volume attachment). Please try the following From 449a8db385c01fb5f0a4cce25b0544fccf530528 Mon Sep 17 00:00:00 2001 From: scalar-boney <58680313+scalar-boney@users.noreply.github.com> Date: Wed, 16 Sep 2020 14:39:48 +0900 Subject: [PATCH 06/10] Update docs/TroubleshootingGuide.md Co-authored-by: Hiroyuki Yamada --- docs/TroubleshootingGuide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/TroubleshootingGuide.md b/docs/TroubleshootingGuide.md index 39d7ab36f..f46f480bd 100644 --- a/docs/TroubleshootingGuide.md +++ b/docs/TroubleshootingGuide.md @@ -5,7 +5,7 @@ This is a guide for troubleshooting scalar-terraform environment. Use this Troubleshooting Guide to: - Node Replacement -## Node Replacement +## Accidental deletion of resources These troubleshooting steps can be used when the node cannot be replaced with normal procedures. This is especially useful when the node or os-disk is accidentally terminated in the **Azure** environment. - Replace accidentally removed node or os-disk - Replace accidentally removed cassandra node or os-disk with existing data disk From 19bdb6724003364d6387f1e529d58b90d8802c9f Mon Sep 17 00:00:00 2001 From: scalar-boney Date: Wed, 16 Sep 2020 22:42:22 +0900 Subject: [PATCH 07/10] modified based on review --- docs/TroubleshootingGuide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/TroubleshootingGuide.md b/docs/TroubleshootingGuide.md index f46f480bd..3a9f9647a 100644 --- a/docs/TroubleshootingGuide.md +++ b/docs/TroubleshootingGuide.md @@ -3,7 +3,7 @@ This is a guide for troubleshooting scalar-terraform environment. Use this Troubleshooting Guide to: -- Node Replacement +- Accidental deletion of resources ## Accidental deletion of resources These troubleshooting steps can be used when the node cannot be replaced with normal procedures. This is especially useful when the node or os-disk is accidentally terminated in the **Azure** environment. From 17c02a34b5eb673262d828256da2809a43d5f850 Mon Sep 17 00:00:00 2001 From: scalar-boney Date: Fri, 18 Sep 2020 09:49:04 +0900 Subject: [PATCH 08/10] document modification [skip ci] --- docs/TroubleshootingGuide.md | 22 ++++++++-------------- 1 file changed, 8 insertions(+), 14 deletions(-) diff --git a/docs/TroubleshootingGuide.md b/docs/TroubleshootingGuide.md index 3a9f9647a..05489e79c 100644 --- a/docs/TroubleshootingGuide.md +++ b/docs/TroubleshootingGuide.md @@ -2,30 +2,24 @@ This is a guide for troubleshooting scalar-terraform environment. -Use this Troubleshooting Guide to: -- Accidental deletion of resources - ## Accidental deletion of resources -These troubleshooting steps can be used when the node cannot be replaced with normal procedures. This is especially useful when the node or os-disk is accidentally terminated in the **Azure** environment. -- Replace accidentally removed node or os-disk -- Replace accidentally removed cassandra node or os-disk with existing data disk +When you accidentally delete a resource manually without terraform, it causes some inconsistencies between the actual state of resources and the state that terraform knows. Thus, you might need to take some extra actions to recover from such situations depending on the Cloud you use. The following explains how to recover from such cases. + +- Recover from accidental deletion of a node +- Recover a node with existing data disk from accidental deletion of a node -### Replace accidentally removed node or os-disk -The following process helps to replace accidentally terminated node or os-disk, but the node should not have an additional data disk. +### Recover from accidental deletion of a node +If you accidentally delete a node that does not have an additional data disk, you can recover it in the following steps. It is mainly applicable for scalardl, envoy, cassy, reaper, monitor and ca nodes. -Note: Mainly applicable for scalardl, envoy, cassy, reaper, monitor and ca nodes. - Please try the following * Delete the os-disk If the node is terminated. -* Terminate the node If the os-disk is deleted. * Follow [Node Replacement](NodeReplacement.md) -### Replace accidentally removed cassandra node or os-disk with existing data disk -This documentation helps to replace accidentally terminated node or os-disk with existing data disk (taint volume attachment). +### Recover a node with existing data disk from accidental deletion of a node +If you accidentally delete a node that contains an additional data disk, you can recover that node with an existing data disk using the following steps. Please try the following * Delete the os-disk If the node is terminated. -* Terminate the node If the os-disk is deleted. * Do `terraform state rm` as follows. ```console From 0284a8fb40e2fba9700bf599171bb0d14eed90c7 Mon Sep 17 00:00:00 2001 From: scalar-boney Date: Wed, 23 Sep 2020 15:08:21 +0900 Subject: [PATCH 09/10] review modification [skip ci] --- docs/TroubleshootingGuide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/TroubleshootingGuide.md b/docs/TroubleshootingGuide.md index 05489e79c..bfc3a05c8 100644 --- a/docs/TroubleshootingGuide.md +++ b/docs/TroubleshootingGuide.md @@ -3,7 +3,7 @@ This is a guide for troubleshooting scalar-terraform environment. ## Accidental deletion of resources -When you accidentally delete a resource manually without terraform, it causes some inconsistencies between the actual state of resources and the state that terraform knows. Thus, you might need to take some extra actions to recover from such situations depending on the Cloud you use. The following explains how to recover from such cases. +When you accidentally delete a resource manually without terraform, it causes some inconsistencies between the actual state of resources and the state that terraform knows. Thus, you might need to take some extra actions to recover from such situations depending on the Cloud you use. The following explains how to recover from such cases in the AZURE scalar-terraform environment. - Recover from accidental deletion of a node - Recover a node with existing data disk from accidental deletion of a node From 920e858d348137a1bde740a9226ebd645d618225 Mon Sep 17 00:00:00 2001 From: scalar-boney Date: Wed, 23 Sep 2020 15:32:22 +0900 Subject: [PATCH 10/10] Review suggestion added [skip ci] --- docs/TroubleshootingGuide.md | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/docs/TroubleshootingGuide.md b/docs/TroubleshootingGuide.md index bfc3a05c8..fa77af768 100644 --- a/docs/TroubleshootingGuide.md +++ b/docs/TroubleshootingGuide.md @@ -3,20 +3,17 @@ This is a guide for troubleshooting scalar-terraform environment. ## Accidental deletion of resources -When you accidentally delete a resource manually without terraform, it causes some inconsistencies between the actual state of resources and the state that terraform knows. Thus, you might need to take some extra actions to recover from such situations depending on the Cloud you use. The following explains how to recover from such cases in the AZURE scalar-terraform environment. +When you accidentally delete a resource manually without terraform, it causes some inconsistencies between the actual state of resources and the state that terraform knows. Thus, you might need to take some extra actions to recover from such situations depending on the Cloud you use. -- Recover from accidental deletion of a node -- Recover a node with existing data disk from accidental deletion of a node - -### Recover from accidental deletion of a node -If you accidentally delete a node that does not have an additional data disk, you can recover it in the following steps. It is mainly applicable for scalardl, envoy, cassy, reaper, monitor and ca nodes. +### Recover from accidental deletion of a node in Azure +If you accidentally delete a node that does not have an additional data disk in Azure, you can recover it in the following steps. It is mainly applicable for scalardl, envoy, cassy, reaper, monitor and ca nodes. Please try the following * Delete the os-disk If the node is terminated. * Follow [Node Replacement](NodeReplacement.md) -### Recover a node with existing data disk from accidental deletion of a node -If you accidentally delete a node that contains an additional data disk, you can recover that node with an existing data disk using the following steps. +### Recover a node with existing data disk from accidental deletion of a node in Azure +If you accidentally delete a node that contains an additional data disk in Azure, you can recover that node with an existing data disk using the following steps. Please try the following * Delete the os-disk If the node is terminated.