From c0ee9501b12292354ffdb5d03529fdc3d6509d78 Mon Sep 17 00:00:00 2001 From: Hiroyuki Yamada Date: Wed, 16 Sep 2020 15:19:02 +0900 Subject: [PATCH 1/8] update --- docs/CassandraOperation.md | 72 ++++++++++++++++++++++---------------- 1 file changed, 42 insertions(+), 30 deletions(-) diff --git a/docs/CassandraOperation.md b/docs/CassandraOperation.md index 761af8833..d05a01462 100644 --- a/docs/CassandraOperation.md +++ b/docs/CassandraOperation.md @@ -1,7 +1,7 @@ -## Cassandra +# Cassandra Operations The Cassandra cluster can be expanded using terraform but due to the sensitive nature of Cassandra it requires additional steps. -### Scale up the Cassandra cluster +## Scale up the Cassandra cluster By setting the `cassandra.resource_count` variable you can control the number of Cassandra nodes to create. All Cassandra nodes will be created in a stopped state and will need an operator to manually start the service. [ [Azure example.tfvars](../examples/azure/cassandra/example.tfvars) ] @@ -21,7 +21,7 @@ cassandra = { * It is only possible to add 1 node to a cluster at a time. * Also when adding a new node to a cluster it will slow down the system. -### Replace a Cassandra Node +## Replace a Cassandra Node The following section is useful when you need to replace a Cassandra node. * The first thing to do is to match the Terraform state file to the actual resource you want to replace. @@ -29,7 +29,7 @@ The following section is useful when you need to replace a Cassandra node. * Match the private_ip address of the VM to the state file using the command below. * You may want to add `-A 10` to the grep command to help find the correct node. -#### Azure Output +### Azure Output ```console terraform show | grep module.cassandra # module.cassandra.module.reaper_cluster.azurerm_virtual_machine.vm-linux[0]: @@ -45,7 +45,7 @@ terraform show | grep module.cassandra # module.cassandra.null_resource.volume_data_local[0]: ``` -#### AWS Output +### AWS Output ```console terraform show | grep module.cassandra # module.cassandra.module.cassandra_cluster.aws_instance.this[0]: @@ -62,17 +62,17 @@ terraform show | grep module.cassandra * Find the `instance` or `vm` you wish to replace and copy the line. * You also need to decide to taint either the *drive attachment* or the *volume* directly. -#### Taint volume attachment +### Taint volume attachment When you taint the volume attachment terraform will try to attach the same data or commit log volume to the new instance. This is the ideal situation as it is the quickest way to replace a node. -##### Azure +#### Azure ```console terraform taint "module.cassandra.module.cassandra_cluster.azurerm_virtual_machine.vm-linux[0]" terraform taint "module.cassandra.azurerm_virtual_machine_data_disk_attachment.cassandra_data_volume_attachment[0]" terraform apply ``` -##### AWS +#### AWS ```console terraform taint "module.cassandra.module.cassandra_cluster.aws_instance.this[0]" terraform taint "module.cassandra.aws_volume_attachment.cassandra_data_volume_attachment[0]" @@ -80,10 +80,10 @@ terraform taint "module.cassandra.aws_volume_attachment.cassandra_data_volume_at terraform apply ``` -#### Taint Volume +### Taint Volume The other option is to taint the volume which should be used as a last resort. This *will permanently delete data* on that volume. Be sure you can recover the data from a backup first. -##### Azure +#### Azure ```console terraform taint "module.cassandra.module.cassandra_cluster.azurerm_virtual_machine.vm-linux[0]" terraform taint "module.cassandra.azurerm_managed_disk.cassandra_data_volume[0]" @@ -91,7 +91,7 @@ terraform taint "module.cassandra.azurerm_managed_disk.cassandra_data_volume[0]" terraform apply ``` -##### AWS +#### AWS ```console terraform taint "module.cassandra.module.cassandra_cluster.aws_instance.this[0]" terraform taint "module.cassandra.aws_ebs_volume.cassandra_data_volume[0]" @@ -119,25 +119,37 @@ sudo /bin/bash -c "echo 'UUID=af767839-b23d-4d1f-8a19-debbcfd6413c /data xfs def sudo mount -a ``` -#### Modify Cassandra config -You need to modify the file `/etc/cassandra/conf/cassandra-env.sh` to add the line below. - -* The IP address should be the same as the node you are replacing, even if the IP happens to be the same as the replacement node. - -``` -JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address_first_boot=" -``` - -You also need to modify the file `/etc/cassandra/conf/cassandra.yaml` if you replaced a seed node and a volume. -* Remove the newly created Cassandra node IP from the seeds of the node. - -Finally you can start the Cassandra service. - -```console -sudo systemctl start cassandra -``` - -If you replaced a seed node (old seed node), you should replace the old seed node's IP of all Cassandra nodes with newly created Cassandra node IP. +### Restart Cassandra +As a last step of Cassandra node replacement, you need to configure Cassandra appropriately before start it. What needs to be done depends on the following cases. + +* Case1: Node (Instance/VM) is repalced (but the volume is not replaced) +* Case2: Volume is repalced (but the node is not replaced) +* Case3: Node and the volume are both replaced + +Here are the steps for each case. + +#### Case1: Node (Instance/VM) is repalced (but the volume is not replaced) +* Remove the IP of the new node from seeds in `casssandra.yaml` +* Add `JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address_first_boot="` to the bottom of `cassandra-env.sh` +* `sudo systemctl start cassandra` +* Repair with Reaper at some later point + +#### Case2: Volume is repalced (but the node is not replaced) +* Restore data from Cassy +* `sudo systemctl start cassandra` +* Repair with Reaper at some later point + +#### Case3: Node and the volume are both replaced +* Restore data from Cassy + * You need to manually update the IP of an entry of `backup_history` table to match with the new IP +* Remove the IP of the new node from seeds in `casssandra.yaml` +* Set `auto_bootstrap` to `false` in `cassandra.yaml` + * this might not be needed. need to verify. +* Add `JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address_first_boot="` to the bottom of `cassandra-env.sh` +* sudo systemctl start cassandra +* Repair with Reaper at some later point + +Please note that, if you replace a seed node (IP-A), you should replace IP-A from seeds with a newly created Cassandra node IP in all the Cassandra nodes. # Related Documents From 60795a779c680df6830fa7e6d8000d79830579c8 Mon Sep 17 00:00:00 2001 From: Hiroyuki Yamada Date: Wed, 16 Sep 2020 15:25:53 +0900 Subject: [PATCH 2/8] Update docs/CassandraOperation.md --- docs/CassandraOperation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/CassandraOperation.md b/docs/CassandraOperation.md index d05a01462..a1be02acf 100644 --- a/docs/CassandraOperation.md +++ b/docs/CassandraOperation.md @@ -120,7 +120,7 @@ sudo mount -a ``` ### Restart Cassandra -As a last step of Cassandra node replacement, you need to configure Cassandra appropriately before start it. What needs to be done depends on the following cases. +As the last step of Cassandra node replacement, you need to configure Cassandra appropriately before starting it. What needs to be done depends on the following cases. * Case1: Node (Instance/VM) is repalced (but the volume is not replaced) * Case2: Volume is repalced (but the node is not replaced) From a6d8adb01793e001c93d6b3c277400e69cfb519a Mon Sep 17 00:00:00 2001 From: Hiroyuki Yamada Date: Wed, 16 Sep 2020 15:26:00 +0900 Subject: [PATCH 3/8] Update docs/CassandraOperation.md --- docs/CassandraOperation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/CassandraOperation.md b/docs/CassandraOperation.md index a1be02acf..90803c139 100644 --- a/docs/CassandraOperation.md +++ b/docs/CassandraOperation.md @@ -122,7 +122,7 @@ sudo mount -a ### Restart Cassandra As the last step of Cassandra node replacement, you need to configure Cassandra appropriately before starting it. What needs to be done depends on the following cases. -* Case1: Node (Instance/VM) is repalced (but the volume is not replaced) +* Case1: Node (Instance/VM) is replaced (but the volume is not replaced) * Case2: Volume is repalced (but the node is not replaced) * Case3: Node and the volume are both replaced From e7038d8db5ac32a71fdbf4de982e4de3b12e54f5 Mon Sep 17 00:00:00 2001 From: Hiroyuki Yamada Date: Wed, 16 Sep 2020 15:26:07 +0900 Subject: [PATCH 4/8] Update docs/CassandraOperation.md --- docs/CassandraOperation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/CassandraOperation.md b/docs/CassandraOperation.md index 90803c139..f17260212 100644 --- a/docs/CassandraOperation.md +++ b/docs/CassandraOperation.md @@ -123,7 +123,7 @@ sudo mount -a As the last step of Cassandra node replacement, you need to configure Cassandra appropriately before starting it. What needs to be done depends on the following cases. * Case1: Node (Instance/VM) is replaced (but the volume is not replaced) -* Case2: Volume is repalced (but the node is not replaced) +* Case2: Volume is replaced (but the node is not replaced) * Case3: Node and the volume are both replaced Here are the steps for each case. From 9402c849720e02f9aa8f8105d0cd15acca9db790 Mon Sep 17 00:00:00 2001 From: Hiroyuki Yamada Date: Wed, 16 Sep 2020 15:26:12 +0900 Subject: [PATCH 5/8] Update docs/CassandraOperation.md --- docs/CassandraOperation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/CassandraOperation.md b/docs/CassandraOperation.md index f17260212..d9f5ff4f9 100644 --- a/docs/CassandraOperation.md +++ b/docs/CassandraOperation.md @@ -128,7 +128,7 @@ As the last step of Cassandra node replacement, you need to configure Cassandra Here are the steps for each case. -#### Case1: Node (Instance/VM) is repalced (but the volume is not replaced) +#### Case1: Node (Instance/VM) is replaced (but the volume is not replaced) * Remove the IP of the new node from seeds in `casssandra.yaml` * Add `JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address_first_boot="` to the bottom of `cassandra-env.sh` * `sudo systemctl start cassandra` From 7a6db5c3336693b92e9136923604f7369564ec6d Mon Sep 17 00:00:00 2001 From: Hiroyuki Yamada Date: Wed, 16 Sep 2020 15:26:18 +0900 Subject: [PATCH 6/8] Update docs/CassandraOperation.md --- docs/CassandraOperation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/CassandraOperation.md b/docs/CassandraOperation.md index d9f5ff4f9..9ffc36005 100644 --- a/docs/CassandraOperation.md +++ b/docs/CassandraOperation.md @@ -134,7 +134,7 @@ Here are the steps for each case. * `sudo systemctl start cassandra` * Repair with Reaper at some later point -#### Case2: Volume is repalced (but the node is not replaced) +#### Case2: Volume is replaced (but the node is not replaced) * Restore data from Cassy * `sudo systemctl start cassandra` * Repair with Reaper at some later point From c57785fec924e48086eb1ea04a7309e36f9427dc Mon Sep 17 00:00:00 2001 From: Hiroyuki Yamada Date: Wed, 16 Sep 2020 15:26:23 +0900 Subject: [PATCH 7/8] Update docs/CassandraOperation.md --- docs/CassandraOperation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/CassandraOperation.md b/docs/CassandraOperation.md index 9ffc36005..712bcb16d 100644 --- a/docs/CassandraOperation.md +++ b/docs/CassandraOperation.md @@ -141,7 +141,7 @@ Here are the steps for each case. #### Case3: Node and the volume are both replaced * Restore data from Cassy - * You need to manually update the IP of an entry of `backup_history` table to match with the new IP + * You need to manually update the IP of the entry of `backup_history` table to match with the new IP * Remove the IP of the new node from seeds in `casssandra.yaml` * Set `auto_bootstrap` to `false` in `cassandra.yaml` * this might not be needed. need to verify. From 85c314bcb9f6ac4f6c500647866a8123b3576ab1 Mon Sep 17 00:00:00 2001 From: Hiroyuki Yamada Date: Thu, 17 Sep 2020 13:19:34 +0900 Subject: [PATCH 8/8] Update docs/CassandraOperation.md --- docs/CassandraOperation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/CassandraOperation.md b/docs/CassandraOperation.md index 712bcb16d..bdf8b516c 100644 --- a/docs/CassandraOperation.md +++ b/docs/CassandraOperation.md @@ -146,7 +146,7 @@ Here are the steps for each case. * Set `auto_bootstrap` to `false` in `cassandra.yaml` * this might not be needed. need to verify. * Add `JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address_first_boot="` to the bottom of `cassandra-env.sh` -* sudo systemctl start cassandra +* `sudo systemctl start cassandra` * Repair with Reaper at some later point Please note that, if you replace a seed node (IP-A), you should replace IP-A from seeds with a newly created Cassandra node IP in all the Cassandra nodes.