From 3b3ddf21ce1505dae24fabd4fa53e46f1a4e1a43 Mon Sep 17 00:00:00 2001 From: despire Date: Tue, 7 May 2024 12:28:33 +0200 Subject: [PATCH 1/7] update backup method --- .../creating-claudie-backup.md | 105 +++++++++++++++++- .../crd/claudie.io_inputmanifests.yaml | 2 + 2 files changed, 106 insertions(+), 1 deletion(-) diff --git a/docs/creating-claudie-backup/creating-claudie-backup.md b/docs/creating-claudie-backup/creating-claudie-backup.md index 22f64463b..be3d0efff 100644 --- a/docs/creating-claudie-backup/creating-claudie-backup.md +++ b/docs/creating-claudie-backup/creating-claudie-backup.md @@ -16,6 +16,109 @@ These are the only services that will have a PVC attached to it, the other are s ## Backing up Claudie +### Manual backup + +Claudie is already deployed on an existing Management Cluster and at least 1 Input Manifest has been applied. + +Create a directory where the backup of the state will be stored. + +```bash +mkdir claudie-backup +``` + +Put your Claudie inputmanifests into the created folder. + +We will now back up the state of the respective input manifests from MongoDB and Minio. + +```bash +kubectl get pods -n claudie + +NAME READY STATUS RESTARTS AGE +ansibler-6f4557cf74-b4dts 1/1 Running 0 18m +builder-5d68987c86-qdfd5 1/1 Running 0 18m +claudie-operator-6d9ddc7f8b-hv84c 1/1 Running 0 18m +context-box-5d75bfffc6-d9qfm 1/1 Running 0 18m +create-table-job-ghb9f 0/1 Completed 1 18m +dynamodb-6d65df988-c626j 1/1 Running 0 18m +kube-eleven-556cfdfd98-jq6hl 1/1 Running 0 18m +kuber-7f8cd4cd89-6ds2w 1/1 Running 0 18m +make-bucket-job-9mjft 0/1 Completed 0 18m +minio-0 1/1 Running 0 18m +minio-1 1/1 Running 0 18m +minio-2 1/1 Running 0 18m +minio-3 1/1 Running 0 18m +mongodb-6ccb5f5dff-ptdw2 1/1 Running 0 18m +scheduler-676bd4468f-6fjn8 1/1 Running 0 18m +terraformer-66c6f67d98-pwr9t 1/1 Running 0 18m +``` + +To backup state from MongoDB execute the following command + +```bash +kubectl exec -n claudie mongodb-6ccb5f5dff-ptdw2 -- sh -c 'mongoexport --uri=mongodb://$MONGO_INITDB_ROOT_USERNAME:$MONGO_INITDB_ROOT_PASSWORD@localhost:27017/claudie -c inputManifests --authenticationDatabase admin' > claudie-backup/inputManifests +``` + +Next we need to backup the state from Minio. Port-forward the minio service so that it is accessible from localhost. + +```bash +kubectl port-forward -n claudie svc/minio 9000:9000 +``` + +Setup an alias for the [mc](https://min.io/docs/minio/linux/reference/minio-mc.html) command line tool. + +```bash +mc alias set claudie-minio http://127.0.0.1:9000 +``` + +!!! note "Provide the access and secret key for minio. The default can be found in the github [repository] (https://github.com/berops/claudie/tree/master/manifests/claudie/minio/secrets). If you have not changed them, we strongly encourage you to do so!" + +Download the state into the backup folder + +```bash +mc mirror claudie-minio/claudie-tf-state-files ./claudie-backup +``` + +You now have everything you need to restore your input manifests to a new management cluster. + +!!! warning "These files will contain your credentials, DO NOT STORE THEM OUT IN THE PUBLIC" + +To restore the state on your new management cluster you can follow these commands. We expect that your default `kubeconfig` points to the new Management Cluster, if it does not, you can override it in the following commands using `--kubeconfig ./path-to-config`. + +Copy the collection into the MongoDB pod. + +```bash +kubectl cp ./claudie-backup/inputManifests mongodb-6ccb5f5dff-jjvz2:/tmp/inputManifests -n claudie +``` + +Import the state to MongoDB. + +```bash +kubectl exec -n claudie mongodb-6ccb5f5dff-jjvz2 -- sh -c 'mongoimport --uri=mongodb://$MONGO_INITDB_ROOT_USERNAME:$MONGO_INITDB_ROOT_PASSWORD@localhost:27017/claudie -c inputManifests --authenticationDatabase admin --file /tmp/inputManifests' +``` + +!!! note "Don't forget to delete the `/tmp/inputManifests` file" + +Import MinIO state. + +```bash +mc cp --recursive ./claudie-backup/ claudie-minio/claudie-tf-state-files +``` + +You can now apply your Claudie inputmanifests which will be immediately in the `DONE` stage. You can verify this with + +```bash +kubectl get inputmanifests -A +``` + +Now you can make any new changes to your inputmanifests on the new management cluster and the state will be reused. + +!!! note "Alternatively you may also use any GUI clients for MongoDB and Minio for more straightforward backup of the state. All you need to backup is the bucket `claudie-tf-state-files` in MinIO and the collection `inputManifests` from MongoDB" + + +### Using Velero + +!!! warning "Velero does not support HostPath volumes. If the PVCs in your management cluster are attached to such volumes, the backup will not work. In this case, use the above backup method." + All resources that are deployed or created by Claudie can be identified with the following label: ``` @@ -83,4 +186,4 @@ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/ velero restore create --from-backup claudie-backup ``` -Once all resources are restored, you should be able to deploy new input manifests and also modify existing infrastructure without any problems. \ No newline at end of file +Once all resources are restored, you should be able to deploy new input manifests and also modify existing infrastructure without any problems. diff --git a/manifests/claudie/crd/claudie.io_inputmanifests.yaml b/manifests/claudie/crd/claudie.io_inputmanifests.yaml index d0057e3eb..9457f765d 100644 --- a/manifests/claudie/crd/claudie.io_inputmanifests.yaml +++ b/manifests/claudie/crd/claudie.io_inputmanifests.yaml @@ -6,6 +6,8 @@ metadata: controller-gen.kubebuilder.io/version: v0.11.3 creationTimestamp: null name: inputmanifests.claudie.io + labels: + app.kubernetes.io/part-of: claudie spec: group: claudie.io names: From e1d4cfbea647218c53640276638cf47023a63cb4 Mon Sep 17 00:00:00 2001 From: despire Date: Tue, 7 May 2024 12:37:19 +0200 Subject: [PATCH 2/7] update backup --- docs/creating-claudie-backup/creating-claudie-backup.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/creating-claudie-backup/creating-claudie-backup.md b/docs/creating-claudie-backup/creating-claudie-backup.md index be3d0efff..29382ec9b 100644 --- a/docs/creating-claudie-backup/creating-claudie-backup.md +++ b/docs/creating-claudie-backup/creating-claudie-backup.md @@ -28,7 +28,7 @@ mkdir claudie-backup Put your Claudie inputmanifests into the created folder. -We will now back up the state of the respective input manifests from MongoDB and Minio. +We will now back up the state of the respective input manifests from MongoDB and MinIO. ```bash kubectl get pods -n claudie @@ -58,7 +58,7 @@ To backup state from MongoDB execute the following command kubectl exec -n claudie mongodb-6ccb5f5dff-ptdw2 -- sh -c 'mongoexport --uri=mongodb://$MONGO_INITDB_ROOT_USERNAME:$MONGO_INITDB_ROOT_PASSWORD@localhost:27017/claudie -c inputManifests --authenticationDatabase admin' > claudie-backup/inputManifests ``` -Next we need to backup the state from Minio. Port-forward the minio service so that it is accessible from localhost. +Next we need to backup the state from MinIO. Port-forward the MinIO service so that it is accessible from localhost. ```bash kubectl port-forward -n claudie svc/minio 9000:9000 @@ -98,7 +98,7 @@ kubectl exec -n claudie mongodb-6ccb5f5dff-jjvz2 -- sh -c 'mongoimport --uri=mon !!! note "Don't forget to delete the `/tmp/inputManifests` file" -Import MinIO state. +Port-forward the MinIO server and import the backed up state. ```bash mc cp --recursive ./claudie-backup/ claudie-minio/claudie-tf-state-files From 5ab3f2de52f727f3485a64983f773ab380ea5091 Mon Sep 17 00:00:00 2001 From: despire Date: Tue, 7 May 2024 12:38:13 +0200 Subject: [PATCH 3/7] fix typo --- docs/creating-claudie-backup/creating-claudie-backup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/creating-claudie-backup/creating-claudie-backup.md b/docs/creating-claudie-backup/creating-claudie-backup.md index 29382ec9b..4217c77ce 100644 --- a/docs/creating-claudie-backup/creating-claudie-backup.md +++ b/docs/creating-claudie-backup/creating-claudie-backup.md @@ -98,7 +98,7 @@ kubectl exec -n claudie mongodb-6ccb5f5dff-jjvz2 -- sh -c 'mongoimport --uri=mon !!! note "Don't forget to delete the `/tmp/inputManifests` file" -Port-forward the MinIO server and import the backed up state. +Port-forward the MinIO service and import the backed up state. ```bash mc cp --recursive ./claudie-backup/ claudie-minio/claudie-tf-state-files From 99d7a06c1c9a60f4beb11dab783afe76e0962f41 Mon Sep 17 00:00:00 2001 From: despire Date: Tue, 7 May 2024 15:28:19 +0200 Subject: [PATCH 4/7] update docs --- docs/creating-claudie-backup/creating-claudie-backup.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/creating-claudie-backup/creating-claudie-backup.md b/docs/creating-claudie-backup/creating-claudie-backup.md index 4217c77ce..cb69e3169 100644 --- a/docs/creating-claudie-backup/creating-claudie-backup.md +++ b/docs/creating-claudie-backup/creating-claudie-backup.md @@ -70,7 +70,7 @@ Setup an alias for the [mc](https://min.io/docs/minio/linux/reference/minio-mc.h mc alias set claudie-minio http://127.0.0.1:9000 ``` -!!! note "Provide the access and secret key for minio. The default can be found in the github [repository] (https://github.com/berops/claudie/tree/master/manifests/claudie/minio/secrets). If you have not changed them, we strongly encourage you to do so!" +!!! note "Provide the access and secret key for minio. The default can be found in the github repository in the `manifests/claudie/minio/secrets` folder. If you have not changed them, we strongly encourage you to do so!" Download the state into the backup folder @@ -80,7 +80,7 @@ mc mirror claudie-minio/claudie-tf-state-files ./claudie-backup You now have everything you need to restore your input manifests to a new management cluster. -!!! warning "These files will contain your credentials, DO NOT STORE THEM OUT IN THE PUBLIC" +!!! warning "These files will contain your credentials, DO NOT STORE THEM OUT IN THE PUBLIC!" To restore the state on your new management cluster you can follow these commands. We expect that your default `kubeconfig` points to the new Management Cluster, if it does not, you can override it in the following commands using `--kubeconfig ./path-to-config`. From 58ce0845929ac883f6c38f880b965a51a79ef328 Mon Sep 17 00:00:00 2001 From: despire Date: Tue, 7 May 2024 16:36:42 +0200 Subject: [PATCH 5/7] add feedback --- .../creating-claudie-backup.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/creating-claudie-backup/creating-claudie-backup.md b/docs/creating-claudie-backup/creating-claudie-backup.md index cb69e3169..7d3836d90 100644 --- a/docs/creating-claudie-backup/creating-claudie-backup.md +++ b/docs/creating-claudie-backup/creating-claudie-backup.md @@ -55,7 +55,7 @@ terraformer-66c6f67d98-pwr9t 1/1 Running 0 18m To backup state from MongoDB execute the following command ```bash -kubectl exec -n claudie mongodb-6ccb5f5dff-ptdw2 -- sh -c 'mongoexport --uri=mongodb://$MONGO_INITDB_ROOT_USERNAME:$MONGO_INITDB_ROOT_PASSWORD@localhost:27017/claudie -c inputManifests --authenticationDatabase admin' > claudie-backup/inputManifests +kubectl exec -n claudie mongodb- -- sh -c 'mongoexport --uri=mongodb://$MONGO_INITDB_ROOT_USERNAME:$MONGO_INITDB_ROOT_PASSWORD@localhost:27017/claudie -c inputManifests --authenticationDatabase admin' > claudie-backup/inputManifests ``` Next we need to backup the state from MinIO. Port-forward the MinIO service so that it is accessible from localhost. @@ -87,13 +87,13 @@ To restore the state on your new management cluster you can follow these command Copy the collection into the MongoDB pod. ```bash -kubectl cp ./claudie-backup/inputManifests mongodb-6ccb5f5dff-jjvz2:/tmp/inputManifests -n claudie +kubectl cp ./claudie-backup/inputManifests mongodb-:/tmp/inputManifests -n claudie ``` Import the state to MongoDB. ```bash -kubectl exec -n claudie mongodb-6ccb5f5dff-jjvz2 -- sh -c 'mongoimport --uri=mongodb://$MONGO_INITDB_ROOT_USERNAME:$MONGO_INITDB_ROOT_PASSWORD@localhost:27017/claudie -c inputManifests --authenticationDatabase admin --file /tmp/inputManifests' +kubectl exec -n claudie mongodb- -- sh -c 'mongoimport --uri=mongodb://$MONGO_INITDB_ROOT_USERNAME:$MONGO_INITDB_ROOT_PASSWORD@localhost:27017/claudie -c inputManifests --authenticationDatabase admin --file /tmp/inputManifests' ``` !!! note "Don't forget to delete the `/tmp/inputManifests` file" @@ -101,7 +101,7 @@ kubectl exec -n claudie mongodb-6ccb5f5dff-jjvz2 -- sh -c 'mongoimport --uri=mon Port-forward the MinIO service and import the backed up state. ```bash -mc cp --recursive ./claudie-backup/ claudie-minio/claudie-tf-state-files +mc cp --recursive ./claudie-backup/ claudie-minio/claudie-tf-state-files ``` You can now apply your Claudie inputmanifests which will be immediately in the `DONE` stage. You can verify this with @@ -110,7 +110,9 @@ You can now apply your Claudie inputmanifests which will be immediately in the ` kubectl get inputmanifests -A ``` -Now you can make any new changes to your inputmanifests on the new management cluster and the state will be reused. +Now you can make any new changes to your inputmanifests on the new management cluster and the state will be re-used. + +!!! note "The secrets for the clusters, namely kubeconfig and cluster-metadata, are re-created after the workflow with the changes has finished." !!! note "Alternatively you may also use any GUI clients for MongoDB and Minio for more straightforward backup of the state. All you need to backup is the bucket `claudie-tf-state-files` in MinIO and the collection `inputManifests` from MongoDB" From c500cdd450cc1cc354bd06b03db4342f990044fe Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mat=C3=BA=C5=A1=20Mrekaj?= Date: Wed, 8 May 2024 16:18:50 +0200 Subject: [PATCH 6/7] Update docs/creating-claudie-backup/creating-claudie-backup.md Co-authored-by: Bernard Halas --- docs/creating-claudie-backup/creating-claudie-backup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/creating-claudie-backup/creating-claudie-backup.md b/docs/creating-claudie-backup/creating-claudie-backup.md index 7d3836d90..d7ea5bdb6 100644 --- a/docs/creating-claudie-backup/creating-claudie-backup.md +++ b/docs/creating-claudie-backup/creating-claudie-backup.md @@ -26,7 +26,7 @@ Create a directory where the backup of the state will be stored. mkdir claudie-backup ``` -Put your Claudie inputmanifests into the created folder. +Put your Claudie inputmanifests into the created folder, e.g. `kubectl get InputManifest -A -oyaml > ./claudie-backup/all.yaml` We will now back up the state of the respective input manifests from MongoDB and MinIO. From 958e117eb20cd31efe533e23152a0e9d96e763cd Mon Sep 17 00:00:00 2001 From: Bernard Halas Date: Wed, 8 May 2024 17:22:24 +0200 Subject: [PATCH 7/7] Chore: Re-order the backup methods (#1376) Velero is the primary way, therefore it should be mentioned first. --- .../creating-claudie-backup.md | 149 +++++++++--------- 1 file changed, 76 insertions(+), 73 deletions(-) diff --git a/docs/creating-claudie-backup/creating-claudie-backup.md b/docs/creating-claudie-backup/creating-claudie-backup.md index d7ea5bdb6..60f76f354 100644 --- a/docs/creating-claudie-backup/creating-claudie-backup.md +++ b/docs/creating-claudie-backup/creating-claudie-backup.md @@ -16,6 +16,81 @@ These are the only services that will have a PVC attached to it, the other are s ## Backing up Claudie +### Using Velero + +This is the primary backup and restore method. + +!!! warning "Velero does not support HostPath volumes. If the PVCs in your management cluster are attached to such volumes (e.g. when running on Kind or MiniKube), the backup will not work. In this case, use the below backup method." + +All resources that are deployed or created by Claudie can be identified with the following label: + +``` + app.kubernetes.io/part-of: claudie +``` + +!!! note "If you want to include your deployed Input Manifests to be part of the backup you'll have to add the same label to them." + +We'll walk through the following scenario step-by-step to back up claudie and then restore it. + +Claudie is already deployed on an existing Management Cluster and at least 1 Input Manifest has been applied. The state +is backed up and the Management Cluster is replaced by a new one on which we restore the state. + +!!! note "To back up the resources we'll be using Velero version v1.11.0." + +The following steps will all be executed with the existing Management Cluster in context. + +1. To create a backup, Velero needs to store the state to external storage. The list of supported + providers for the external storage can be found in the [link](https://velero.io/docs/v1.11/supported-providers/). + In this guide we'll be using AWS S3 object storage for our backup. + + +2. Prepare the S3 bucket by following the first two steps in this [setup guide](https://github.com/vmware-tanzu/velero-plugin-for-aws#setup), excluding the installation step, as this will be different for our use-case. + + +!!! note "If you do not have the `aws` CLI locally installed, follow the [user guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) to set it up." + +3. Execute the following command to install Velero on the Management Cluster. +```bash +velero install \ +--provider aws \ +--plugins velero/velero-plugin-for-aws:v1.6.0 \ +--bucket $BUCKET \ +--secret-file ./credentials-velero \ +--backup-location-config region=$REGION \ +--snapshot-location-config region=$REGION \ +--use-node-agent \ +--default-volumes-to-fs-backup +``` + +Following the instructions in step 2, you should have a `credentials-velero` file with the access and secret keys for the aws setup. The env variables `$BUCKET` and `$REGION` should be set to the name and region for the bucket created in AWS S3. + +By default Velero will use your default config `$HOME/.kube/config`, if this is not the config that points to your Management Cluster, you can override it with the `--kubeconfig` argument. + +4. Backup claudie by executing +```bash +velero backup create claudie-backup --selector app.kubernetes.io/part-of=claudie +``` + +To track the progress of the backup execute +```bash +velero backup describe claudie-backup --details +``` + +From this point the new Management Cluster for Claudie is in context. +We expect that your default `kubeconfig` points to the new Management Cluster, if it does not, you can override it in the following commands using `--kubeconfig ./path-to-config`. + +1. Repeat the step to install Velero, but now on the new Management Cluster. +2. Install cert manager to the new Management Cluster by executing: +```bash +kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.0/cert-manager.yaml +``` +3. To restore the state that was stored in the S3 bucket execute +```bash +velero restore create --from-backup claudie-backup +``` + +Once all resources are restored, you should be able to deploy new input manifests and also modify existing infrastructure without any problems. + ### Manual backup Claudie is already deployed on an existing Management Cluster and at least 1 Input Manifest has been applied. @@ -116,76 +191,4 @@ Now you can make any new changes to your inputmanifests on the new management cl !!! note "Alternatively you may also use any GUI clients for MongoDB and Minio for more straightforward backup of the state. All you need to backup is the bucket `claudie-tf-state-files` in MinIO and the collection `inputManifests` from MongoDB" - -### Using Velero - -!!! warning "Velero does not support HostPath volumes. If the PVCs in your management cluster are attached to such volumes, the backup will not work. In this case, use the above backup method." - -All resources that are deployed or created by Claudie can be identified with the following label: - -``` - app.kubernetes.io/part-of: claudie -``` - -!!! note "If you want to include your deployed Input Manifests to be part of the backup you'll have to add the same label to them." - -We'll walk through the following scenario step-by-step to back up claudie and then restore it. - -Claudie is already deployed on an existing Management Cluster and at least 1 Input Manifest has been applied. The state -is backed up and the Management Cluster is replaced by a new one on which we restore the state. - -!!! note "To back up the resources we'll be using Velero version v1.11.0" - -The following steps will all be executed with the existing Management Cluster in context. - -1. To create a backup, Velero needs to store the state to external storage. The list of supported - providers for the external storage can be found in the [link](https://velero.io/docs/v1.11/supported-providers/). - In this guide we'll be using AWS S3 object storage for our backup. - - -2. Prepare the S3 bucket by following the first two steps in this [setup guide](https://github.com/vmware-tanzu/velero-plugin-for-aws#setup), excluding the installation step, as this will be different for our use-case. - - -!!! note "If you do not have the aws CLI locally installed, follow the [user guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) to set it up." - -3. Execute the following command to install Velero on the Management Cluster. -```bash -velero install \ ---provider aws \ ---plugins velero/velero-plugin-for-aws:v1.6.0 \ ---bucket $BUCKET \ ---secret-file ./credentials-velero \ ---backup-location-config region=$REGION \ ---snapshot-location-config region=$REGION \ ---use-node-agent \ ---default-volumes-to-fs-backup -``` - -Following the instructions in step 2, you should have a `credentials-velero` file with the access and secret keys for the aws setup. The env variables `$BUCKET` and `$REGION` should be set to the name and region for the bucket created in AWS S3. - -By default Velero will use your default config `$HOME/.kube/config`, if this is not the config that points to your Management Cluster, you can override it with the `--kubeconfig` argument. - -4. Backup claudie by executing -```bash -velero backup create claudie-backup --selector app.kubernetes.io/part-of=claudie -``` - -To track the progress of the backup execute -```bash -velero backup describe claudie-backup --details -``` - -From this point the new Management Cluster for Claudie is in context. -We expect that your default `kubeconfig` points to the new Management Cluster, if it does not, you can override it in the following commands using `--kubeconfig ./path-to-config`. - -1. Repeat the step to install Velero, but now on the new Management Cluster. -2. Install cert manager to the new Management Cluster by executing: -```bash -kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.0/cert-manager.yaml -``` -3. To restore the state that was stored in the S3 bucket execute -```bash -velero restore create --from-backup claudie-backup -``` - -Once all resources are restored, you should be able to deploy new input manifests and also modify existing infrastructure without any problems. +Once all data is restored, you should be able to deploy new input manifests and also modify existing infrastructure without any problems.