diff --git a/docs/creating-claudie-backup/creating-claudie-backup.md b/docs/creating-claudie-backup/creating-claudie-backup.md index 22f64463b..60f76f354 100644 --- a/docs/creating-claudie-backup/creating-claudie-backup.md +++ b/docs/creating-claudie-backup/creating-claudie-backup.md @@ -16,6 +16,12 @@ These are the only services that will have a PVC attached to it, the other are s ## Backing up Claudie +### Using Velero + +This is the primary backup and restore method. + +!!! warning "Velero does not support HostPath volumes. If the PVCs in your management cluster are attached to such volumes (e.g. when running on Kind or MiniKube), the backup will not work. In this case, use the below backup method." + All resources that are deployed or created by Claudie can be identified with the following label: ``` @@ -29,7 +35,7 @@ We'll walk through the following scenario step-by-step to back up claudie and th Claudie is already deployed on an existing Management Cluster and at least 1 Input Manifest has been applied. The state is backed up and the Management Cluster is replaced by a new one on which we restore the state. -!!! note "To back up the resources we'll be using Velero version v1.11.0" +!!! note "To back up the resources we'll be using Velero version v1.11.0." The following steps will all be executed with the existing Management Cluster in context. @@ -41,7 +47,7 @@ The following steps will all be executed with the existing Management Cluster in 2. Prepare the S3 bucket by following the first two steps in this [setup guide](https://github.com/vmware-tanzu/velero-plugin-for-aws#setup), excluding the installation step, as this will be different for our use-case. -!!! note "If you do not have the aws CLI locally installed, follow the [user guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) to set it up." +!!! note "If you do not have the `aws` CLI locally installed, follow the [user guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) to set it up." 3. Execute the following command to install Velero on the Management Cluster. ```bash @@ -83,4 +89,106 @@ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/ velero restore create --from-backup claudie-backup ``` -Once all resources are restored, you should be able to deploy new input manifests and also modify existing infrastructure without any problems. \ No newline at end of file +Once all resources are restored, you should be able to deploy new input manifests and also modify existing infrastructure without any problems. + +### Manual backup + +Claudie is already deployed on an existing Management Cluster and at least 1 Input Manifest has been applied. + +Create a directory where the backup of the state will be stored. + +```bash +mkdir claudie-backup +``` + +Put your Claudie inputmanifests into the created folder, e.g. `kubectl get InputManifest -A -oyaml > ./claudie-backup/all.yaml` + +We will now back up the state of the respective input manifests from MongoDB and MinIO. + +```bash +kubectl get pods -n claudie + +NAME READY STATUS RESTARTS AGE +ansibler-6f4557cf74-b4dts 1/1 Running 0 18m +builder-5d68987c86-qdfd5 1/1 Running 0 18m +claudie-operator-6d9ddc7f8b-hv84c 1/1 Running 0 18m +context-box-5d75bfffc6-d9qfm 1/1 Running 0 18m +create-table-job-ghb9f 0/1 Completed 1 18m +dynamodb-6d65df988-c626j 1/1 Running 0 18m +kube-eleven-556cfdfd98-jq6hl 1/1 Running 0 18m +kuber-7f8cd4cd89-6ds2w 1/1 Running 0 18m +make-bucket-job-9mjft 0/1 Completed 0 18m +minio-0 1/1 Running 0 18m +minio-1 1/1 Running 0 18m +minio-2 1/1 Running 0 18m +minio-3 1/1 Running 0 18m +mongodb-6ccb5f5dff-ptdw2 1/1 Running 0 18m +scheduler-676bd4468f-6fjn8 1/1 Running 0 18m +terraformer-66c6f67d98-pwr9t 1/1 Running 0 18m +``` + +To backup state from MongoDB execute the following command + +```bash +kubectl exec -n claudie mongodb- -- sh -c 'mongoexport --uri=mongodb://$MONGO_INITDB_ROOT_USERNAME:$MONGO_INITDB_ROOT_PASSWORD@localhost:27017/claudie -c inputManifests --authenticationDatabase admin' > claudie-backup/inputManifests +``` + +Next we need to backup the state from MinIO. Port-forward the MinIO service so that it is accessible from localhost. + +```bash +kubectl port-forward -n claudie svc/minio 9000:9000 +``` + +Setup an alias for the [mc](https://min.io/docs/minio/linux/reference/minio-mc.html) command line tool. + +```bash +mc alias set claudie-minio http://127.0.0.1:9000 +``` + +!!! note "Provide the access and secret key for minio. The default can be found in the github repository in the `manifests/claudie/minio/secrets` folder. If you have not changed them, we strongly encourage you to do so!" + +Download the state into the backup folder + +```bash +mc mirror claudie-minio/claudie-tf-state-files ./claudie-backup +``` + +You now have everything you need to restore your input manifests to a new management cluster. + +!!! warning "These files will contain your credentials, DO NOT STORE THEM OUT IN THE PUBLIC!" + +To restore the state on your new management cluster you can follow these commands. We expect that your default `kubeconfig` points to the new Management Cluster, if it does not, you can override it in the following commands using `--kubeconfig ./path-to-config`. + +Copy the collection into the MongoDB pod. + +```bash +kubectl cp ./claudie-backup/inputManifests mongodb-:/tmp/inputManifests -n claudie +``` + +Import the state to MongoDB. + +```bash +kubectl exec -n claudie mongodb- -- sh -c 'mongoimport --uri=mongodb://$MONGO_INITDB_ROOT_USERNAME:$MONGO_INITDB_ROOT_PASSWORD@localhost:27017/claudie -c inputManifests --authenticationDatabase admin --file /tmp/inputManifests' +``` + +!!! note "Don't forget to delete the `/tmp/inputManifests` file" + +Port-forward the MinIO service and import the backed up state. + +```bash +mc cp --recursive ./claudie-backup/ claudie-minio/claudie-tf-state-files +``` + +You can now apply your Claudie inputmanifests which will be immediately in the `DONE` stage. You can verify this with + +```bash +kubectl get inputmanifests -A +``` + +Now you can make any new changes to your inputmanifests on the new management cluster and the state will be re-used. + +!!! note "The secrets for the clusters, namely kubeconfig and cluster-metadata, are re-created after the workflow with the changes has finished." + +!!! note "Alternatively you may also use any GUI clients for MongoDB and Minio for more straightforward backup of the state. All you need to backup is the bucket `claudie-tf-state-files` in MinIO and the collection `inputManifests` from MongoDB" + +Once all data is restored, you should be able to deploy new input manifests and also modify existing infrastructure without any problems. diff --git a/manifests/claudie/crd/claudie.io_inputmanifests.yaml b/manifests/claudie/crd/claudie.io_inputmanifests.yaml index d0057e3eb..9457f765d 100644 --- a/manifests/claudie/crd/claudie.io_inputmanifests.yaml +++ b/manifests/claudie/crd/claudie.io_inputmanifests.yaml @@ -6,6 +6,8 @@ metadata: controller-gen.kubebuilder.io/version: v0.11.3 creationTimestamp: null name: inputmanifests.claudie.io + labels: + app.kubernetes.io/part-of: claudie spec: group: claudie.io names: