Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update backup method #1375

Merged
merged 8 commits into from
May 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
114 changes: 111 additions & 3 deletions docs/creating-claudie-backup/creating-claudie-backup.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@ These are the only services that will have a PVC attached to it, the other are s

## Backing up Claudie

### Using Velero

This is the primary backup and restore method.

!!! warning "Velero does not support HostPath volumes. If the PVCs in your management cluster are attached to such volumes (e.g. when running on Kind or MiniKube), the backup will not work. In this case, use the below backup method."

All resources that are deployed or created by Claudie can be identified with the following label:

```
Expand All @@ -29,7 +35,7 @@ We'll walk through the following scenario step-by-step to back up claudie and th
Claudie is already deployed on an existing Management Cluster and at least 1 Input Manifest has been applied. The state
is backed up and the Management Cluster is replaced by a new one on which we restore the state.

!!! note "To back up the resources we'll be using Velero version v1.11.0"
!!! note "To back up the resources we'll be using Velero version v1.11.0."

The following steps will all be executed with the existing Management Cluster in context.

Expand All @@ -41,7 +47,7 @@ The following steps will all be executed with the existing Management Cluster in
2. Prepare the S3 bucket by following the first two steps in this [setup guide](https://github.com/vmware-tanzu/velero-plugin-for-aws#setup), excluding the installation step, as this will be different for our use-case.


!!! note "If you do not have the aws CLI locally installed, follow the [user guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) to set it up."
!!! note "If you do not have the `aws` CLI locally installed, follow the [user guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html) to set it up."

3. Execute the following command to install Velero on the Management Cluster.
```bash
Expand Down Expand Up @@ -83,4 +89,106 @@ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/
velero restore create --from-backup claudie-backup
```

Once all resources are restored, you should be able to deploy new input manifests and also modify existing infrastructure without any problems.
Once all resources are restored, you should be able to deploy new input manifests and also modify existing infrastructure without any problems.

### Manual backup

Claudie is already deployed on an existing Management Cluster and at least 1 Input Manifest has been applied.

Create a directory where the backup of the state will be stored.

```bash
mkdir claudie-backup
```

Put your Claudie inputmanifests into the created folder, e.g. `kubectl get InputManifest -A -oyaml > ./claudie-backup/all.yaml`

We will now back up the state of the respective input manifests from MongoDB and MinIO.

```bash
kubectl get pods -n claudie

NAME READY STATUS RESTARTS AGE
ansibler-6f4557cf74-b4dts 1/1 Running 0 18m
builder-5d68987c86-qdfd5 1/1 Running 0 18m
claudie-operator-6d9ddc7f8b-hv84c 1/1 Running 0 18m
context-box-5d75bfffc6-d9qfm 1/1 Running 0 18m
create-table-job-ghb9f 0/1 Completed 1 18m
dynamodb-6d65df988-c626j 1/1 Running 0 18m
kube-eleven-556cfdfd98-jq6hl 1/1 Running 0 18m
kuber-7f8cd4cd89-6ds2w 1/1 Running 0 18m
make-bucket-job-9mjft 0/1 Completed 0 18m
minio-0 1/1 Running 0 18m
minio-1 1/1 Running 0 18m
minio-2 1/1 Running 0 18m
minio-3 1/1 Running 0 18m
mongodb-6ccb5f5dff-ptdw2 1/1 Running 0 18m
scheduler-676bd4468f-6fjn8 1/1 Running 0 18m
terraformer-66c6f67d98-pwr9t 1/1 Running 0 18m
```

To backup state from MongoDB execute the following command

```bash
bernardhalas marked this conversation as resolved.
Show resolved Hide resolved
kubectl exec -n claudie mongodb-<your-mongdb-pod> -- sh -c 'mongoexport --uri=mongodb://$MONGO_INITDB_ROOT_USERNAME:$MONGO_INITDB_ROOT_PASSWORD@localhost:27017/claudie -c inputManifests --authenticationDatabase admin' > claudie-backup/inputManifests
```

Next we need to backup the state from MinIO. Port-forward the MinIO service so that it is accessible from localhost.

```bash
kubectl port-forward -n claudie svc/minio 9000:9000
```

Setup an alias for the [mc](https://min.io/docs/minio/linux/reference/minio-mc.html) command line tool.

```bash
mc alias set claudie-minio http://127.0.0.1:9000 <ACCESSKEY> <SECRETKEY>
```

!!! note "Provide the access and secret key for minio. The default can be found in the github repository in the `manifests/claudie/minio/secrets` folder. If you have not changed them, we strongly encourage you to do so!"

Download the state into the backup folder

```bash
mc mirror claudie-minio/claudie-tf-state-files ./claudie-backup
```

You now have everything you need to restore your input manifests to a new management cluster.

!!! warning "These files will contain your credentials, DO NOT STORE THEM OUT IN THE PUBLIC!"

To restore the state on your new management cluster you can follow these commands. We expect that your default `kubeconfig` points to the new Management Cluster, if it does not, you can override it in the following commands using `--kubeconfig ./path-to-config`.

Copy the collection into the MongoDB pod.

```bash
kubectl cp ./claudie-backup/inputManifests mongodb-<your-mongodb-pod>:/tmp/inputManifests -n claudie
```

Import the state to MongoDB.

```bash
kubectl exec -n claudie mongodb-<your-mongodb-pod> -- sh -c 'mongoimport --uri=mongodb://$MONGO_INITDB_ROOT_USERNAME:$MONGO_INITDB_ROOT_PASSWORD@localhost:27017/claudie -c inputManifests --authenticationDatabase admin --file /tmp/inputManifests'
```

!!! note "Don't forget to delete the `/tmp/inputManifests` file"

Port-forward the MinIO service and import the backed up state.

```bash
mc cp --recursive ./claudie-backup/<your-folder-name-downloaded-from-minio> claudie-minio/claudie-tf-state-files
```

You can now apply your Claudie inputmanifests which will be immediately in the `DONE` stage. You can verify this with

```bash
kubectl get inputmanifests -A
```

Now you can make any new changes to your inputmanifests on the new management cluster and the state will be re-used.

!!! note "The secrets for the clusters, namely kubeconfig and cluster-metadata, are re-created after the workflow with the changes has finished."

!!! note "Alternatively you may also use any GUI clients for MongoDB and Minio for more straightforward backup of the state. All you need to backup is the bucket `claudie-tf-state-files` in MinIO and the collection `inputManifests` from MongoDB"

Once all data is restored, you should be able to deploy new input manifests and also modify existing infrastructure without any problems.
2 changes: 2 additions & 0 deletions manifests/claudie/crd/claudie.io_inputmanifests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ metadata:
controller-gen.kubebuilder.io/version: v0.11.3
creationTimestamp: null
name: inputmanifests.claudie.io
labels:
app.kubernetes.io/part-of: claudie
spec:
group: claudie.io
names:
Expand Down
Loading