This repository contains example files and scripts for backing up and restoring an OpenShift cluster. The resources provided here are intended to help you understand the process and customize it for your environment.
- DPA, Backups, and Cloud Credential
- Restore Files
- Resource Modifier
- Unmounted PVs
- Scripts
- Main Script
The dpa-backups-cloud-credential directory contains a file for the Data Protection Applications (DPA), example backup files and a cloud-credenial file.
-
oadp-dpa.yaml
This YAML file configures backup and snapshot locations, specifying the necessary details for storing and managing data protection. Kopia is an open-source tool used for creating, managing, and restoring backups. In the context of the OADP,kopiaserves as theuploadertype for thenodeAgent, which means it is the mechanism responsible for efficiently transferring and storing backup data.Note you can have multiple backup locations specified in the DPA, you would need to duplicate lines 8-22 and update for a second location.
-
ibmcos-bucket-creds.json This file needs to be updated to contain the HMAC credentials from your COS bucket. Refer back to the blog for more details on how to retrieve your credentials and create the credentials on the cluster.
-
backup-csi.yaml
This YAML file configures a backup resource namedbackup-name-csiin theopenshift-adpnamespace. It specifiesdpa-name-1as the volume snapshot location and excludes theopenshift-adpandopenshift-storagenamespaces from the backup. The backup uses volume snapshots (snapshotVolumes: true) and is stored inbackup-location-name-from-dpa. It has a TTL of 72000 hours, meaning it will be deleted after 3000 days. -
backup-filesystem.yaml
This YAML file configures a backup resource namedbackup-name-filesystemin theopenshift-adpnamespace. It specifiesdpa-name-1as the volume snapshot location and excludes theopenshift-adpandopenshift-storagenamespaces from the backup and defaults to file system backups for all volumes. It enables volume snapshots and their data movement to be stored inbackup-location-name-from-dpa. It has a TTL of 72000 hours, meaning it will be deleted after 3000 days.
-
Clone this repository:
git clone https://github.com/IBM/openshift-oadp-examples.git cd openshift-oadp-examples -
Review and modify the example files as necessary to fit your OpenShift environment.
-
You will need to authenticate to your cluster through the command-line.
To log in to an OpenShift cluster using the command line, follow these steps:
-
Install the OpenShift CLI (oc): If you haven't already, you need to download and install the OpenShift CLI (
oc). You can find the installation instructions here. -
Obtain the Login Command: You need the login command provided by your OpenShift cluster administrator or found in the OpenShift web console. This typically includes the API server URL and a token or username and password.
-
Run the Login Command: Use the provided login command in your terminal. The command should look like the following:
- Log in with this token:
oc login --token=sha256..... --server=https.....
-
-
Execute the main script to perform a restore.
The restores directory contains YAML files necessary for restoring your OpenShift cluster. These files define the state of the cluster resources to be restored, such as deployments, services, and persistent volume claims.
-
restore-1-example.yaml
In summary, this YAML file is configuring a restore operation namedrestore-1-examplein theopenshift-adpnamespace, restoring from a backup namedbackup-name, and including specific namespaces (namespace-1andnamespace-2) and resources (project,secrets,routes). Additionally, it specifies a timeout for operations and uses a ConfigMap namedresource-modifierfor resource modification during the restore process. -
restore-2-example-csi.yaml
This YAML file defines a restore operation namedrestore-2-example-csiin theopenshift-adpnamespace. It will restore resources from a backup namedbackup-name, specifically targeting resources innamespace-3and including persistent volumes, persistent volume claims, and volume snapshots. The operation has a timeout of 20 minutes per item and will restore persistent volumes as well. -
restore-2-example-filesystem.yaml
This YAML file defines a restore operation namedrestore-2-example-filesystemin theopenshift-adpnamespace. It will restore resources from a backup namedbackup-name, focusing on thenamespace-3namespace and restoring only pods, persistent volumes, and persistent volume claims. Each item operation in the restore process has a timeout limit of 20 minutes.
The resource-modifier directory includes a configuration file used to modify resources. These can be used to change resource configurations to fit the target environment.
This file is an example that can be used if you need to update a route or a secret.
The provided resource modifier file demonstrates how to apply patches to secrets and routes. This example serves as a template for basic modifications, showing the standard method for patching these resources.
In our experience, not all secrets can be patched using a resource modifier. For some complex secrets, a script was used to achieve the desired modifications. This script offers a more flexible approach to handle intricate scenarios. The script is provided in this repo search-replace-secrets.sh.
- resource-modifier.yaml:
This YAML file defines a Kubernetes ConfigMap namedresource-modifierin theopenshift-adpnamespace, containing rules to modify specific resources. It updates a secret'sstringData.urlfield in thesecret-namespace-herenamespace and replaces a value in a route's path in theroute-namespace-herenamespace.
The unmounted-pvs directory contains a file related to mounting a Persistent Volume (PV) if it is not mounted. This step is crucial for accessing persistent data during the backup and restore process. If a PV is not mounted, it will not be backed up.
- mounting-pod.yaml: YAML file to create and mount the required volumes that are not mounted.
The scripts directory contains various utility scripts used throughout the restore process and are referenced in the main-script.sh. These scripts automate repetitive tasks and ensure consistency.
-
restore-utilities.sh
This script contains various task that are needed multiple times in the main script. -
delete-pods.sh
Delete pods in a given namespace. -
delete-secrets.sh:
Delete secrets of a certain type in a given namespace. -
search-replace-secrets.sh:
Search and replace the old cluster name and updates with the new cluster name in a given namesapce. Multiple secrets can be patched at a time, need to be seperated by a comma.
For example: "secret1,secret2,secret3,secret4,secret5"
The main-script.sh is the primary script orchestrating the entire restore process. It calls other scripts and manages the workflow.
- You are able to run this script in a dry run mode, so you are able to test the script without making any real changes to the cluster. This is a parameter that will need to be specified when running the script.
- Specify using filesystem or CSI. If you are using filesystem set to true, if using CSI set to false. This is also a parameter that will be specified when running the script.
This script ensures that resources are modified appropriately, and restores are executed in a sequence.
Commands to run the main scrips are below:
This example command will run the script in dry run mode for CSI. Change DRY_RUN to false if you wish to run the script for real. Make sute to update the command with your cluster URL.
DRY_RUN=true USE_FILESYSTEM=false ./main-script.sh https://.cluster-name-123b456bb0fc123456d8bb0dff2739ca-0000.us-east.containers.appdomain.cloud backup-nameThis command will run the script in dry run mode for Filesystem. Change DRY_RUN to false if you wish to run the script for real. Make sute to update the command with your cluster URL.
DRY_RUN=true USE_FILESYSTEM=true ./main-script.sh https://.cluster-name-123b456bb0fc123456d8bb0dff2739ca-0000.us-east.containers.appdomain.cloud backup-name- Sign-in to IBM Cloud account
- On the left hand side, open the Navigation Menu
- Navigate to Openshift > Clusters
- Select cluster
- Make sure you are on the Overview tab
- Scroll down till you find the Ingress subdomain
- Make sure to add https://