Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
README.md
busybox-recover.yaml
busybox.yaml
patch-replica-dep-nodename.json
recovery-pvc.yaml

README.md

This document contains notes on how to recover data from a backedup jiva replica files. OpenEBS jiva volumes, save the data in /var/openebs/pvc-id/. All the replica's contain identical data. Before performing a cluster re-build, it suffices to have a backup of data from one of the replica's /var/openebs/pvc-id/

These steps are required for OpenEBS 0.5.4 where replicas could get scheduled onto a node where the data doesn't exist.

Step 1: Run an sample application that generates some data.

For this example, I ran a busybox pod that saves hostname and date into the mounted openebs volume.

kubectl apply -f busybox.yaml

Wait for the busybox app to run, exec into it to check data is generated. May be put some additional content if required.

Note the following details:

  • PV id kubectl get pv (say source-pv-id ).
  • nodes on which PV replica pods are running kubectl get pods -o wide | grep source-pv-id (say replica-hostname).

Delete the busybox pod kubectl delete -f busybox.yaml. Note that the data folders will remain on the nodes even though the pod and pvs are deleted.

Step 2: Setup a Recovery PVC

Deploy a Recovery PVC with a single replica - kubectl apply -f recovery-pvc.yaml

Note the following details:

  • PV id kubectl get pv (say recovery-pv-id ).
  • PV replica deployment name kubectl get deploy | grep recovery-pv-id (say recovery-replica-deploy ).
  • Recovery PVC namespace, if you have changed it to something other than default. (say recovery-replica-ns).

Step 3: Patch the Recovery PV Replica to stick to the replica-hostname

Replica hostname in patch-replica-dep-nodename.json with the replica-hostname that was obtained in Step 1. It is the node where source/backuped up replica data is available. If the backup data is available on a remote machine, you can set the hostname to the current node where Replica is running.

kubectl patch deploy -n <replica-replica-ns> <recovery-replica-deploy>  -p "$(cat patch-replica-dep-nodename.json)"

After the patch is applied, you will notice that the replica pod is restarted on the hostname specified. Since this replica deployment is patched, you will see an orphaned replica set kubectl get rs. You can go ahead and deleted it.

Step 4: Copy the backup data into Recovery Replica

  • ssh into the node (replica-hostname)
  • cd /var/openebs/recovery-pv-id/ (/var/openebs if you are using default pool.)
  • sudo rm -rf *
  • copy contents from earlier volume (/var/openebs/source-pv-id or from remote server) into /var/openebs/recovery-pv-id/
  • you will see peer.details, revision.counter, volume.meta and a bunch of .img and .meta files.
  • edit peer.details to set ReplicaCount=1
  • exit

Step 5: Restart the Volume Pods

  • kubectl delete replica-pod. note that it gets rescheduled on to the same node (replica-hostname)
  • kubectl delete controller-pod. Wait for these pods to get back to running state.

Step 6: Use the recovery volume to retrieve the data.

You can either launch the source application or a recovery application that now makes use of this recovery volume. In this example, I use the busybox-recovery pod that displays the content of a file that was supposed to be generated by the source application.

kubectl apply -f busybox-recovery.yaml

You can also exec into this application to check the content, retrieve the files or use the application to check the content.