velero backup Kubernetes Persistent Volume failed with minio

[duyanghao]

What steps did you take and what happened:

# step1: Install velero+Minio
$ velero install \
      --provider aws \
      --plugins velero/velero-plugin-for-aws:v1.0.0 \
      --bucket velero \
      --secret-file ./credentials-velero \
      --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio.velero.svc:9000 \
      --snapshot-location-config region=minio

# step2: Deploy nginx application(with-pv and remove fsfreeze)
$ kubectl apply -f examples/nginx-app/with-pv.yaml

$ kubectl get svc -nnginx-example
NAME       TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
my-nginx   LoadBalancer   x.x.x.x   <pending>     80:32648/TCP   4m43s
$ curl x.x.x.x
...
<title>Welcome to nginx!</title>
...

$ kubectl exec nginx-deployment-7d46f5c9d-2t895 -nnginx-example -- cat /var/log/nginx/access.log
172.70.0.1 - - [19/Mar/2020:10:56:54 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
172.70.0.1 - - [19/Mar/2020:10:56:55 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
172.70.0.1 - - [19/Mar/2020:10:56:56 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
172.70.0.1 - - [19/Mar/2020:10:56:56 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
172.70.0.1 - - [19/Mar/2020:10:56:57 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
172.70.0.1 - - [19/Mar/2020:10:56:57 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
172.70.0.1 - - [19/Mar/2020:10:56:57 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
172.70.0.1 - - [19/Mar/2020:10:56:58 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
172.70.0.1 - - [19/Mar/2020:10:56:58 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
172.70.0.1 - - [19/Mar/2020:10:56:58 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"

# step3: Create a backup with PV snapshotting
$ velero backup create nginx-backup-with-pv --include-namespaces nginx-example

$ velero backup get 
NAME                   STATUS      CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
nginx-backup-with-pv   Completed   2020-03-19 18:53:34 +0800 CST   29d       default            <none>

# step4: Simulate a disaster
$ kubectl delete namespace nginx-example
namespace "nginx-example" deleted

# step5: Restore
$ velero restore create --from-backup nginx-backup-with-pv
Restore request "nginx-backup-with-pv-20200319190204" submitted successfully.
Run `velero restore describe nginx-backup-with-pv-20200319190204` or `velero restore logs nginx-backup-with-pv-20200319190204` for more details.
$ velero restore get
NAME                                   BACKUP                  STATUS      WARNINGS   ERRORS   CREATED                         SELECTOR
nginx-backup-with-pv-20200319190204   nginx-backup-with-pv   Completed   0          0        2020-03-19 19:02:05 +0800 CST   <none>

# step6: Check Persistent Volume
$ kubectl exec nginx-deployment-7d46f5c9d-2t895 -nnginx-example -- cat /var/log/nginx/access.log
[nothing disappear] 

What did you expect to happen:

velero restore will write access logs to /var/log/nginx/access.log, and the cat command should output as below:

$ kubectl exec nginx-deployment-7d46f5c9d-2t895 -nnginx-example -- cat /var/log/nginx/access.log
172.70.0.1 - - [19/Mar/2020:10:56:54 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
172.70.0.1 - - [19/Mar/2020:10:56:55 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
172.70.0.1 - - [19/Mar/2020:10:56:56 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
172.70.0.1 - - [19/Mar/2020:10:56:56 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
172.70.0.1 - - [19/Mar/2020:10:56:57 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
172.70.0.1 - - [19/Mar/2020:10:56:57 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
172.70.0.1 - - [19/Mar/2020:10:56:57 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
172.70.0.1 - - [19/Mar/2020:10:56:58 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
172.70.0.1 - - [19/Mar/2020:10:56:58 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"
172.70.0.1 - - [19/Mar/2020:10:56:58 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.29.0" "-"

Environment:

• Velero version (use velero version): v1.3.1
• Velero features (use velero client config get features): features: <NOT SET>
• Kubernetes version (use kubectl version): v1.17.0

Addition:

$ kubectl exec -it minio-d787f4bf7-jgmbq -nvelero sh
/ # ls /storage/velero/backups/nginx-backup-with-pv/
nginx-backup-with-pv-logs.gz                   nginx-backup-with-pv-volumesnapshots.json.gz
nginx-backup-with-pv-podvolumebackups.json.gz  nginx-backup-with-pv.tar.gz
nginx-backup-with-pv-resource-list.json.gz     velero-backup.json

both nginx-backup-with-pv-volumesnapshots.json.gz and nginx-backup-with-pv-podvolumebackups.json.gz are empty.

[skriss]

@duyanghao for backing up the contents of volumes you have two options:

1. Use a volumesnapshotter plugin to take native snapshots of your volumes. This would work if you are running in AWS proper, and you have EBS volumes that you want to take snapshots of.
2. Use velero's restic integration to take file-level backups of the contents of your volumes. You can use this to back up any type of volume (except hostPath).

From your install command, I gather that you're not running in AWS itself, so you probably want to look into using the restic integration for volume backups. You'll need to re-run your install command with the --use-restic flag, and then add an annotation to the nginx pod defining which pod volume(s) you want to back up.

Closing this out, but feel free to reach out again as needed.

[duyanghao]

@skriss thanks for your reply, there are still some questions about bur(backup and restore) of kubernetes persistent volume using velero:

1. Is there a do-one-command or automatic way for us to backup all persistent volume resources in the cluster? Right now, we have to add relevant annotations to all of pods with pv resources in cluster, which is complicated and unpractical in a production environment.

2. There are two things we want to backup for kubernetes:
   Kuberentes data: Kubernetes stores its state in etcd, we can backup etcd for this purpose.
   Application data: i.e. persistent volumes, we want to use velero for this purpose(this is what I want to talk about here).
   According to my tests, velero with restic Integration can't restore an existing kubernete-persistent-volume, or to be more specific, it can't overwrite the data of kubernete-persistent-volume.

An explanation for my second question:
I will do following steps to backup and restore my kubernetes cluster:
backup
step1 - backup Kuberentes data(i.e. etcd)
This post wont talk about backing up etcd as its quite well documented here

step2 - backup Application data(i.e. kubernete-persistent-volume)

(add annotations by hand...)
$ velero backup create xxx --include-resources persistentvolumeclaims,persistentvolumes

restore(imagine that cluster has been deleted totally)
step1 - restore Kuberentes data(i.e. etcd)
Refers to this post again

Attention: There are already pods and relevant pvs(no application data) after etcd restoration

step2 - restore Application data(i.e. kubernete-persistent-volume)

$ velero restore create --from-backup xxx --include-resources persistentvolumeclaims,persistentvolumes

And now velero restore create --from-backup xxx --include-resources persistentvolumeclaims,persistentvolumes will do nothing for the existing pvs that we want to restore.

[skriss]

Is there a do-one-command or automatic way for us to backup all persistent volume resources in the cluster? Right now, we have to add relevant annotations to all of pods with pv resources in cluster, which is complicated and unpractical in a production environment.

Velero doesn't have a built-in command to do this, you'd need to write your own script/automation. You could check out this community project: https://github.com/bitsbeats/velero-pvc-watcher

According to my tests, velero with restic Integration can't restore an existing kubernete-persistent-volume, or to be more specific, it can't overwrite the data of kubernete-persistent-volume.

That's correct, Velero does not support overwriting existing resources, you'd need to delete/recreate the application (YAML) and PV data together. #504 and #469 are relevant RFEs.

[tirelibirefe]

@skriss thanks for your reply, there are still some questions about bur(backup and restore) of kubernetes persistent volume using velero:

1. Is there a do-one-command or automatic way for us to backup all persistent volume resources in the cluster? Right now, we have to add relevant annotations to all of pods with pv resources in cluster, which is complicated and unpractical in a production environment.
2. There are two things we want to backup for kubernetes:
   Kuberentes data: Kubernetes stores its state in etcd, we can backup etcd for this purpose.
   Application data: i.e. persistent volumes, we want to use velero for this purpose(this is what I want to talk about here).
   According to my tests, velero with restic Integration can't restore an existing kubernete-persistent-volume, or to be more specific, it can't overwrite the data of kubernete-persistent-volume.

An explanation for my second question:
I will do following steps to backup and restore my kubernetes cluster:
backup
step1 - backup Kuberentes data(i.e. etcd)
This post wont talk about backing up etcd as its quite well documented here

step2 - backup Application data(i.e. kubernete-persistent-volume)

(add annotations by hand...)
$ velero backup create xxx --include-resources persistentvolumeclaims,persistentvolumes

restore(imagine that cluster has been deleted totally)
step1 - restore Kuberentes data(i.e. etcd)
Refers to this post again

Attention: There are already pods and relevant pvs(no application data) after etcd restoration

step2 - restore Application data(i.e. kubernete-persistent-volume)

$ velero restore create --from-backup xxx --include-resources persistentvolumeclaims,persistentvolumes

And now velero restore create --from-backup xxx --include-resources persistentvolumeclaims,persistentvolumes will do nothing for the existing pvs that we want to restore.

@duyanghao you're right in your considerations. Velero is not a mature yet.