How do you use Velero? Let us know!

[VMmore]

This really isn't an issue, but we'd love to hear about your use of Velero so we thought we'd post this to find out more from you.

Please feel free to add a comment below and let us know.

[lledru]

Hi, we used to run ark for protecting our stateful production workloads for almost a year, now we finish migrating to velero. We are internal service provider and give our customers the capacity to self enroll to a backup policy. It simply rocks ! Thanks a lot ! As usual with heptio's work :)

[dharmab]

We're currently using it to back up Kubernetes objects in Azure and AWS. We're interested in using it as a migration tool to migrate workloads between clusters in different cloud accounts or even between self-managed and cloud-managed clusters.

[harshal-shah]

We use velero to gather hourly backups of all application resources of our clusters. We have also used it to perform critical migrations such as change of kubernetes networking plugin on a running production cluster or moving our production environment to eks. Both these cases were achieved without incurring any downtime.

[agolomoodysaada]

We use it as our first line of disaster recovery for AKS and GKE clusters. We have additional manual cron backups using native mongodump for MongoDB and elasticsearch snapshots for Elasticsearch. We would love to use Velero for cross-cloud migrations as well. Having native Velero plugins for the currently manual cron jobs would be a great improvement.

[skriss]

@agolomoodysaada would love to hear more about how you'd see the mongo & elastic backups integrating into velero. How are those run today? Do you use a sidecar, or exec into the workload pod, or something else?

[agolomoodysaada]

We're using ES Snapshot API and Mongodump containers using Kubernetes Cronjob objects. Both are backed by Google Cloud Storage and Azure Blob Storage. The cronjob talks to ES Snapshot API and ES handles the rest against its corresponding blob storage. Mongodump container pipes the dump stdout from a mongo secondary pod into a remote gzip file. But it would be really awesome if Velero had a plugin for each of those datastores that could track the progress of these backups and leverage those during the restore events using builtin Velero hooks for example.

Today's alternative is to rely on Restic, but unfortunately it still feels too manual and complicated to setup (adding annotations for each volume and so on).

I imagine an API like the following

# elasticsearch
velero plugin install elasticsearch
velero plugin elasticsearch add my-es-cluster "http://my-es-cluster.my-es-namespace:9200"

# mongodump
velero plugin install mongodb
velero plugin mongodb add my-mongo-cluster "mongodb://USERNAME:PASSWORD@my-mongo-cluster.my-mongo-namespace:27017/production?readPreference=secondaryPreferred"

# velero backup and restore
velero schedule create <SCHEDULE NAME> --schedule "0 7 * * *"
velero restore create --from-backup <SCHEDULE NAME>-<TIMESTAMP>

UPDATE:

We would much prefer to rely on restic + hooks if the experience was easier. For example, I want to be able to kubectl apply -f myresticselectors.yaml and Velero would automatically backup all volumes based on those selectors. For example, we can create the following

kind: VeleroResticResource
spec:
  volumes: my-data # defaults to all volumes
  selector:
    "statefulset.kubernetes.io/pod-name": "mongodb-replicaset-0"
    "release": "mongodb"
  pre-backup-hook: "mongo --eval 'rs.stepDown(); db.fsyncLock()'"
  post-backup-hook: "mongo --eval 'db.fsyncUnlock()'"

[agolomoodysaada]

@skriss let me know if my comment should be its own Github issue as a feature request

[tillepille]

We are using it experimentally to backup our Cert Manager so we can retain our LE generated Certificates when tearing down a dev-cluster.
There is no workload yet but we will probably back it up, too. All into an s3 bucket.

I discovered some pitfalls with velero but read more at #1508

[skriss]

@VMmore do you want to leave this open, or can we close it given it hasn't seen any activity in several months?

[skriss]

Closing this out as inactive.

[hyp3ri0n-ng]

I was actually a bit confused on how to use it to be honest. I think I was just new to kubernetes and being thick. My cluster completely crashed and burned ('twas Cillium killed the beast). I tried to restore and it told me I needed to have a working cluster. I was kinda confused because, well, that was the whole point of Velero - I didn't have a working cluster anymore and I wanted to recover it. Looking back though (this was a while ago) I think I was probably supposed to create like one control plane node that was working and restore it from there?

[blackpiglet]

@hyp3ri0n-ng
No, that's not how the Velero works.
Velero interacts with the kube-apiserver to provision the resources included in the backup.
Another important feature of Velero is that it can also restore the data in the volumes.

Velero focuses on the k8s resource backup, but Velero cannot provision a new k8s cluster.