backup/restore procedure [question]

[eleaner]

Hi

I am looking for backup/restore procedure in the docs and cannot find one

any help here?

[bencompton]

(source) ...step two is to completely rework the state management such that you no longer need to persist state in k8s. This is a much larger goal. I've fooled around a lot with the persistence layer in k8s (this project is running on sqlite3) so I'm basically doing a lot of work of figuring what state I can throw away. The theory is that all desired state comes from your yaml files. Actual state is actual (what really exists). Everything else in k8s should be purely derived and not important.

(source) ...but right now HA is not available, you can only create a high resilient system by backing up and restoring the single master

It sounds like the ultimate goal is for the backup/restore process to be dead simple. At this point, is all state that can't be regenerated automatically stored in SQLite or is there important state elsewhere?

[bencompton]

The video from the meetup yesterday answers my question, and it sounds like all important state is indeed stored in SQLite:

(source) ...it's just a single file you have to backup if you want to manage it and do some type of automated backup and restore

(source) ...you can get a highly-resilient setup, which means if the master goes down, you can restore it very quickly somewhere else because the state is stored in a single file

If that's the case, a cron job regularly running sqlite3 .backup should do the trick. Then restoring should be as simple as spinning up a new master from your original image and running sqlite3 .restore. I haven't tested this out myself, though.

[warmchang]

The video from the meetup yesterday answers my question, and it sounds like all important state is indeed stored in SQLite:

(source) ...it's just a single file you have to backup if you want to manage it and do some type of automated backup and restore

(source) ...you can get a highly-resilient setup, which means if the master goes down, you can restore it very quickly somewhere else because the state is stored in a single file

If that's the case, a cron job regularly running sqlite3 .backup should do the trick. Then restoring should be as simple as spinning up a new master from your original image and running sqlite3 .restore. I haven't tested this out myself, though.

That's great. But the slqlite should switch to etcd in HA mode, and can use etcd snapshot to do this.
FYI: #42 (comment)

[erikwilson]

Since we use sqlite and maybe etcd in the future it might be nice to have a solution which doesn't depend on implementation. I was trying to find a backup solution that might just use shell scripts and kubectl, not sure of the feasibility of something like that tho.

[ndarilek]

So to confirm, cluster state backups are essentially just backing up /var/lib/rancher/k3s/server/db/state.db?

I have file/database backups, but my database backup solution (KubeDB) requires that the database superuser credentials are identical for WAL restores, and those are autogenerated and stored in secrets. So I need some way of keeping those secrets around if I have to restore backups, and I'd rather just let them be autogenerated and leave them in-cluster than manage them myself.

Thanks.

[ahmedmagdiosman]

This issue is a bit old now, I'm wondering if things changed. Can we get some pointers about the current recommended method of backing up/restoring a k3s environment?

Thanks!

[huapox]

+1

[brandonkal]

I’d like to reset k3s without having to download the container images again.

So beyond the database, I'm not sure which directories should also be restored after a reinstall.

I have a single master+agent server.

[brandonkal]

With some investigation, it appears restoring the state.db file works (as long as the shm and wal files are deleted before restore).

I looked through containerd folders and note that the image blobs are here:

/var/lib/rancher/k3s/agent/containerd/io.containerd.content.v1.content

However, deleting the other folders inside /var/lib/rancher/k3s/agent/containerd
and then restarting the k3s service appears to cause issues. My pods then show:

MountVolume.SetUp failed for volume "cert-manager-cainjector-token-92bgb" : failed to sync secret cache: timed out waiting for the condition

followed by a Pulling image event. This is unfortunate as I'd like to use k3s in a low-bandwidth location and pulling images constantly is painful.

@ibuildthecloud do you have any recommendations or should I just be snapshotting the entire "/var/lib/rancher/k3s" folder?

In my mind I should only need to backup the blobs and the database in order to revert back to a working state with minimal downloads but doesn't appear that way.

[brandond]

@brandonkal this isn't really a backup/restore issue, but it sounds like you want the airgap install option with pre-downloaded images.

https://rancher.com/docs/k3s/latest/en/installation/airgap/

[VladimirAkopyan]

At least there needs to be documentation on how to backup k3s cluster state. It could be really simple, it just needs to be documented!

[eranyanay]

What other files should be backed up aside from the db?
I assume the following are required as well:

/var/lib/rancher/k3s/server/cred
/var/lib/rancher/k3s/server/node-token
/var/lib/rancher/k3s/server/tls
/var/lib/rancher/k3s/server/token

Can anyone confirm?

[gbonnefille]

As information/inspiration, k0s has such feature: https://docs.k0sproject.io/v1.21.2+k0s.0/backup/

[gilesknap]

@eranyanay @VladimirAkopyan @brandond

For information I have tested an approach to backup and restore the sqlite DB and documented it here.

The naive approach saves most of the server directory. But it works even though I tested restoring to a x86 VM instead of the arm64 metal master that did the backup.

[gbonnefille]

@eranyanay @VladimirAkopyan @brandond

For information I have tested an approach to backup and restore the sqlite DB and documented it here.

The naive approach saves most of the server directory. But it works even though I tested restoring to a x86 VM instead of the arm64 metal master that did the backup.

Generally, it is not recommended to backup files of a database as consistency is not guaranteed. A dump could be better.

[gilesknap]

Generally, it is not recommended to backup files of a database as consistency is not guaranteed. A dump could be better.

Good point. I'll update the script to tar the files except the db and then dump the db.

[cwayne18]

Changing this over to a discussion as it's more of a question than an issue.

[defo89]

Maybe k3s team could consider building in native Litestream support (for both backup and restore)?

Currently our approach is to back up static files /etc/rancher and /var/lib/rancher/k3s/server via k8s CronJob to s3 and run Litestream Deployment with replicate command (which also gives us prometheus metrics).

In case of k3s node redeployment, oneshot script will restore static files with aws s3 sync and database file is restored with litestream restore -if-replica-exists /var/lib/rancher/k3s/server/db/state.db.

[AshtonKem]

Some clarity on this would be awesome in the docs. I'm trying to setup backup for my home server, for example, and it's not clear to me if I can safely exclude the containerd directories, which would be nice given the rather large size of the overlayfs directories.

[brandond]

1. Always back up your token.
2. If you're using etcd snapshots, take snapshots and back them up somewhere. If you're using sqlite, stop the server and back up the sqlite files. If you're using an external database, back that up.
3. If you have persistent volumes, use whatever tool your storage vendor recommends.

That's it, that's all you should need.

[urvashipareta15]

We are using local pv and pvc provisioner in k3s. When I take backup of db folder and restore, is there any way to restore that pv? by the way, I used sqlite as db I was able to restore from 1.30 to 1.29 but all my application pods were gone. It was only able to restore the core components of k3s default pods. Any way we can have app pods also.

[brandond]

The PV contents are not stored in the Kubernetes datastore. Restoring the datastore will restore the record of the PV but not its contents. For that you need some other tool like Velero.

I'm not sure what you mean by "core components". All the pods are recorded in the datastore. It sounds like you perhaps didn't stop K3s before backing up the DB file and lost some data.

[urvashipareta15]

when i restored the db folder : traefik,coredns , pod, local provisioner, helm pod they all came back but application pods were gone. When i checked my sysctl k3s status then i saw my data is getting divided into /app/data and /var/lib/rancher , but i just restored the default data directory thinking all will come up as it is..my default was in var/lib/rancher/k3s/server/db . I should have copied the full folder var/lib/rancher/k3s/server rather than just db folder and replaced it . And thanks for telling me about this PV part, you make sense. I will get back if I face issues with backup.