what will be restored from a k3s etc-snapshot ?

[shakibamoshiri]

For testing a snapshot restoration , I installed in single master node with some services running for two days.
Today I stopped the k3s systemd, restored the last auto-taken snapshot, removed the old data path, start the k3s systemd
But I do not see any services, they are gone ! which part are restored (or we should expect to) be restored from a snapshot ?

list of snapshots

root@208:/var/lib/rancher/k3s/server/db/snapshots# ll
total 60324
-rw------- 1 root root 11517984 Aug 12 12:00 etcd-snapshot-208-1723464004
-rw------- 1 root root 12558368 Aug 13 00:00 etcd-snapshot-208-1723507200
-rw------- 1 root root 12558368 Aug 13 12:00 etcd-snapshot-208-1723550405
-rw------- 1 root root 12558368 Aug 14 00:00 etcd-snapshot-208-1723593601
-rw------- 1 root root 12558368 Aug 14 12:00 etcd-snapshot-208-1723636805

stat the last one

root@208:/var/lib/rancher/k3s/server/db/snapshots# stat etcd-snapshot-208-1723636805
  File: etcd-snapshot-208-1723636805
  Size: 12558368  	Blocks: 24536      IO Block: 4096   regular file
Device: 801h/2049d	Inode: 769870      Links: 1
Access: (0600/-rw-------)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-08-14 13:57:14.668301075 +0000
Modify: 2024-08-14 12:00:04.605305678 +0000
Change: 2024-08-14 12:00:04.613306337 +0000
 Birth: 2024-08-14 12:00:04.509297775 +0000

restoring

root@208:/var/lib/rancher/k3s/server/db/snapshots# k3s server --cluster-reset --cluster-reset-restore-path=$PWD/etcd-snapshot-208-1723636805
...
...
...
INFO[0012] Managed etcd cluster membership has been reset, restart without --cluster-reset flag now. Backup and delete ${datadir}/server/db on each peer etcd server and rejoin the nodes

before restoring

root@208:~/k3s# kubectl get svc
NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
hy2-7000-svc    ClusterIP   10.43.253.254   <none>        7000/TCP   2d
hy2-709-1-svc   ClusterIP   10.43.92.50     <none>        7091/TCP   16h
kubernetes      ClusterIP   10.43.0.1       <none>        443/TCP    3d5h

after restoring and start k3s

root@208:/var/lib/rancher/k3s/server/db/snapshots# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP   22m

version tested

k3s --version 
k3s version v1.30.3+k3s1 (f6466040)
go version go1.22.5

and is this relevant ?

• [BUG] k3s etcd snapshot restore from s3 wipes everything from cluster #42251

[brandond]

Today I stopped the k3s systemd, restored the last auto-taken snapshot, removed the old data path, start the k3s systemd

The instructions say to do that on OTHER servers (the peers), not on the server you just restored the snapshot on.

Backup and delete ${datadir}/server/db on each peer etcd server and rejoin the nodes

You just deleted everything you restored. Don't do that.

[shakibamoshiri]

This is the confusing part , because first I did not delete that path (I read the doc that said do it on other master nodes), I started the k3s systemd and it did not get started and showed error : k3s Failed to test data store connection: this server is a not a member of the etcd cluster then I found this link server is a not a member of the etcd cluster #5352 -- I did and removed the path /var/lib/rancher/k3s/server/db/etcd then k3s systemd started successfully :|

Please notice I tested a single master node and no worker https://docs.k3s.io/cli/etcd-snapshot?etcdsnap=Single+Server -- since the k3s systemd failed to started I deleted the path /var/lib/rancher/k3s/server/db/etcd and it got working .

[brandond]

The "server is not a member of the etcd cluster" error should tell you what is in the cluster, vs what it is expecting to see. Are you overriding the node IP or something like that? If you have --node-ip set in the args in the systemd unit, you should pass those same flags when doing a cluster reset, otherwise you'll end up with etcd running with the default IP during the reset, and your requested IP when you start the service - and etcd will fail to start because the IP of the cluster member has changed.

[shakibamoshiri]

If you have --node-ip set in the args in the systemd unit, you should pass those same flags when doing a cluster reset, otherwise you'll end up with etcd running with the default IP during the reset, and your requested IP when you start the service - and etcd will fail to start because the IP of the cluster member has changed.

I see , and you know -- it is not mentioned in the doc , thank you I will rerun a test with the same configuration

I do not think is is needed to say that this part is risky and crucial for admins so please make it clear and give examples in the documentation (if PR are accepted for docs I can do it)

[brandond]

If you pass config to the server via CLI args in the systemd unit, you have to pass those same args to the server when running it from the CLI to do a reset.

If you pass config via the config file (/etc/rancher/k3s/config.yaml) it will be loaded automatically, without having to remember to pass it by hand when you do a reset.

It sounds like you probably would prefer to use the config file.