Can't defrag a 300 MB etcd database on Raspberry Pis after one year of uptime; can't remove Pis from the etcd cluster

[Morveus]

Hello there!

Environmental Info:
K3s Version:
k3s version v1.25.11+k3s1 (582f07cf)

Node(s) CPU architecture, OS, and Version:

• Mix of ARM (running Raspbian) -- 14 nodes
• x86_64 (running Ubuntu 22 LTS) -- 1 node for now

Cluster Configuration:

• All of these are "Servers"
  (I now understand it is a bad idea)

Describe the bug:
After one year of running without maintenance, the etcd database is now 300 MB and none of the Pis will allow it to run defrag. I've seen someone here #5881 complaining about a much bigger database, so I'm not sure this is the actual root of the problem. Also, he gets a timeout after 16 seconds, I (seem to) get a timeout after 3 seconds.

Every time k3s starts on one of the Raspberries, I get

time="2023-07-21T21:28:08+02:00" level=info msg="Starting k3s v1.24.3+k3s1 (990ba0e8)"
time="2023-07-21T21:28:08+02:00" level=info msg="Managed etcd cluster bootstrap already complete and initialized"
time="2023-07-21T21:28:08+02:00" level=info msg="Starting temporary etcd to reconcile with datastore"
....
time="2023-07-21T21:28:45+02:00" level=info msg="Defragmenting etcd database"
{"level":"info","ts":"2023-07-21T21:28:45.774+0200","caller":"v3rpc/maintenance.go:89","msg":"starting defragment"}
{"level":"info","ts":"2023-07-21T21:28:45.785+0200","caller":"backend/backend.go:497","msg":"defragmenting","path":"/var/lib/rancher/k3s/server/db/etcd-tmp/member/snap/db","current-db-size-bytes":305672192,"current-db-size":"306 MB","current-db-size-in-use-bytes":305180672,"current-db-size-in-use":"305 MB"}
{"level":"warn","ts":"2023-07-21T21:28:49.779+0200","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x4000678000/127.0.0.1:2399","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
time="2023-07-21T21:28:49+02:00" level=info msg="Failed to test temporary data store connection: failed to defragment etcd database: context deadline exceeded"

Steps To Reproduce:

• Installed K3s:

Did that for all 15 nodes a year ago:

curl -sfL https://get.k3s.io/ | K3S_TOKEN="xxxxxxxxxxx" K3S_KUBECONFIG_MODE="644" K3S_NODE_NAME="tata" sh -s - --disable servicelb --disable traefik --server https://firstserver:6443/

• Ran a few deployments (home automation, tools, websites etc)
• Waited for a year

Expected behavior:

1. Being able to start the cluster, or defrag etcd, or even just remove all Raspberry Pi nodes and just start with one (more powerful) node to at least run the services until I fix my mistakes
2. or being able to remove all Raspberry Pis from the more powerful node's point of view, and start this node alone. I couldn't find any solution, everything revolves around using kubectl, but I'm stuck at starting k3s, see next section```

Actual behavior:
None of the nodes will start because of the defrag issue; either there's something corrupted, or the defrag timeout is extremely aggressive? I don't see it lasting 16 seconds between the log lines (maybe the countdown has started in the background)

The most powerful node (which I guess could run the entire cluster pods for a while all by itself) won't start because I can't remove all of the other nodes from its point of view. Since k3s won't start, kubectl remove node... is not an option because the API kubectl can talk to is not running (Error from server (ServiceUnavailable): apiserver not ready)

"Powerful" node gets these messages in a loop while trying to start k3s

{"level":"warn","ts":"2023-07-21T22:20:58.039+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"213a13f2a74bcaa","rtt":"0s","error":"dial tcp 10.10.10.104:2380: connect: connection refused"}
{"level":"warn","ts":"2023-07-21T22:20:58.039+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"121c093df1b8c1d3","rtt":"0s","error":"dial tcp 10.10.10.101:2380: connect: connection refused"}
{"level":"warn","ts":"2023-07-21T22:20:58.039+0200","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"dfea4f719c3899e","rtt":"0s","error":"dial tcp 10.10.10.106:2380: connect: connection refused"}

See logs below

Additional context / logs:
k3s logs.txt

Any suggestion, a way to fix this (I've been searching for a few days before submittiing this issue), or even a way to start the Core i3 node all alone (without it trying to ping the other servers which will not reply since they're down) would be greatly appreciated

Thanks in advance!

[brandond]

We generally recommend against using embedded etcd on raspberry pi nodes, at least using their default eMMC or SD Card storage. etcd is very write-intensive and will burn these out in short order, making the filesystem either very slow, or completely unusable.

You should be able to remove the Pi nodes from the etcd cluster by starting the x86_64 node with --cluster-reset to reset the cluster to a single node. Note that this will remove the other nodes from the etcd cluster, but not the Kubernetes cluster. If you want to demote them down to agents instead of servers, you can use kubectl delete node to delete them from the cluster, and then reinstall them as agents. Do all that once you've fixed etcd, of course.

[Morveus]

Thank you very much!