Excessive memory usage among HA servers during member restart

[kskalski]

Environmental Info:
K3s Version: v1.27.1-rc1+k3s1 (but also observed in earlier versions, e.g. v1.25.9+k3s1)

Node(s) CPU architecture, OS, and Version:
Linux vps-96dabacf 5.15.0-70-generic #77-Ubuntu SMP Tue Mar 21 14:02:37 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
3 servers running in HA mode, all with config

ExecStart=/usr/local/bin/k3s \
    server \
	'--disable=traefik' \
	'--disable=servicelb' \

two of them have 2 cores + 4GB ram, 1 has 4 cores and 8GB ram, they run on VPSed in the same cluster with internal IPs in range 10.42.*.* and public IPs in range 146.*.*.*, public IPs are used in etcd membership list.

Describe the bug:
Normally k3s in HA server mode in my setup uses around 700-800MB of ram, however recently I observed the memory usage growth to 3000MB or more, which so far seems to be related to other servers in the membership list being restarted.
This took me by surprise: the memory usage didn't fit with my other payload -> servers started swapping activity -> cpu saturated -> etcd/raft messages started getting high latency or drops -> lots of error messages flooded logs and eventually the server gets killed - maybe by oom killer or by accumulating some errors and deciding it's enough.

Note the proper provisioning of resources to servers require me to keep all the slack memory free, so effectively it requires counting k3s server reserved memory at ~3GB plus some margin.

Steps To Reproduce:
My installation is a pre-existing set-up which I upgraded from earlier versions, and previously I disabled flannel and used calico with wireguard, only recently re-configured cluster to use default flannel backend and disabled calico (not sure this makes a difference, but possibly there is some crust accumulated in my cluster that made it behave this way, though now it happens on all the nodes).

• install 3 k3s servers pointing the subsequent servers to the first one using --server=https://public_ip:6443 flag
• restart one of servers
• observe other servers memory usage grow from ~800MB to ~2000-3000MB, also cpu usage spikes to 100% on all cores, but if machine has enough ram it goes down after a few seconds (high RES memory stays though)

vs
most recently restarted server:

Expected behavior:
Ideally restarting servers and running them in any situation (stable or degraded health of other members) should keep memory usage at some reasonable level. Clearly, establishing new leader requires additional operations outside of regular function in stable mode, but I think it shouldn't be more than tens of MB, maybe 100-200 MB.

Actual behavior:
It seems that in stable situation the most recent restarted / re-joining server grows to its regular 800MB and when no more restarts / communication issues arise, it stays this way. However in the meantime the other 2 servers whether they used <1GB or more, they start rapid memory usage expansion, usually to 2300MB, sometimes more. After situation stabilizes, they usually stay at that elevated memory usage.

Additional context / logs:
I'm attaching log from the server experiencing memory growth (the healthy one that observed its peer being restarted)
file:///home/nazgul/rep/gradcap/backtest-py/log.txt

[brandond]

Check out https://etcd.io/docs/v3.5/tuning/ - in particular the sections on the snapshot count. However, since you're seeing the memory utilization only go up when one of the nodes is offline, I suspect that you might be running into issues with the peer's buffer growing, and golang not freeing the memory back to the host operating system once the peer comes back online.

You might try:

• disabling swap on your node. Best practice for Kubernetes is to disable swap, especially since you mention that high swap IO is what eventually kills your node.
• Setting a memory limit on the k3s systemd service. Without a memory limit, golang will not aggressively free memory back to the host OS, which will make your swap problems worse.