using custom path for storing container images and state

[benp20]

Environmental Info:
K3s Version:
k3s version v1.18.4+k3s1 (97b7a0e)

Node(s) CPU architecture, OS, and Version:
Linux nvidia-desktop 4.9.140-tegra #1 SMP PREEMPT Wed Apr 8 18:15:20 PDT 2020 aarch64 aarch64 aarch64 GNU/Linux

Cluster Configuration:
1 master

Describe the bug:
I'd like to change the default path where containerd under k3s stores container related images, state etc (bydefault /run/k3s/containerd/) since my root partition does not have enough spare space.
I;d like to use my data partition instead.
What is the recommended procedure for doing so with k3s?

I tried referring to https://rancher.com/docs/k3s/latest/en/advanced/#configuring-containerd, but did not find the information there.

Steps To Reproduce:

• Installed K3s:

Expected behavior:

Actual behavior:

Additional context / logs:

[brandond]

The easiest thing to do would probably be to bind-mount your data partition as /run/k3s

[benp20]

Thanks for the suggestion. I mounted the data partition at /run/k3s and now I see overlays for containers getting created at that path (and using the data partition).

Earlier, I was seeing an issue where the node is tainted due to disk pressure and hence the pods are left in a pending state:
Warning FailedScheduling <unknown> default-scheduler 0/1 nodes are available: 1 node(s) had taint {node.kubernetes.io/disk-pressure: }, that the pod didn't tolerate.

I had to reduce thresholds to get around this issue (see my k3s launch command below). I don't see the issue anymore. I am guessing it is because it is (somehow) recognize that more space is available in the new partition?

sudo /usr/local/bin/k3s server --kubelet-arg='eviction-soft=nodefs.available<15%' --kubelet-arg='eviction-soft-grace-period=nodefs.available=60m' --kubelet-arg='eviction-hard=nodefs.available<5%' --kubelet-arg='eviction-soft=nodefs.inodesFree<5%' --kubelet-arg='eviction-soft-grace-period=nodefs.inodesFree=120m' --kubelet-arg='eviction-hard=nodefs.inodesFree<5%'

[benp20]

Update: I still see the node reporting diskpressure when I run pods with larger containers even through /run/k3s is mapped to the data partition already.
Any advice on how to address this problem through use of the data partition? (seemingly it is still using my root partition somewhere which is quite full)

Based on kubernetes documentation (https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/) seems like the issue is with nodefs.

kubelet supports only two filesystem partitions.

The nodefs filesystem that kubelet uses for volumes, daemon logs, etc.
The imagefs filesystem that container runtimes uses for storing images and container writable layers.

Does this have to be mounted to use the partition, and if so how?

Thanks!

[brandond]

Have you considered just throwing a larger SD card or alternative partition layout at the problem? I run k3s on a couple Pi4b nodes with 32GB SD cards without any special configuration.

[jeroenjacobs79]

I'm not sure this solves anything, but on my CentOS server, k3s container images are stored in /var/run/k3s/containerd/, not /run/k3s/containerd/.

[stale]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[sourcehawk]

Is there any solution for this as of today?

[mdrakiburrahman]

K3s uses containerd in a close to vanilla state.

• containerd root: /var/lib/rancher/k3s/agent/containerd (all your images, container files etc)
• containerd state: /run/k3s/containerd (scratch space blown up during containerd reboots)

Practically, what we want is for everything that can grow to be backed by some large drive.

So say, you have a large mounted drive at /mnt, this script below does the trick (the Kubernetes nodefs and imagefs will both be backed by /mnt, and so will kubelet and the local-path persistent volumes) - so your machine's OS directory stays unbloated:

# =======================================
# Storage prep to "/mnt" drive (~500 GB+)
# =======================================
MNT_DIR="/mnt"
K3S_VERSION="v1.25.4+k3s1"

# nodefs
#
KUBELET_DIR="${MNT_DIR}/kubelet"
sudo mkdir -p "${KUBELET_DIR}"

# imagefs: containerd has a root and state directory
#
# - https://github.com/containerd/containerd/blob/main/docs/ops.md#base-configuration
#
# containerd root -> /var/lib/rancher/k3s/agent/containerd
#
CONTAINERD_ROOT_DIR_OLD="/var/lib/rancher/k3s/agent"
CONTAINERD_ROOT_DIR_NEW="${MNT_DIR}/containerd-root/containerd"
sudo mkdir -p "${CONTAINERD_ROOT_DIR_OLD}"
sudo mkdir -p "${CONTAINERD_ROOT_DIR_NEW}"
sudo ln -s "${CONTAINERD_ROOT_DIR_NEW}" "${CONTAINERD_ROOT_DIR_OLD}"

# containerd state -> /run/k3s/containerd
#
CONTAINERD_STATE_DIR_OLD="/run/k3s"
CONTAINERD_STATE_DIR_NEW="${MNT_DIR}/containerd-state/containerd"
sudo mkdir -p "${CONTAINERD_STATE_DIR_OLD}"
sudo mkdir -p "${CONTAINERD_STATE_DIR_NEW}"
sudo ln -s "${CONTAINERD_STATE_DIR_NEW}" "${CONTAINERD_STATE_DIR_OLD}"

# pvs -> /var/lib/rancher/k3s/storage
#
PV_DIR_OLD="/var/lib/rancher/k3s"
PV_DIR_NEW="${MNT_DIR}/local-path-provisioner/storage"
sudo mkdir -p "${PV_DIR_OLD}"
sudo mkdir -p "${PV_DIR_NEW}"
sudo ln -s "${PV_DIR_NEW}" "${PV_DIR_OLD}"

# =======
# Install
# =======
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="$K3S_VERSION" INSTALL_K3S_EXEC="--kubelet-arg "root-dir=$KUBELET_DIR"" sh -
sudo chmod 644 /etc/rancher/k3s/k3s.yaml

When kubernetes comes up, you see everything is backed with a lot of space (in my case, /mnt has 600 GB) - so nodefs and imagefs is nice and full:

And nodefs and imagefs are at 99% - meaning Eviction Manager will not fire under normal circumstances:

[LarsBingBong]

@mdrakiburrahman your way of doing things looks quite cool. However, why not used the documented approach by the K3s project: https://docs.k3s.io/advanced#configuring-containerd - if anything else now you have it for reference.

Have a great day

[mdrakiburrahman]

@mdrakiburrahman your way of doing things looks quite cool. However, why not used the documented approach by the K3s project: https://docs.k3s.io/advanced#configuring-containerd - if anything else now you have it for reference.

Have a great day

Seemed easier to use ln, it's simple and effective tech 😄

Besides K3s, my team manages a bunch of other Kubernetes flavors that doesn't have Rancher's toml file. ln works everywhere.

[LarsBingBong]

@mdrakiburrahman totally fair! I agree with your points. I'm reaching out on Rancher Slack, on the K3s channel, to see if someone from the K3s project can elaborate more on the somewhat documented approach.

Thanks

[brandond]

I personally would probably just set up another mount point and symlink things into place (as @mdrakiburrahman has done) instead of modifying the containerd config template. If you provide your own config template, then you're responsible for keeping it up to date with any changes we make to the default template. We don't do that often, but I feel like it's more fragile than a couple symlinks.

[LarsBingBong]

@brandond and @mdrakiburrahman with your input I'm going with the symlink approach. Thank you very much. Low-key Linux conf. FTW once again 👍🏿 .... have a great day.