This repository has been archived by the owner on Jan 4, 2022. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 41
No more space in btrfs volume /var/lib/machines #66
Labels
Milestone
Comments
It's not a distro-specific issue, as it also happens on Fedora. |
As far as I can see, systemd-machined only grows the filesystem at image import/pull time (https://github.com/systemd/systemd/search?utf8=%E2%9C%93&q=grow_machine_directory&type=). |
dongsupark
pushed a commit
to dongsupark/kube-spawn
that referenced
this issue
Jul 17, 2017
Enlarge the storage pool that is actually a btrfs volume backed by /var/lib/machines.raw. Every btrfs volume would need more space for its metadata as much as the volume grows through CoW operations. /var/lib/machines should have more space than its original image size. Otherwise /var/lib/machines could become full when files start to be written to nspawn containers. Unfortunately it's a temporary workaround. How much space we should add more to the image might depend on estimation during run-time operations. In the long run, systemd itself should be able to reserve more space for the storage pool, every time when it pulls an image to store in the pool. Fixes kinvolk#66
dongsupark
pushed a commit
that referenced
this issue
Jul 17, 2017
Enlarge the storage pool that is actually a btrfs volume backed by /var/lib/machines.raw. Every btrfs volume would need more space for its metadata as much as the volume grows through CoW operations. /var/lib/machines should have more space than its original image size. Otherwise /var/lib/machines could become full when files start to be written to nspawn containers. Unfortunately it's a temporary workaround. How much space we should add more to the image might depend on estimation during run-time operations. In the long run, systemd itself should be able to reserve more space for the storage pool, every time when it pulls an image to store in the pool. Fixes #66
dongsupark
pushed a commit
to dongsupark/kube-spawn
that referenced
this issue
Jul 18, 2017
Enlarge the storage pool that is actually a btrfs volume backed by /var/lib/machines.raw. Every btrfs volume would need more space for its metadata as much as the volume grows through CoW operations. /var/lib/machines should have more space than its original image size. Otherwise /var/lib/machines could become full when files start to be written to nspawn containers. Unfortunately it's a temporary workaround. How much space we should add more to the image might depend on estimation during run-time operations. In the long run, systemd itself should be able to reserve more space for the storage pool, every time when it pulls an image to store in the pool. Fixes kinvolk#66
2 tasks
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
While testing kube-spawn on Ubuntu/Debian, I saw a strange case of btrfs volume
/var/lib/machines
became suddenly read-only mounted, which obviously caused nspawn containers to fail completely.machinectl pull-raw
downloads Container Linux imagecoreos.raw
under/var/lib/machines
, which belongs to a storage pool backed by/var/lib/machines.raw
. Initially it allocates exactly the same size in the pool as the original image. That's fine until kubeadm starts to write files inside nspawn containers, where at some point there's no more space for btrfs metadata. Even when a simpledf
command shows that more space is available, e.g. 30% free, metadata space for btrfs might be already full, as shown withsudo btrfs filesystem usage /var/lib/machines
.It's possible to avoid this by recreating the storage pool with a larger size, and re-execute nspawn containers manually. Then it works fine. But that's obviously not a solution. We need to think about a general way of solving this issue.
The text was updated successfully, but these errors were encountered: