Skip to content
This repository has been archived by the owner on Jan 4, 2022. It is now read-only.

No more space in btrfs volume /var/lib/machines #66

Closed
dongsupark opened this issue Jul 14, 2017 · 2 comments
Closed

No more space in btrfs volume /var/lib/machines #66

dongsupark opened this issue Jul 14, 2017 · 2 comments
Milestone

Comments

@dongsupark
Copy link
Member

While testing kube-spawn on Ubuntu/Debian, I saw a strange case of btrfs volume /var/lib/machines became suddenly read-only mounted, which obviously caused nspawn containers to fail completely.

machinectl pull-raw downloads Container Linux image coreos.raw under /var/lib/machines, which belongs to a storage pool backed by /var/lib/machines.raw. Initially it allocates exactly the same size in the pool as the original image. That's fine until kubeadm starts to write files inside nspawn containers, where at some point there's no more space for btrfs metadata. Even when a simple df command shows that more space is available, e.g. 30% free, metadata space for btrfs might be already full, as shown with sudo btrfs filesystem usage /var/lib/machines.

It's possible to avoid this by recreating the storage pool with a larger size, and re-execute nspawn containers manually. Then it works fine. But that's obviously not a solution. We need to think about a general way of solving this issue.

@dongsupark
Copy link
Member Author

It's not a distro-specific issue, as it also happens on Fedora.
Related issue: systemd/systemd#5859

@iaguis
Copy link
Contributor

iaguis commented Jul 17, 2017

As far as I can see, systemd-machined only grows the filesystem at image import/pull time (https://github.com/systemd/systemd/search?utf8=%E2%9C%93&q=grow_machine_directory&type=).

dongsupark pushed a commit to dongsupark/kube-spawn that referenced this issue Jul 17, 2017
Enlarge the storage pool that is actually a btrfs volume backed by
/var/lib/machines.raw. Every btrfs volume would need more space for its
metadata as much as the volume grows through CoW operations.
/var/lib/machines should have more space than its original image size.
Otherwise /var/lib/machines could become full when files start to be
written to nspawn containers.

Unfortunately it's a temporary workaround. How much space we should add
more to the image might depend on estimation during run-time operations.
In the long run, systemd itself should be able to reserve more space
for the storage pool, every time when it pulls an image to store in
the pool.

Fixes kinvolk#66
dongsupark pushed a commit that referenced this issue Jul 17, 2017
Enlarge the storage pool that is actually a btrfs volume backed by
/var/lib/machines.raw. Every btrfs volume would need more space for its
metadata as much as the volume grows through CoW operations.
/var/lib/machines should have more space than its original image size.
Otherwise /var/lib/machines could become full when files start to be
written to nspawn containers.

Unfortunately it's a temporary workaround. How much space we should add
more to the image might depend on estimation during run-time operations.
In the long run, systemd itself should be able to reserve more space
for the storage pool, every time when it pulls an image to store in
the pool.

Fixes #66
dongsupark pushed a commit to dongsupark/kube-spawn that referenced this issue Jul 18, 2017
Enlarge the storage pool that is actually a btrfs volume backed by
/var/lib/machines.raw. Every btrfs volume would need more space for its
metadata as much as the volume grows through CoW operations.
/var/lib/machines should have more space than its original image size.
Otherwise /var/lib/machines could become full when files start to be
written to nspawn containers.

Unfortunately it's a temporary workaround. How much space we should add
more to the image might depend on estimation during run-time operations.
In the long run, systemd itself should be able to reserve more space
for the storage pool, every time when it pulls an image to store in
the pool.

Fixes kinvolk#66
@dongsupark dongsupark added this to the 0.1.0 milestone Jul 31, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants