Provide support for ZFS

[myers]

I would like full support for ZFS in microk8s. If microk8s sees that the root fs is ZFS it should use settings that work on ZFS.

background

With the newest release (492) microk8s switched from docker to containerd. Because I use ZFS on all my disks this did not go well.

I had previously altered args/docker-daemon.json with the directions here: https://docs.docker.com/storage/storagedriver/zfs-driver/ (adding "storage-driver": "zfs")

With the snap autoupgrade this left me with a broken k8s, where none of the pods would start.

After some investigation I attempted to switch containerd to use zfs:

I updated containerd-template.toml changing snapshotter = "overlayfs" to snapshotter = "zfs" and restarted. Then it complained about a missing metadata.db. Creating a new ZFS dataset and mounting it where it expected it made that error go away, but no pods start. There were errors logged but I don't recall what they were.

I apologize for missing information in this issue. The system in question is offline due to a hardware problem, but when It comes back up I will edit in more details.

[myers]

with snapshotter set to "zfs" we get this:

$ kubectl describe pod lukluk-6677ddcbb7-cn8cm
[...]
  Warning  FailedCreatePodSandBox  9s (x9 over 117s)  kubelet, gir       Failed create pod sandbox: rpc error: code = Unknown desc = failed to reserve sandbox name "lukluk-6677ddcbb7-cn8cm_default_a67f2dcd-5a0b-11e9-8f86-002590129c4e_0": name "lukluk-6677ddcbb7-cn8cm_default_a67f2dcd-5a0b-11e9-8f86-002590129c4e_0" is reserved for "51d79673e8314d1b2677c4a6899d4cd6ad08e4754803626a3f30eff39864559f"

[myers]

Would it be possible to mark the change from docker to containerd as a new epoch of the snap? I'm finding containerd very poorly documented compared to docker, making it difficult to debug problem.

[myers]

A solution: the problem is that microk8s started up and attempted to create containers for your pods, but is unable to finish because you are using ZFS and cannot use overlay. It's not enough to just switch the snapshotter and mount a new zfs dataset in the right location. You need to remove all the state from these containers as well.

[myers]

see note at bottom!

so you want to use microk8s on ZFS?

1. Stop microk8s. microk8s.stop
2. Remove old state of containerd. sudo rm -rf /var/snap/microk8s/common/var/lib/containerd.
3. Configure containerd to use ZFS. Edit /var/snap/microk8s/current/args/containerd-template.toml replacing "snapshotter = "overlayfs" to snapshotter = "zfs".
4. Create new zfs dataset for containerd to use. zfs create -o mountpoint=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.zfs $POOL/containerd
5. Start microk8s. microk8s.start

Me: Two years later

Having used the zfs snapshotter for the past 2 years I'd like to throw my $0.02 in. DON'T. The current verison of ZFS tools (like the zfs cli) are not ready for how the zfs snapshotter uses datasets. Docker images are made up of layers, 10's up to 100's of layers per image. Each of the these layers get created as a dataset in zfs. On my system there are over 20,000 of these datasets. zfs list takes over a minute to run. update-grub takes 6 minutes to run because it does zfs list once for every pool in your system (probably something they can improve).

I suspect the snapshotter maybe loosing track of these datasets, as I think 20,000 seems a bit high.

I am now running microk8s on a zvol. That's a part of a zfs pool that you can format as any filesystem. The big downside is that it's a fixed size.

how to run microk8s on zfs using a zvol

The pool we want to store the microk8s image files and container fs is rpool. We want to make a 900GB volume. This will make you download all the images for your pods again.

As root do:

microk8s.stop
mv /var/snap/microk8s/common/var/lib/containerd /var/snap/microk8s/common/var/lib/containerd.backup
mkdir /var/snap/microk8s/common/var/lib/containerd
zfs create -V 900gb rpool/containerd
mkfs.ext4 /dev/zvol/rpool/containerd
echo "/dev/zvol/rpool/containerd /var/snap/microk8s/common/var/lib/containerd ext4 errors=remount-ro 0 2" >> /etc/fstab
mount /var/snap/microk8s/common/var/lib/containerd
microk8s.start

[qlyoung]

@ktsakalozos this is still being hit on a stock install, 9 months later. Is it possible to at least get a doc change? Honestly, snaps don't work on nonstandard homedirs, snaps don't work on nfs, microk8s doesn't work on zfs...all these would be great to document in a "gotchas" section in the user guide.

[ktsakalozos]

@qlyoung thank you for your suggestion. Would you be interested in improving the docs? I think the issues you mention would fit in the "Common issues" in https://microk8s.io/docs/troubleshooting#common-issues . There is a link at the bottom allowing to propose changes (https://github.com/canonical-web-and-design/microk8s.io/edit/master/docs/troubleshooting.md). We appreciate your help. Thank you in advance.

[qlyoung]

@ktsakalozos Ye olde "patches welcome"... 😉
Done.

[ktsakalozos]

We would very much like to automate the fix proposed above. The difficult part in this automation is the pool creation. How would we know which pool to use and would a user want to have pools created in such a transparent way. Also, I am not sure if this automatic pool creation aligns with the future plan of having MicroK8s strictly confined. I think we should leave the pool creation and the snapshotter configuration in the hands of the admin. Yet, failing to function out of the box is not the right behavior either.

Here is how we can move forward. In some testing the "native" snapshotter (snapshotter = "native") showed it could function properly in zfs. Please provide some feedback if it worked for you also. When we start containerd we could detect the filesystem and use the native snapshotter instead of overlay.

We could use something like
df -PTh /var/snap/microk8s/common/var/lib/containerd | awk '{print $2}' in https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/wrappers/run-containerd-with-args#L25 and set the native snapshotter instead of overlay. What do you think? Again, please let me know if the native snapshotter worked for you. I couldn't find much about it. Thanks

[tvansteenburgh]

This was fixed in the 1.18 release.