Can't use Docker on ZFS

[brandond]

From @sonicaj

Environmental Info:
K3s Version:
K3s 1.20.2

Node(s) CPU architecture, OS, and Version:
N/A

Cluster Configuration:
N/A

Describe the bug:
While using ZFS on debian and K3s with docker, I am unable to get k3s working as the snapshotter value is being validated and the validation fails.

Steps To Reproduce:

• Install Docker on host with ZFS root filesystem
• Start K3s with --docker
• Note error about snapshotter

Expected behavior:
No snapshotter warning, as not using containerd

Actual behavior:
K3s fails to start, suggests different snapshotter

Additional context / logs:

[rancher-max]

@sonicaj I am unable to reproduce this issue on k3s v1.20.2. I haven't used zfs before, so I'd appreciate any assistance in what you're doing differently so that I can validate it is definitely fixed with no other k3s regressions for your needs. Here are my steps:

1. Using Linode, I brought up a fresh Deb 10 VM:

# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

2. Added following sources to install zfs in /etc/apt/sources.list

deb http://deb.debian.org/debian buster-backports main contrib non-free
deb-src http://deb.debian.org/debian buster-backports main contrib non-free

3. Installed zfs:

apt update
apt install linux-headers-`uname -r`
apt install -t buster-backports zfsutils-linux

4. Started zfs:

sudo modprobe zfs
systemctl restart zfs-import-cache.service
systemctl restart zfs-mount.service
systemctl restart zfs-share.service

sudo truncate -s 100M /root/z1
sudo truncate -s 100M /root/z2
sudo zpool create tank /root/z1 /root/z2
sudo zpool scrub tank
sudo zpool status

reboot

5. Install docker using https://docs.docker.com/engine/install/debian/#install-using-the-repository
6. Install k3s: curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--docker" INSTALL_K3S_VERSION=v1.20.2+k3s1 sh -

Seemed to install fine, all nodes and pods were up and running. Any ideas?

[koaps]

I have the same issue, but I'm trying to run k3s in docker on a debian system with ZFS root, I was looking at these issues also:

#66
#2779

I was going to try the ext4 disk trick to see if it works better.

I did try following this post: https://blog.nobugware.com/post/2019/k3s-containterd-zfs/
but ran into the snapshotter issue.

@rancher-max In your test, are you using ZFS for the root filesystem? if not, then /var/lib for docker or containerd aren't on a ZFS based filesystem.

There's a GIST here that was done for K3s on Ubuntu 20 with encrypted ZFS root: https://gist.github.com/koaps/7f7bc10bf7a0d326769c8ad7ce674036

You can use the scripts from there to setup a test node with root on ZFS, it would be a better testing setup for you, I would just leave out the encryption stuff.

For instance this is how I created my root pool:

zpool create \
    -o ashift=12 \
    -O acltype=posixacl -O canmount=off -O compression=lz4 \
    -O dnodesize=auto -O normalization=formD -O relatime=on \
    -O xattr=sa -O mountpoint=/ -R /mnt \
    rpool raidz2 ${DISK1}-part4 ${DISK2}-part4 cache ${DISK0}

Then I created the rest of mounts like it's done in the GIST.

[H4R0]

I tried to setup k3s on zfs root some months ago but had to fallback to ext4 and checked #66 every couple weeks and just saw this issue has been created.

I just tried again with @rancher-max "curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--docker" INSTALL_K3S_VERSION=v1.20.2+k3s1 sh -" but it still fails to work.

@rancher-max It seems like you are only creating a zfs pool but not actually using it in any way. I would recommend you to install proxmox which installer has built-in support for zfs on root, it's debian with a ubuntu kernel. With that you should easily be able to reproduce. In the installer storage section select, Options -> Filesystem zfs (raid0).

This does not only affect --docker but the normal setup as well.

Feb 21 23:17:20 node1 k3s[664]: time="2021-02-21T23:17:20.044460702Z" level=error msg="Failed to retrieve agent config: \"overlayfs\" snapshotter cannot be enabled for \"
Feb 21 23:17:20 node1 k3s[664]: time="2021-02-21T23:17:20.266609130Z" level=info msg="Waiting for control-plane node node1 startup: nodes \"node1\" not found"
Feb 21 23:17:21 node1 k3s[664]: time="2021-02-21T23:17:21.275582114Z" level=info msg="Waiting for control-plane node node1 startup: nodes \"node1\" not found"
...

I was searching for alternative storage implementations back then but could not find anything for k3s in rancher docs.

Note that only overlayfs is affected by missing support from zfs, docker itself migrated to overlayfs2 a while ago which does work with zfs.

There has been some work on the zfs side to support overlayfs but it has been stale for some time and I think it is unlikely it will be implemented soon, nor the right way since there seem to be better alternatives. openzfs/zfs#9414

Just my two cents.

[brandond]

@H4R0 either way this should be fixed by the PR above. ZFS should work with containerd or external docker.

[H4R0]

Using v1.20.4-rc1+k3s1 indeed works.

[rancher-max]

@H4R0 Thank you for the help. I was able to recreate this issue by creating an AMI using https://github.com/jen20/packer-ubuntu-zfs. Using curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--docker" INSTALL_K3S_VERSION=v1.20.4+k3s1 sh - does install k3s now. I validated this is also successful on an HA k3s install with both servers and agents. I ran some regression checks including workload and volume tests and all are functioning as expected.

This issue is specifically related to docker, however, as mentioned in #2920 (comment) this affects the default install of k3s without docker, which is still NOT fixed in v1.20.4. We will keep an eye on further ZFS support for the future, especially if it moves out of experimental and becomes the default in Ubuntu.