dqlite/cluster-init failing due to zfs

[iwilltry42]

Hey, since a short while, multi-master setups of k3s using dqlite are failing, because the initializing master node does not come up anymore.
I'm running k3s inside docker (for k3d development).
I recently upgraded my OS to Ubuntu 20.04 (details below) and ZFS

Version:

Docker Image Versions tested:

• rancher/k3s:v1.17.3-k3s1
• rancher/k3s:v1.17.4-k3s1

k3s arguments:

• --cluster-init

Describe the bug

• server startup fails with the following logs:

INFO[2020-04-26T14:57:36.013365090Z] Starting k3s v1.17.4+k3s1 (3eee8ac3)         
FATA[2020-04-26T14:57:36.129858830Z] starting kubernetes: preparing server: start cluster and https: raft_init(): io: create I/O capabilities probe file: posix_allocate: operation not supported on socket 

To Reproduce

• docker run --rm -it rancher/k3s:v1.17.4-k3s1 server --cluster-init

Expected behavior

• Normal startup with server initializing dqlite setup

Further information

• OS: Ubuntu 20.04 LTS
• Kernel: Linux 5.4.0-26-generic x86_64
• Docker: 19.03.8

[iwilltry42]

Most probably caused by zfs, which I missed during research 🙄
Solved for me via workaround described here: #66 (comment)

$ zfs create -s -V 50GB rpool/ROOT/docker
...
$ mkfs.ext4 /dev/zvol/rpool/ROOT/docker 
...
$ echo "/dev/zvol/rpool/ROOT/docker /var/lib/docker ext4 defaults 0 0" >> /etc/fstab
...
$ reboot now