Cannot start Docker with zfs storage driver on Ubuntu 16.04 Xenial

[parisk]

Description

I try to get the Docker Engine work with zfs as its storage driver according to the official instructions but it's stuck to AUFS with ZFS as the backing file system.

Steps to reproduce the issue (on Digital Ocean):

1. Provision a new Ubuntu 16.04 VM
2. Set up ZFS as instructed at http://docs-stage.docker.com/engine/userguide/storagedriver/zfs-driver/
3. Install the Docker Engine as instructed at https://docs.docker.com/engine/installation/linux/ubuntulinux
4. Set DOCKER_OPTS="--storage-driver=zfs" in /etc/default/docker and restart Docker Engine with service docker restart

Describe the results you received:

Docker Engine loads with AUFS storage driver (with zfs as the backing file system).

I expected the Docker Engine to load using

Describe the results you expected:

I expected the Docker Engine to start with zfs as its storage driver as described in http://docs-stage.docker.com/engine/userguide/storagedriver/zfs-driver/

Additional information you deem important:

I tried the following with no results:

1. Stopping the Docker Engine (double checked that it stopped)
2. Deleted everything in /var/lib/docker (run rm -rf /var/lib/docker*)
3. Start Docker engine

Also worth mentioning that /var/lib/docker uses ZFS:

root@ubuntu-1gb-nyc1-01:~# df -h /var/lib/docker/
Filesystem           Size  Used Avail Use% Mounted on
zpool-docker/docker  9.7G  128K  9.7G   1% /var/lib/docker

Output of zfs list

NAME                  USED  AVAIL  REFER  MOUNTPOINT
zpool-docker          202K  9.63G    19K  /zpool-docker
zpool-docker/docker   122K  9.63G   122K  /var/lib/docker

Output of docker version:

Client:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 22:01:48 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.3
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   6b644ec
 Built:        Wed Oct 26 22:01:48 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.12.3
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: zfs
 Dirs: 0
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host bridge null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-47-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 992.5 MiB
Name: ubuntu-1gb-nyc1-01
ID: Z6QZ:IDVT:EEDV:FWDV:LTNM:I3KS:6UYE:OQSU:HU3O:WFIT:V3Q7:ZIYP
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):

I am using a 1GB 16.04.1 x64 DigitalOcean droplet with a 10GB block storage volume, as seen below

[thaJeztah]

Thanks for reporting! It looks like this is a documentation issue; the /etc/default/docker file is only used on systems using "upstart" or "sysvinit" as unit system (such as ubuntu 14.04); there's a comment at the top of that file explaining that.

Ubuntu 16.04 uses systemd, and requires a different method to set daemon options. You can do so through systemd drop-in filed, or (easier), use a daemon.json configuration file; https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file

Create the file at /etc/docker/daemon.json if it does not exist, and add "storage-driver":"zfs".

The file needs to be valid JSON, otherwise the daemon won't start. If this is the only option you're setting, the file should look like this;

{"storage-driver": "zfs"}

After saving the file, (re)start docker;

sudo systemctl restart docker

Hope this helps

[parisk]

Hi @thaJeztah thanks for replying! This solves my issue, so I will move forward with closing this.

Two questions though:

1. Should this be clarified in the ZFS documentation?
2. Why didn't the daemon automatically set the storage driver to zfs, as it was mentioned in the documentation?

[thaJeztah]

You're welcome!

Why didn't the daemon automatically set the storage driver to zfs, as it was mentioned in the documentation?

The daemon has a priority list in which storage drivers are tried (if no storage driver is set in the configuration); https://github.com/docker/docker/blob/299beff1c4e2406d1e96c90a9e86e0770dd4342d/daemon/graphdriver/driver_linux.go#L55-L63

If your system supports aufs, it is selected as a default. I assume the person writing that part of the documentation did not have the linux-image-extra package installed, hence no aufs support, and the next working driver (zfs) was selected.

Should this be clarified in the ZFS documentation?

Yes, I think it should. I think we should use the "daemon.json" file as the default way of configuring the daemon in the documentation, as it works with all init systems. Also, I think it's best practice to explicitly set the storage driver to use, instead of relying on the "automatic" selection, which is just there to allow bootstrapping docker without configuration.

Would you be interested in contributing, and opening a pull request in the documentation repository? https://github.com/docker/docker.github.io/blob/master/engine/userguide/storagedriver/zfs-driver.md

Otherwise, please open an issue in that repository's issue tracker, so that we don't forget 😄

[pirmax]

{"storage-driver": "zfs"}

After saving the file, (re)start docker;

sudo systemctl restart docker

I've try to make this, but when I want restart Docker, I've this error:

Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.

Jan 22 10:16:49 discourse systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jan 22 10:16:49 discourse systemd[1]: docker.service: Failed with result 'exit-code'.
Jan 22 10:16:49 discourse systemd[1]: Failed to start Docker Application Container Engine.
Jan 22 10:16:49 discourse systemd[1]: docker.service: Service hold-off time over, scheduling restart.
Jan 22 10:16:49 discourse systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Jan 22 10:16:49 discourse systemd[1]: Stopped Docker Application Container Engine.
Jan 22 10:16:49 discourse systemd[1]: docker.service: Start request repeated too quickly.
Jan 22 10:16:49 discourse systemd[1]: docker.service: Failed with result 'exit-code'.
Jan 22 10:16:49 discourse systemd[1]: Failed to start Docker Application Container Engine.

[thaJeztah]

@pirmax is you underlying filesystem zfs as well? Note that journalctl -xe usually doesn't give much information; journalctl -fu docker.service and journalctl -fu containerd.service should show more details as to why they fail to start.

[pirmax]

I've this errors:

Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.805312140Z" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.813204800Z" level=info msg="loading plugin "io.containerd.differ.v1.walking"..." type=io.containerd.differ.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.813337179Z" level=info msg="loading plugin "io.containerd.gc.v1.scheduler"..." type=io.containerd.gc.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.813435973Z" level=info msg="loading plugin "io.containerd.service.v1.containers-service"..." type=io.containerd.service.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.813533131Z" level=info msg="loading plugin "io.containerd.service.v1.content-service"..." type=io.containerd.service.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.813614143Z" level=info msg="loading plugin "io.containerd.service.v1.diff-service"..." type=io.containerd.service.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.813693438Z" level=info msg="loading plugin "io.containerd.service.v1.images-service"..." type=io.containerd.service.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.813774188Z" level=info msg="loading plugin "io.containerd.service.v1.leases-service"..." type=io.containerd.service.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.813850469Z" level=info msg="loading plugin "io.containerd.service.v1.namespaces-service"..." type=io.containerd.service.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.813927253Z" level=info msg="loading plugin "io.containerd.service.v1.snapshots-service"..." type=io.containerd.service.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.815025297Z" level=info msg="loading plugin "io.containerd.runtime.v1.linux"..." type=io.containerd.runtime.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.815162277Z" level=info msg="loading plugin "io.containerd.runtime.v2.task"..." type=io.containerd.runtime.v2
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.815289118Z" level=info msg="loading plugin "io.containerd.monitor.v1.cgroups"..." type=io.containerd.monitor.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.815570056Z" level=info msg="loading plugin "io.containerd.service.v1.tasks-service"..." type=io.containerd.service.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.815656651Z" level=info msg="loading plugin "io.containerd.internal.v1.restart"..." type=io.containerd.internal.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.815752613Z" level=info msg="loading plugin "io.containerd.grpc.v1.containers"..." type=io.containerd.grpc.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.815866979Z" level=info msg="loading plugin "io.containerd.grpc.v1.content"..." type=io.containerd.grpc.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.815944791Z" level=info msg="loading plugin "io.containerd.grpc.v1.diff"..." type=io.containerd.grpc.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.816021349Z" level=info msg="loading plugin "io.containerd.grpc.v1.events"..." type=io.containerd.grpc.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.816097416Z" level=info msg="loading plugin "io.containerd.grpc.v1.healthcheck"..." type=io.containerd.grpc.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.816173919Z" level=info msg="loading plugin "io.containerd.grpc.v1.images"..." type=io.containerd.grpc.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.816249224Z" level=info msg="loading plugin "io.containerd.grpc.v1.leases"..." type=io.containerd.grpc.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.816325505Z" level=info msg="loading plugin "io.containerd.grpc.v1.namespaces"..." type=io.containerd.grpc.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.816403110Z" level=info msg="loading plugin "io.containerd.internal.v1.opt"..." type=io.containerd.internal.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.816532385Z" level=info msg="loading plugin "io.containerd.grpc.v1.snapshots"..." type=io.containerd.grpc.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.816612273Z" level=info msg="loading plugin "io.containerd.grpc.v1.tasks"..." type=io.containerd.grpc.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.816693093Z" level=info msg="loading plugin "io.containerd.grpc.v1.version"..." type=io.containerd.grpc.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.816769301Z" level=info msg="loading plugin "io.containerd.grpc.v1.introspection"..." type=io.containerd.grpc.v1
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.816927187Z" level=info msg=serving... address="/run/containerd/containerd.sock"
Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.816998309Z" level=info msg="containerd successfully booted in 0.023649s"
Jan 22 13:04:40 discourse systemd[1]: containerd.service: Failed to reset devices.list: Operation not permitted
Jan 22 13:04:41 discourse systemd[1]: containerd.service: Failed to reset devices.list: Operation not permitted
Jan 22 13:04:41 discourse systemd[1]: containerd.service: Failed to reset devices.list: Operation not permitted
Jan 22 13:04:41 discourse systemd[1]: containerd.service: Failed to reset devices.list: Operation not permitted
Jan 22 13:04:41 discourse systemd[1]: containerd.service: Failed to reset devices.list: Operation not permitted
Jan 22 13:04:42 discourse systemd[1]: containerd.service: Failed to reset devices.list: Operation not permitted

I don't want use zfs because Discourse want use overlay2 and I can't change that.

[thaJeztah]

I don't want use zfs because Discourse want use overlay2 and I can't change that.

But in your comment above you configured the docker daemon to use zfs, or do I misunderstand your comment?

Note that this message can be harmless; container checks all the available storage-drivers to see what's supported;

Jan 22 13:04:40 discourse containerd[1828]: time="2019-01-22T13:04:40.805312140Z" level=warning msg="could not use snapshotter zfs in metadata plugin" error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"

Are you running docker inside LXC? if so, there's some existing issues there (related to configuration of LXC); see for example lxc/incus#2004

[pirmax]

Yes, whatever I set up in daemon.json, Docker refuses to restart / start. If I delete it, Docker starts correctly. I do not want to use vfs but overlay2 for example.

[thaJeztah]

The Failed to reset devices.list: Operation not permitted error looks to be the culprit, but can't tell the reason why that occurs (other than the LXC situation I mentioned); make sure to check your system logs as well (perhaps SELinux or AppArmor block something), also make sure you have the current version of docker installed (Docker CE 18.09.1 at this point)

However, your situation looks to be unrelated to the original issue, so if you can't find the reason from the above steps, and think there's a bug at hand, please open a new ticket instead, and provide the details that are requested in the issue template.