Docker will not work in a eCryptFs mount

[alethenorio]

Output of docker version:

Client:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:38:55 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:38:55 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.11.1
Storage Driver: overlay
 Backing Filesystem: <unknown>
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: host bridge null
Kernel Version: 4.2.0-35-generic
Operating System: Ubuntu 15.10
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.58 GiB
Name: mymachine
ID: GKGG:HESC:P6F2:KQZK:TH2L:ZEFS:KCFG:SWVA:NIL4:SXI4:SCQG:XQT2
Docker Root Dir: /home/byteflinger/.docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):
Docker root dir is in a drive encrypted with eCryptFs

Steps to reproduce the issue:

1. sudo mkdir /etc/systemd/system/docker.service.d
2. sudo vim /etc/systemd/system/docker.service.d/docker.conf
3. Insert the following data

[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon --graph="/home/byteflinger/.docker" --storage-driver=devicemapper --dns 192.168.9.1 -H fd://

1. sudo systemctl daemon-reload
2. sudo systemctl restart docker

Describe the results you received:
I have changed dockers root dir from /var/lib/docker to a directory under my home folder due to space issues and then when I restarted docker and tried to pull an image I got some error. After a bit of searching I stumbled upon a docker comment about AUFS not working under ecryptfs (#19336 (comment)) so I tried changing to devicemapper however the moment I restart the docker service the restart command never comes back and the system starts getting filled until eventually I run out of disk space.
It seems a file called "data" gets created unde the devicemapper folder in docker's root dir and even though ls reports the file to be 0 bytes, du reports that directory to be the one taking all the space.

Describe the results you expected:
Docker to start normally using devicemapper

Additional information you deem important (e.g. issue happens only occasionally):
Able to reproduce every time

[phemmer]

Do you have DM_THIN_PROVISIONING enabled in your kernel?
You can use this script to check your kernel for things docker needs: https://github.com/docker/docker/blob/master/contrib/check-config.sh

[alethenorio]

Thank you. I have run the script and I do have it enabled in my kernel

- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: enabled (as module)
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: enabled (as module)
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

It should be noted that none of the 3 storage drivers I tried (devicemapper, aufs and overlay) will work. After a little more research it is my understanding that even AUFS should work fine under a eCryptFs mounted drive.

[alethenorio]

Still lost guys. I upgraded the underlying OS 16.04 and the issue is still there. I have been using docker for a while and I don't see what I could be done wrong here. I feel like it has something to do with docker and eCryptFs but all other mentions of those 2 together I can find do not seem to have any issues.

[thaJeztah]

What errors are you getting? You describe "fails with some error", but don't describe the error you're seeing.

Note that eCryptfs is not tested in our CI, so we cannot give any guarantee it would work. Perhaps @dustinkirkland has some ideas if this would be possible?

[alethenorio]

Hi @thaJeztah

This is what happens

$ docker build -t myimage .
Sending build context to Docker daemon 1.728 GB
Step 1 : FROM ubuntu:14.04
14.04: Pulling from library/ubuntu
6599cadaf950: Pull complete 
23eda618d451: Pull complete 
f0be3084efe9: Pull complete 
52de432f084b: Pull complete 
a3ed95caeb02: Pull complete 
Digest: sha256:15b79a6654811c8d992ebacdfbd5152fcf3d165e374e264076aa435214a947a3
Status: Downloaded newer image for ubuntu:14.04
 ---> 90d5884b1ee0
Step 2 : RUN apt-get update && apt-get -y install   git     rsync   cmake   lib32z1     lib32ncurses5   lib32bz2-1.0
error creating aufs mount to /home/byteflinger/.docker/aufs/mnt/23ba485d147d0a5fdfdd257a4f5216de6331cd529c706e8a8bf37303c730b31b-init: invalid argument

I have tried deleting /var/lib/docker but no success, still same issue
This is what the daemon logs return when I run it in debug mode and try to run the following command
$docker run -ti --rm ubuntu:14.04 /bin/bash

DEBU[0010] Calling POST /v1.23/containers/create        
DEBU[0010] form data: {"AttachStderr":true,"AttachStdin":true,"AttachStdout":true,"Cmd":["/bin/bash"],"Domainname":"","Entrypoint":null,"Env":[],"HostConfig":{"AutoRemove":false,"Binds":null,"BlkioBps":0,"BlkioDeviceReadBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceWriteIOps":null,"BlkioIOps":0,"BlkioWeight":0,"BlkioWeightDevice":null,"CapAdd":null,"CapDrop":null,"Cgroup":"","CgroupParent":"","ConsoleSize":[0,0],"ContainerIDFile":"","CpuCount":0,"CpuPercent":0,"CpuPeriod":0,"CpuQuota":0,"CpuShares":0,"CpusetCpus":"","CpusetMems":"","Devices":[],"DiskQuota":0,"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":null,"GroupAdd":null,"IpcMode":"","Isolation":"","KernelMemory":0,"Links":null,"LogConfig":{"Config":{},"Type":""},"Memory":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":-1,"NetworkMode":"default","OomKillDisable":false,"OomScoreAdj":0,"PidMode":"","PidsLimit":0,"PortBindings":{},"Privileged":false,"PublishAllPorts":false,"ReadonlyRootfs":false,"RestartPolicy":{"MaximumRetryCount":0,"Name":"no"},"SandboxSize":0,"SecurityOpt":null,"ShmSize":0,"StorageOpt":null,"UTSMode":"","Ulimits":null,"UsernsMode":"","VolumeDriver":"","VolumesFrom":null},"Hostname":"","Image":"ubuntu:14.04","Labels":{},"NetworkingConfig":{"EndpointsConfig":{}},"OnBuild":null,"OpenStdin":true,"StdinOnce":true,"Tty":true,"User":"","Volumes":{},"WorkingDir":""} 
ERRO[0010] Couldn't run auplink before unmount /home/byteflinger/.docker/aufs/mnt/f5da4568662baa85cbce3e5b1d9920c1cc1e242d9f6a3f9163c8acf7987799ce-init: exit status 22 
ERRO[0010] Clean up Error! Cannot destroy container 40e08a3b3660ae36908bf73b4d64313882967fccd0b526415fe2e0ddc7b2417a: No such container: 40e08a3b3660ae36908bf73b4d64313882967fccd0b526415fe2e0ddc7b2417a 
ERRO[0010] Handler for POST /v1.23/containers/create returned error: error creating aufs mount to /home/byteflinger/.docker/aufs/mnt/f5da4568662baa85cbce3e5b1d9920c1cc1e242d9f6a3f9163c8acf7987799ce-init: invalid argument

[unclejack]

This has come up in the past:
#19336
#10380
#7554

eCryptfs is already a stacked file system. It runs as an overlay on top of an existing file system and that means it's not likely to work with any other similar file system on top of it (aufs/overlay on top of eCryptfs on top of anything). eCryptfs also has known issues with NFS. This means eCryptfs has known compatibility issues both with some file systems used below it and with any potential file system on top of it.