Docker bails on systems with btrfs + SELinux w/o regard for SELinux status

[dfarrell07]

I'm seeing "Permission denied" errors from shared library code on a Fedora 20 systems using btrfs.

I've replicated this on two bare metal up-to-date Fedora 20 systems running btrfs, including one that was totally fresh. As a control, everything works fine on an up-to-date Fedora 20 system where Docker uses the devicemapper storage driver (running on an OpenStack deployment, also a very fresh install).

Steps to reproduce:

sudo yum update -y
sudo yum install docker-io -y
sudo systemctl start docker
sudo docker pull fedora
sudo docker run fedora <some command>

The exact result on btrfs storage driver systems varies with the command I attempt to run, but generally some shared lib fails to load with a permission denied error.

Echo (sudo docker run fedora echo "hello world") causes:

echo: error while loading shared libraries: libc.so.6: cannot open shared object file: Permission denied

A Bash shell (sudo docker run fedora /bin/bash) causes:

/bin/bash: error while loading shared libraries: libtinfo.so.5: cannot open shared object file: Permission denied

Results on devicemapper storage driver system are as expected.

The output of the three systems is identical for docker version:

[~]$ sudo docker version
Client version: 1.1.2
Client API version: 1.13
Go version (client): go1.2.2
Git commit (client): d84a070/1.1.2
Server version: 1.1.2
Server API version: 1.13
Go version (server): go1.2.2
Git commit (server): d84a070/1.1.2

Output of docker info for the (failing) fresh btrfs system:

[~]$ sudo docker info
Containers: 4
Images: 3
Storage Driver: btrfs
Execution Driver: native-0.2
Kernel Version: 3.11.10-301.fc20.x86_64

Output of docker info for the (working) nearly-fresh devicemapper system:

[~]$ sudo docker info
Containers: 3
Images: 5
Storage Driver: devicemapper
 Pool Name: docker-252:1-131261-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 975.2 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 1.3 Mb
 Metadata Space Total: 2048.0 Mb
Execution Driver: native-0.2
Kernel Version: 3.11.10-301.fc20.x86_64

The (working) devicemapper install and the (failing) totally fresh btrfs install have the same kernel:

# Fresh (failing) btrfs F20 install
[~]$ uname -a
Linux localhost.localdomain 3.11.10-301.fc20.x86_64 #1 SMP Thu Dec 5 14:01:17 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
# Nearly-fresh (working) devicemapper F20 install
[~]$ uname -a
Linux dfarrell-odl-docker 3.11.10-301.fc20.x86_64 #1 SMP Thu Dec 5 14:01:17 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

The (failing) not-fresh btrfs install has a newer kernel:

# Not-fresh (failing) btrfs F20 install
[~]$ uname -a
Linux localhost.localdomain 3.15.10-201.fc20.x86_64 #1 SMP Wed Aug 27 21:10:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Note that I can prevent this error, even on the btrfs systems, by either putting SELinux in Permissive mode (sudo setenforce 0, as described by the OP here) or passing the --privileged flag (sudo docker run --privileged fedora echo "hello world", as described here).

# Example of proper behavior on fresh btrfs system when SELinux is in Permissive mode
[~]$ getenforce
Enforcing
[~]$ sudo setenforce 0
[~]$ getenforce
Permissive
[~]$ sudo docker run fedora echo "hello world"
hello world
[~]$ sudo setenforce 1
[~]$ sudo docker run fedora echo "hello world"
echo: error while loading shared libraries: libc.so.6: cannot open shared object file: Permission denied

# Example of proper behavior on not-fresh btrfs system when --privileged flag passed
[~]$ sudo getenforce
Enforcing
[~]$ sudo docker run --privileged fedora echo "hello world"
hello world

My libselinux version is the same on all three systems:

[~]$ sudo yum info libselinux
Installed Packages
Name        : libselinux
Arch        : x86_64
Version     : 2.2.1
Release     : 6.fc20
<snip>

This Docker GitHub Issue comment mentioned switching from aufs to devicemapper as a fix for a seemingly similar issue. That may support the theory that this is a btrfs-related issue.

[dfarrell07]

Note that issue #7318 seems to be related.

The "fix" proposed there was to just disable SELinux. This doesn't fit the use case of folks who would like to use Docker in production (likely okay for the Docker *-dev versions they were using at the time).

[estesp]

This section on file system support from the following article published on 9/3 may be helpful (although it doesn't fix your problem): https://opensource.com/business/14/9/security-for-docker

File system support

SELinux currently will only work with the device mapper back end. SELinux does not work with BTRFS. BTRFS does not support context mount labeling yet, which prevents SELinux from relabeling all content when the container starts via the mount command. Kernel engineers are working on a fix for this and potentially Overlayfs if it gets merged into the container.

[dfarrell07]

SELinux currently will only work with the device mapper back end. SELinux does not work with BTRFS. BTRFS does not support context mount labeling yet, which prevents SELinux from relabeling all content when the container starts via the mount command. Kernel engineers are working on a fix for this and potentially Overlayfs if it gets merged into the container.

That's great info, thank you @estesp.

I found the relevant bug on RHEL7's Bugzilla. It looks like Daniel Walsh has recently assigned someone to work on it, so there should be a fix coming from upstream.

#6452 seems to be where the issue was introduced.

I'm up for closing this, now that it's clear that it needs to be addressed upstream (wasn't clear to me until this morning) and it's linked to the primary bug report.

[dfarrell07]

It'd be awesome if we could document this issue a bit more clearly, maybe on the install pages for distros that support btrfs. I burned ~12 hours on it yesterday, so the "duplicate no effort" part of me is screaming to help others avoid that pain.

[dfarrell07]

Pull request #7956 includes doc updates for the issue described here.

[dfarrell07]

After updating to 1.2.0, I'm seeing #7709.

[~]$ sudo docker version
Client version: 1.2.0
Client API version: 1.14
Go version (client): go1.2.2
Git commit (client): fa7b24f/1.2.0
OS/Arch (client): linux/amd64
2014/09/14 16:42:45 Cannot connect to the Docker daemon. Is 'docker -d' running on this host?
[~]$ sudo systemctl start docker
Job for docker.service failed. See 'systemctl status docker.service' and 'journalctl -xn' for details.

journalctl -xn gives this relevant info:

Sep 14 16:43:20 localhost.localdomain docker[6368]: 2014/09/14 16:43:20 SELinux is not supported with the BTRFS graph driver!

This is actually worse than the error at 1.1.2, as it fails without regard to the status of SELinux.

[~]$ getenforce
Permissive

The pull request that introduced this behavior, #6452, claims that it is checking for the status of SELinux. Based on what I'm seeing, that doesn't appear to be the case.

[dfarrell07]

Now that #7956 and #7709 are closed, this is the correct place to talk about the SELinux + btrfs issue.

As a recap, the 1.2.0 failure behavior is that Docker bails on btrfs systems without regard for the status of SELinux. The expected behavior is that if SELinux is in Permissive mode, Docker should continue.

[voxadam]

I believe that I'm hitting this bug as well. I'm running a fresh Fedora 20 install on btrfs. Are there _any_ workarounds aside from running Docker in a VM (which kind of defeats the purpose of Docker)?

OS: Fedora 20 (fresh install)
Kernel: 3.16.2-200.fc20.x86_64 #1 SMP
Docker: 1.2.0 (2.fc20)
libselinux: 2.2.1 (6.fc20)

 adam@dekatron  ~/bin  sudo setenforce 0                                                                                                                        
 adam@dekatron  ~/bin  sudo getenforce                                                                                                                          
Permissive
 adam@dekatron  ~/bin  sudo systemctl start docker
Job for docker.service failed. See 'systemctl status docker.service' and 'journalctl -xn' for details.
 ✘ adam@dekatron  ~/bin  sudo journalctl -xn        
-- Logs begin at Wed 2014-09-17 14:41:46 PDT, end at Fri 2014-09-19 16:16:52 PDT. --
Sep 19 16:16:49 dekatron.voxadam.com sudo[14285]: adam : TTY=pts/2 ; PWD=/home/adam/bin ; USER=root ; COMMAND=/bin/systemctl start docker
Sep 19 16:16:49 dekatron.voxadam.com systemd[1]: Starting Docker Application Container Engine...
-- Subject: Unit docker.service has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit docker.service has begun starting up.
Sep 19 16:16:49 dekatron.voxadam.com docker[14288]: 2014/09/19 16:16:49 docker daemon: 1.2.0 fa7b24f/1.2.0; execdriver: native; graphdriver:
Sep 19 16:16:49 dekatron.voxadam.com docker[14288]: [072bebee] +job serveapi(fd://)
Sep 19 16:16:49 dekatron.voxadam.com docker[14288]: [info] Listening for HTTP on fd ()
Sep 19 16:16:49 dekatron.voxadam.com docker[14288]: 2014/09/19 16:16:49 SELinux is not supported with the BTRFS graph driver!
Sep 19 16:16:49 dekatron.voxadam.com systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Sep 19 16:16:49 dekatron.voxadam.com systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit docker.service has failed.
-- 
-- The result is failed.
Sep 19 16:16:49 dekatron.voxadam.com systemd[1]: Unit docker.service entered failed state.
Sep 19 16:16:52 dekatron.voxadam.com sudo[14306]: adam : TTY=pts/2 ; PWD=/home/adam/bin ; USER=root ; COMMAND=/bin/journalctl -xn
 adam@dekatron  ~/bin  uname -a
Linux dekatron.voxadam.com 3.16.2-200.fc20.x86_64 #1 SMP Mon Sep 8 11:54:45 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
 adam@dekatron  ~/bin  

[brightonbob]

Same bug (same error - SELinux is not supported with the BTRFS graph driver!) on Fedora 21 (fresh install)
Kernel: 3.16.3-300.fc21.x86_64 #1 SMP
Docker: 1.2.0-2.fc21
libselinux: 2.3-4.fc21

I have tried selinux in permissive and diabled states - same error.

Are there any workarounds or fixes forecast or should I reninstall using XFS or similar?

[voxadam]

The only workaround that I've found is to run something like boot2docker or CoreOS in a VM.

[rosstimson]

Just adding myself to the list of users that are suffering from this issue:

Fedora 20 (on BTRFS)
Kernel: 3.16.2-201.fc20.x86_64 #1 SMP
Docker: 1.2.0-2.fc20
libselinux: 2.2.1-6.fc20

Currently have SELinux disabled.

[voxadam]

I suppose that I should mention that there are a couple related bugs in the Red Hat Bugzilla system. There's one for RHEL, and one for Fedora.