Docker 0.8.0: Seeing container directory permission differences from same image on Ubuntu vs. Arch

[VictorLowther]

Arch system is running Docker 0.8.0 using the devicemapper storage backend:

victor@m4700:~/src/opencrowbar/core (master)
$ docker version
Client version: 0.8.0
Go version (client): go1.2
Git commit (client): cc3a8c8
Server version: 0.8.0
Git commit (server): cc3a8c8
Go version (server): go1.2
Last stable version: 0.8.0
victor@m4700:~/src/opencrowbar/core (master)
$ docker info
Containers: 6
Images: 25
Driver: devicemapper
 Pool Name: docker-254:2-195-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 6773.1 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 9.0 Mb
 Metadata Space Total: 2048.0 Mb
Username: opencrowbar
Registry: [https://index.docker.io/v1/]
WARNING: No swap limit support

Ubuntu system is running Docker 0.8.0 using the AUFS storage backend:

crowbar@build30:~/core$ docker info
Containers: 23
Images: 4
Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 50
WARNING: No swap limit support
crowbar@build30:~/core$ docker version
Client version: 0.8.0
Go version (client): go1.2
Git commit (client): cc3a8c8
Server version: 0.8.0
Git commit (server): cc3a8c8
Go version (server): go1.2
Last stable version: 0.8.0

On both systems, I am running the opencrowbar/centos:6.5-2 image
(created on the Arch system from the centos:6.4 image) using the
following command:

docker run -i -t opencrowbar/centos:6.5-2 /bin/bash -i

On the Arch system, things operate normally and the directory
permissions are as expected:

victor@m4700:~/src/opencrowbar/core (master)
$ docker run -i -t opencrowbar/centos:6.5-2 /bin/bash -i
bash-4.1# ls -al /
total 6776
dr-xr-xr-x  24 root root    4096 Feb 11 13:34 .
dr-xr-xr-x  24 root root    4096 Feb 11 13:34 ..
-rw-------   1 root root     357 Feb  6 16:33 .bash_history
-rw-r--r--   1 root root     115 Feb 11 13:34 .dockerenv
-rwx------   1 root root 6830663 Feb  6 16:54 .dockerinit
drwxr-xr-x   3 root root    4096 Jan 22 06:36 .gem
drwxr-----   3 root root    4096 Jan 22 06:30 .pki
dr-xr-xr-x   2 root root    4096 Jan 22 06:30 bin
dr-xr-xr-x   2 root root    4096 Sep 23  2011 boot
drwxr-xr-x   6 root root    4096 Feb 11 13:34 dev
drwxr-xr-x  46 root root    4096 Feb 11 13:34 etc
drwxr-xr-x   3 root root    4096 Jan 22 06:28 home
dr-xr-xr-x   7 root root    4096 Jan 22 06:32 lib
dr-xr-xr-x   7 root root   12288 Feb  5 13:37 lib64
drwxr-xr-x   2 root root    4096 Sep 23  2011 media
drwxr-xr-x   2 root root    4096 Sep 23  2011 mnt
drwxr-xr-x   4 root root    4096 Jan 22 06:35 opt
dr-xr-xr-x 366 root root       0 Feb 11 13:34 proc
dr-xr-x---   2 root root    4096 Feb  6 16:30 root
dr-xr-xr-x   2 root root    4096 Jan 22 06:30 sbin
drwxr-xr-x   2 root root    4096 Sep 23  2011 selinux
drwxr-xr-x   2 root root    4096 Sep 23  2011 srv
drwxr-xr-x  13 root root       0 Jun 28  2011 sys
drwxr-xr-x   4 root root    4096 Feb  6 16:30 tftpboot
drwxrwxrwt   2 root root    4096 Feb  6 16:30 tmp
drwxr-xr-x  14 root root    4096 Jan 22 06:36 usr
drwxr-xr-x  18 root root    4096 Jan 22 06:35 var
bash-4.1#

On the Ubuntu system, however, the directory permissions of several
key directories are screwed up leading to breakage when we try to run
our canned dev/test environment:

crowbar@build30:~/core$ docker run -i -t opencrowbar/centos:6.5-2 /bin/bash -i
bash-4.1# ls -al /
total 14864
drwxr-xr-x  65 root root     4096 Feb 11 13:25 .
drwxr-xr-x  65 root root     4096 Feb 11 13:25 ..
-rw-------   1 root root      357 Feb  6 16:33 .bash_history
dr-xr-xr-x   2 root root     4096 Jan 22 06:30 bin
dr-xr-xr-x   2 root root     4096 Sep 23  2011 boot
drwxr-xr-x   6 root root     4096 Feb  6 16:30 dev
-rw-r--r--   1 root root      656 Feb 11 13:25 .dockerenv
-rwx------   1 root root 15119491 Feb 11 12:31 .dockerinit
drwxr-xr-x  87 root root     4096 Feb 11 13:26 etc
d--x--x---   4 root root     4096 Feb 11 00:19 .gem
d--x--x---   6 root root     4096 Feb 11 00:20 home
dr-xr-xr-x  10 root root     4096 Jan 22 06:32 lib
dr-xr-xr-x   9 root root     4096 Feb  5 13:37 lib64
drwxr-xr-x   2 root root     4096 Sep 23  2011 media
drwxr-xr-x   2 root root     4096 Sep 23  2011 mnt
drwxr-xr-x   4 root root     4096 Jan 22 06:35 opt
drwxr-----   3 root root     4096 Jan 22 06:30 .pki
dr-xr-xr-x 101 root root        0 Feb 11 13:25 proc
dr-xr-x---   3 root root     4096 Feb 11 13:26 root
dr-xr-xr-x   2 root root     4096 Jan 22 06:30 sbin
drwxr-xr-x   2 root root     4096 Sep 23  2011 selinux
drwxr-xr-x   2 root root     4096 Sep 23  2011 srv
dr-xr-xr-x  13 root root        0 Feb 11 13:25 sys
drwxr-xr-x   4 root root     4096 Feb  6 16:30 tftpboot
drwxrwxrwt   2 root root     4096 Feb  6 16:30 tmp
d--x--x---  29 root root     4096 Feb 11 00:20 usr
d--x--x---  43 root root     4096 Feb 11 00:21 var
bash-4.1#

Note the directory permissions on /usr, /var, and /home -- they are
755 on the container running on the Arch host, and 110 on the
container running on the Ubuntu host. As part of our workflow
involves running commands as a non-root user these broken permissions
cause all sorts of entertaining failures on the container running on
the Ubuntu host.

[VictorLowther]

Having Docker use the devicemapper backend on the Ubuntu system yields correct directory permissions.

[mhennings]

From what i noticed so far: When using aufs there are some circumstances where permissions can not be changed after initial creation in following layers.

Normally this is not a problem, but can be, when multiple layers are stacked where permissions differ between layers.

A workaround is to export and import an image after significant changes / initial creation.

[koterpillar]

A minimal test case:

FROM ubuntu
RUN apt-get -qq update

Push this somewhere and then pull on an aufs host, then:

docker run ls -l /

This will show:

d--x--x--- 12 root root 4096 Feb 21 00:28 var

Note that it doesn't matter if the original image was built on aufs or devicemapper.

[stoopsj]

I've encountered this as well in cases where an image was built on docker+devicemapper and then run on docker+aufs.

Export/import on a devicemapper machine fixes this, as mentioned above. So does building the same image on docker+aufs to start with.

[tomfotherby]

I'm stuck on this issue as well. As far as I understand it as long as kernel extras are installed on ALL my machines I am ok. But if I forgot to do

sudo apt-get install linux-image-extra-`uname -r`

on some of them and build and push a image, the other machines have problems when running the container. For me it's the /var folder that ends up with the wrong permissions.

[koterpillar]

I still get the wrong /var permissions if I build on AUFS, push and pull on another AUFS machine.

I also ran git bisect with the following script:

#!/bin/bash -e

sudo rm -rf bundles
make binary || exit 125
sudo service docker stop
sleep 2
sudo cp bundles/*/binary/docker-* $(which docker)
sudo rm -rf /var/lib/docker
sudo service docker start
sleep 2
docker run -rm our-private-contyard.test/aufs110 ls -l
docker run our-private-contyard.test/aufs110 \
    sh -c 'ls -l / | grep var' | \
    grep -qv 'd--x'

our-private-contyard.test/aufs110 is an image built with the following commands:

FROM ubuntu
RUN apt-get -qq update

Result points to a4868e2:

a4868e233c7ebce259fc02d3dbfb241c23471a4a is the first bad commit
commit a4868e233c7ebce259fc02d3dbfb241c23471a4a
Author: Alexander Larsson <alexl@redhat.com>
Date:   Fri Dec 20 12:08:34 2013 +0100

    Implement UnTar via archive/tar

    This replaces the shelling out to tar with a reimplementation of untar
    based on the archive/tar code and the pre-existing code from ApplyLayer
    to create real files from tar headers.

    Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)

:040000 040000 eb538a9cad7f1f6484083627618d84d0a4d40f04 641ac237f753b694c2c6be8895a467dcf25a5373 M  archive
bisect run success

[koterpillar]

Confirmed the commit is the culprit, revert fixes the behaviour: https://github.com/infoxchange/docker/tree/undo-tar-changes-4068

[koterpillar]

Investigation:

The permissions are broken whenever a layer contains a change inside the existing directory, e.g. /directory/file is changed, but /directory is pre-existing.
The resulting tar file only contains /directory/file, not /directory.
When recreating the layer, AUFS creates /directory manually since it does not exist in the diff directory.
The permissions for this new directory are then set to be 600 (0600 was probably intended, but this is not the whole problem).

To fix the problem properly, the permissions for new /directory must be copied from the previous layer.

[ross-w]

Any chance of some action here? I'm surprised more people haven't encountered this bug as it is readily reproducible on default settings! We're seeing the exact same issue but @koterpillar 's fix above does work.

Ideally the bug should be sorted properly (permissions from the previous layer rather than fudging the perms) but having docker it in its current state with permissions being set to unreadable is pretty unacceptable, and will likely cause people trying docker to encounter significant problems and perhaps lose faith in the whole docker concept.

[fzerorubigd]

I have this problem in version 0.91:

Client version: 0.9.1
Go version (client): go1.2.1
Git commit (client): 3600720
Server version: 0.9.1
Git commit (server): 3600720
Go version (server): go1.2.1
Last stable version: 0.9.1

So i think the problem still exists.

[koterpillar]

It's fixed in master, but haven't made its way into a release yet.

[mastef]

This issue seems to still persist in latest Docker version, had to dist-upgrade Precise 64 and add the linux-headers. When I built the Docker images on this system, su user would give /bin/bash: Permission denied

[ghost]

I think I got caught in the same issue, I've tried with docker 1.6 and 1.8 and in both, the containers show odd permissions on /. So changing to 755 could be a workaround:

[root@afd0c24bb2f2 /]# ls -lsa /
total 80
4 drwxr-x--- 21 root root 4096 Sep 9 10:15 .
4 drwxr-x--- 21 root root 4096 Sep 9 10:15 ..

Test case, inside a container with this issue:
[root@afd0c24bb2f2 /]# adduser test
[root@afd0c24bb2f2 /]# su test
su: /bin/bash: Permission denied
[root@afd0c24bb2f2 /]# chmod 755 /
[root@afd0c24bb2f2 /]# su test
[test@afd0c24bb2f2 /]$ id
uid=500(test) gid=500(test) groups=500(test)