permission denied if chown after chmod

[snarlysodboxer]

This runs, it gives a 0 Exit status:

FROM ubuntu:trusty

RUN adduser --disabled-password --gecos "" davidamick
RUN mkdir /home/davidamick/testDir
RUN echo "hello world" > /home/davidamick/testDir/testFile

###
RUN chown -R davidamick. /home/davidamick
RUN chmod 700 /home/davidamick/testDir
RUN chmod 600 /home/davidamick/testDir/testFile
###

USER davidamick
ENTRYPOINT ["/bin/bash"]
CMD ["-l", "-c", "touch /home/davidamick/testDir/testFile"]

However if I simply change the chown command to be just after the chmod commands like this:

FROM ubuntu:trusty

RUN adduser --disabled-password --gecos "" davidamick
RUN mkdir /home/davidamick/testDir
RUN echo "hello world" > /home/davidamick/testDir/testFile

###
RUN chmod 700 /home/davidamick/testDir
RUN chmod 600 /home/davidamick/testDir/testFile
RUN chown -R davidamick. /home/davidamick
###

USER davidamick
ENTRYPOINT ["/bin/bash"]
CMD ["-l", "-c", "touch /home/davidamick/testDir/testFile"]

then running it fails with:

touch: cannot touch '/home/davidamick/testDir/testFile': Permission denied

I have confirmed the this with several other things too, for example changing the command like so:

CMD ["-l", "-c", "ls -ahlR /home/davidamick/testDir"]

as well as just running bash interactively and manually attempting to touch/list the files that were chown-ed.
It also does not seem to be related to bash or the environment (that I know of,) since I discovered this issue while doing this:

FROM ubuntu:trusty

RUN apt-get update && apt-get install -y ssh
RUN mkdir /var/run/sshd

RUN adduser --disabled-password --gecos "" davidamick
ADD my_pubkey /home/davidamick/.ssh/authorized_keys

###
RUN chmod 700 /home/davidamick/.ssh
RUN chmod 600 /home/davidamick/.ssh/authorized_keys
RUN chown -R davidamick. /home/davidamick
###

ENTRYPOINT ["/usr/sbin/sshd"]
CMD ["-d"]

which produced the sshd debug error:

debug1: Could not open authorized keys '/home/davidamick/.ssh/authorized_keys': Permission denied

and all it took for it to start working was to move the chown command above the chmod commands.

Is this a bug or am I missing something here? Thanks all.

UPDATE: with my host info:
Ubuntu 12.04

uname -a
Linux my-hostname 3.8.0-39-generic #58~precise1-Ubuntu SMP Fri May 2 21:33:40 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

docker version
Client version: 0.11.0
Client API version: 1.11
Go version (client): go1.2.1
Git commit (client): 15209c3
Server version: 0.11.0
Server API version: 1.11
Git commit (server): 15209c3
Go version (server): go1.2.1
Last stable version: 0.11.1, please update docker

docker info
Containers: 76
Images: 364
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 516
Execution Driver: native-0.2
Kernel Version: 3.8.0-39-generic
WARNING: No swap limit support

[abevoelker]

edit: disregard my issue, Docker compiled from master works fine so I think this issue has already been fixed

I am also getting similar "Permission denied" errors. This is a Dockerfile I have that reproduces my issue:

FROM ubuntu:trusty

RUN adduser web --home /home/web --shell /bin/bash --disabled-password --gecos ""

ADD . /var/www
RUN chown -R web:web /var/www
RUN chmod -R 777 /var/www

USER web

CMD ["/bin/bash", "-l"]

I get the permission denied errors when trying to browse to /var/www as the web user inside the container. I'm really puzzled as to why this is happening from a Unix standpoint... hopefully I'm not overlooking something obvious or some feature of Docker that I should know about.

$ docker build -t foo .                                                       
Uploading context  2.56 kB
Uploading context 
Step 0 : FROM ubuntu:trusty
 ---> a0a7c8a0138b
Step 1 : RUN adduser web --home /home/web --shell /bin/bash --disabled-password 
--gecos ""
 ---> Using cache
 ---> e4a9f4e67a2e
Step 2 : ADD . /var/www
 ---> Using cache
 ---> 07a0df75d2f5
Step 3 : RUN chown -R web:web /var/www
 ---> Using cache
 ---> fe93cd0e7c26
Step 4 : RUN chmod -R 777 /var/www
 ---> Using cache
 ---> 60f91bf16ee1
Step 5 : USER web
 ---> Using cache
 ---> dc41b5b59af8
Step 6 : CMD ["/bin/bash", "-l"]
 ---> Using cache
 ---> c8703ed77168
Successfully built c8703ed77168

$ docker run -i -t foo /bin/bash                                              
web@c9a18c705c94:/$ cd /var/www
bash: cd: /var/www: Permission denied

web@c9a18c705c94:/$ ls -l /var/www
ls: cannot access /var/www/Dockerfile: Permission denied
total 0
-????????? ? ? ? ?            ? Dockerfile

web@c9a18c705c94:/$ ls -l /var
total 40
drwxr-xr-x  2 root root   4096 Apr 10 22:12 backups
drwxr-xr-x 17 root root   4096 May 26 21:31 cache
drwxr-xr-x 47 root root   4096 May 26 21:55 lib
drwxrwsr-x  2 root staff  4096 Apr 10 22:12 local
lrwxrwxrwx  1 root root      9 Apr 16 20:35 lock -> /run/lock
drwxrwxr-x 12 root syslog 4096 May 27 00:48 log
drwxrwsr-x  2 root mail   4096 Apr 16 20:35 mail
drwxr-xr-x  2 root root   4096 Apr 16 20:35 opt
lrwxrwxrwx  1 root root      4 Apr 16 20:35 run -> /run
drwxr-xr-x  5 root root   4096 Apr 16 20:36 spool
drwxrwxrwt  2 root root   4096 Apr 16 20:36 tmp
drwxrwxrwx  2 web  web    4096 May 29 04:32 www

web@c9a18c705c94:/$ ls -ln /var
total 40
drwxr-xr-x  2    0    0 4096 Apr 10 22:12 backups
drwxr-xr-x 17    0    0 4096 May 26 21:31 cache
drwxr-xr-x 47    0    0 4096 May 26 21:55 lib
drwxrwsr-x  2    0   50 4096 Apr 10 22:12 local
lrwxrwxrwx  1    0    0    9 Apr 16 20:35 lock -> /run/lock
drwxrwxr-x 12    0  104 4096 May 27 00:48 log
drwxrwsr-x  2    0    8 4096 Apr 16 20:35 mail
drwxr-xr-x  2    0    0 4096 Apr 16 20:35 opt
lrwxrwxrwx  1    0    0    4 Apr 16 20:35 run -> /run
drwxr-xr-x  5    0    0 4096 Apr 16 20:36 spool
drwxrwxrwt  2    0    0 4096 Apr 16 20:36 tmp
drwxrwxrwx  2 1000 1000 4096 May 29 04:32 www

web@c9a18c705c94:/$ /usr/bin/id
uid=1000(web) gid=1000(web) groups=1000(web)

I'm running Ubuntu 14.04 64-bit on the host machine. Additional host machine info:

$ uname -a
Linux abe-PC 3.13.0-27-generic #50-Ubuntu SMP Thu May 15 18:06:16 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

$ docker version
Client version: 0.11.1
Client API version: 1.11
Go version (client): go1.2.1
Git commit (client): fb99f99
Server version: 0.11.1
Server API version: 1.11
Git commit (server): fb99f99
Go version (server): go1.2.1
Last stable version: 0.11.1

$ docker info
Containers: 38
Images: 618
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 695
Execution Driver: native-0.2
Kernel Version: 3.13.0-27-generic
Username: abevoelker
Registry: [https://index.docker.io/v1/]
WARNING: No swap limit support

[abevoelker]

Nevermind! I just compiled Docker from master and it works fine. I'll be eagerly awaiting Docker 0.11.2 😄

$ docker version
Client version: 0.11.1-dev
Client API version: 1.12
Go version (client): go1.2.1
Git commit (client): 8622641
Server version: 0.11.1-dev
Server API version: 1.12
Go version (server): go1.2.1
Git commit (server): 8622641

[thomasleveil]

This issue still exists in Docker 1.0.1

[vagrant@docker:~/tmp] 1 $ docker version
Client version: 1.0.1
Client API version: 1.12
Go version (client): go1.2.1
Git commit (client): 990021a
Server version: 1.0.1
Server API version: 1.12
Go version (server): go1.2.1
Git commit (server): 990021a

This fails:

[vagrant@docker:~/tmp] $ cat Dockerfile
FROM ubuntu:latest

RUN useradd larry && echo 'larry:docker' | chpasswd
RUN mkdir -p /home/larry/.ssh/

RUN chmod 700 /home/larry/.ssh
RUN chown -R larry:larry /home/larry

USER larry
CMD ["/bin/bash", "-il"]


[vagrant@docker:~/tmp] $ docker build --rm -t gh-6047 .
[vagrant@docker:~/tmp] $ docker run --rm -it gh-6047
larry@0d91f9949f2a:/$ cd /home/larry/.ssh/
bash: cd: /home/larry/.ssh/: Permission denied

While swapping chown and chmod fix it:

[vagrant@docker:~/tmp] $ cat Dockerfile
FROM ubuntu:latest

RUN useradd larry && echo 'larry:docker' | chpasswd
RUN mkdir -p /home/larry/.ssh/

RUN chown -R larry:larry /home/larry
RUN chmod 700 /home/larry/.ssh

USER larry
CMD ["/bin/bash", "-il"]

[vagrant@docker:~/tmp] $ docker build --rm -t gh-6047-2 .
[vagrant@docker:~/tmp] $ docker run --rm -it gh-6047-2
larry@163be13b0097:/$ cd /home/larry/.ssh/

[techthumb]

I'm seeing a similar issue with my Dockerfile at https://registry.hub.docker.com/u/techthumb/dockerfiles/dockerfile

However, when I build the same Dockerfile using boot2docker on OS X, the permissions are preserved

docker version
Client version: 1.1.1
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): bd609d2
Server version: 1.1.1
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): bd609d2

[unclejack]

Using Docker 1.1.1 and Docker built from master, I couldn't reproduce the bug with two Dockerfiles:

FROM ubuntu:trusty

RUN adduser --disabled-password --gecos "" davidamick
RUN mkdir /home/davidamick/testDir
RUN echo "hello world" > /home/davidamick/testDir/testFile

###
RUN chmod 700 /home/davidamick/testDir
RUN chmod 600 /home/davidamick/testDir/testFile
RUN chown -R davidamick. /home/davidamick
###

USER davidamick
ENTRYPOINT ["/bin/bash"]
CMD ["-l", "-c", "touch /home/davidamick/testDir/testFile"]

FROM ubuntu:latest

RUN useradd larry && echo 'larry:docker' | chpasswd
RUN mkdir -p /home/larry/.ssh/

RUN chmod 700 /home/larry/.ssh
RUN chown -R larry:larry /home/larry

USER larry
CMD ["/bin/bash", "-il"]

I couldn't reproduce this on AUFS, devicemapper and btrfs. This looks like an environment problem (kernel, fs or other configuration problem).

Can anyone who's running into this provide the full output of docker info, docker version and uname -a using the latest Docker, please?
If you're running aufs, please tell us what file system you're using for your /var/lib/docker root folder.

[larrycai]

For last case (larry), I still met the problem

larry@8015900dda8b:/home/larry$ ls -al
total 12
drwxr-xr-x 5 larry larry 4096 Jul 22 12:10 .
drwxr-xr-x 5 root  root  4096 Jul 22 12:10 ..
drwx------ 2 larry larry 4096 Jul 22 12:10 .ssh
larry@8015900dda8b:/home/larry$ cd .ssh
bash: cd: .ssh: Permission denied
larry@8015900dda8b:/home/larry$ id
uid=1000(larry) gid=1000(larry) groups=1000(larry)

And related conf

docker@boot2docker:~$ docker info
Containers: 6
Images: 148
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Dirs: 160
Execution Driver: native-0.2
Kernel Version: 3.15.3-tinycore64
Debug mode (server): true
Debug mode (client): false
Fds: 22
Goroutines: 18
EventsListeners: 0
Init Path: /usr/local/bin/docker
Sockets: [unix:///var/run/docker.sock tcp://0.0.0.0:2375]

docker@boot2docker:~$ docker version
Client version: 1.1.1
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): bd609d2
Server version: 1.1.1
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): bd609d2

docker@boot2docker:~$ uname -a
Linux boot2docker 3.15.3-tinycore64 #1 SMP Thu Jul 10 01:43:00 UTC 2014 x86_64 GNU/Linux

Later me know if need more information

[larrycai]

same on Ubuntu 14.04 VM in Windows

user@ubuntu:~$ docker info
Containers: 7
Images: 99
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 159
Execution Driver: native-0.2
Kernel Version: 3.13.0-24-generic
Username: larrycai
Registry: [https://index.docker.io/v1/]
WARNING: No swap limit support
user@ubuntu:~$ uname -a
Linux ubuntu 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

fs type for /var/lib is ext4

[jdef]

The machine giving me errors:

root@kubernetes-minion-2:~# docker info
Containers: 28
Images: 145
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 201
Execution Driver: native-0.2
Kernel Version: 3.14.0-0.bpo.1-amd64
Debug mode (server): true
Debug mode (client): false
Fds: 33
Goroutines: 32
EventsListeners: 0
Init Path: /usr/bin/docker
Sockets: [unix:///var/run/docker.sock]
Username: jdef
Registry: [https://index.docker.io/v1/]
WARNING: No swap limit support
root@kubernetes-minion-2:~# docker -v
Docker version 1.1.2, build d84a070
root@kubernetes-minion-2:~# uname -a
Linux kubernetes-minion-2 3.14-0.bpo.1-amd64 #1 SMP Debian 3.14.12-1~bpo70+1 (2014-07-13) x86_64 GNU/Linux
root@kubernetes-minion-2:~# cat /etc/default/docker
DOCKER_OPTS="--bridge cbr0 --iptables=false --debug=true"

The machine I built the image on:

[root@mesos-dev hornetq]# docker -v
Docker version 1.1.2, build d84a070/1.1.2
[root@mesos-dev hornetq]# docker info
Containers: 18
Images: 414
Storage Driver: devicemapper
 Pool Name: docker-253:0-34191812-pool
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 13823.1 Mb
 Data Space Total: 102400.0 Mb
 Metadata Space Used: 20.6 Mb
 Metadata Space Total: 2048.0 Mb
Execution Driver: native-0.2
Kernel Version: 3.10.0-123.4.2.el7.x86_64
Username: jdef
Registry: [https://index.docker.io/v1/]
[root@mesos-dev hornetq]# uname -a
Linux mesos-dev 3.10.0-123.4.2.el7.x86_64 #1 SMP Mon Jun 30 16:09:14 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

The errors (and mixed file ownership?!) I see:

[hornetq@bf1079c2c372 hornetq-2.4.0.Final]$ ls config
ls: cannot open directory config: Permission denied
[hornetq@bf1079c2c372 hornetq-2.4.0.Final]$ ls bin
ls: cannot open directory bin: Permission denied
[hornetq@bf1079c2c372 hornetq-2.4.0.Final]$ ls data
[hornetq@bf1079c2c372 hornetq-2.4.0.Final]$ ls docs
api  quickstart-guide  user-manual
[hornetq@bf1079c2c372 hornetq-2.4.0.Final]$ ls -laF
total 44
drwxr-xr-x 43 root    root    4096 Aug 20 06:24 ./
drwxr-xr-x 17 root    root    4096 Aug 20 06:24 ../
drwxrwx---  6 hornetq hornetq 4096 Aug 20 06:19 bin/
drwxr-xr-x 18 root    root    4096 Aug 20 06:24 config/
drwxrwxr-x  2 hornetq hornetq 4096 Aug 20 06:25 data/
drwxr-xr-x 10 root    root    4096 Aug 20 06:24 docs/
drwxr-xr-x 19 root    root    4096 Aug 20 06:24 examples/
drwxrwxr-x  2 hornetq hornetq 4096 Aug 14 09:10 lib/
drwxrwx---  2 hornetq hornetq 4096 Dec 14  2013 licenses/
drwxrwxr-x  2 hornetq hornetq 4096 Aug 20 06:25 logs/
drwxrwxr-x  2 hornetq hornetq 4096 Aug 20 06:19 schema/

Dockerfile commands:

RUN groupadd --gid 253 hornetq
RUN useradd --gid 253 --uid 253 --base-dir /opt --create-home hornetq
ADD hornetq-2.4.0.Final-bin.tar.gz      /opt/hornetq/
    ADD jmxagent.jar                        /opt/hornetq/hornetq-2.4.0.Final/bin/
    ADD hack_hq_config                      /opt/hornetq/
    ADD hack_jms_config                     /opt/hornetq/
    ADD augeas                              /opt/hornetq/augeas/

WORKDIR /opt/hornetq
RUN ln -s hornetq-2.4.0.Final hornetq
RUN mkdir -p hornetq/{logs,data}
RUN touch hornetq/{logs,data}/.init_volume

## -- this is how downstream images can easily add their custom configuration
ONBUILD ADD ./config                    /opt/hornetq/hornetq-2.4.0.Final/config/
ONBUILD USER root
ONBUILD RUN chown -R hornetq:hornetq    /opt/hornetq/hornetq-2.4.0.Final/config/
ONBUILD USER hornetq

# -- do this last since we're hacking it a bit
ADD bootstrap.sh                        /opt/hornetq/hornetq-2.4.0.Final/bin/run.sh
RUN chown -R hornetq:hornetq            /opt/hornetq/
RUN find /opt/hornetq -type d -exec chmod ug+rwx \{\} \;
RUN find /opt/hornetq -type f -exec chmod ug+rw \{\} \;
RUN find /opt/hornetq -type f -perm /111 -exec chmod a+x \{\} \;

## -- hornetq
VOLUME [ "/opt/hornetq/hornetq-2.4.0.Final/logs", "/opt/hornetq/hornetq-2.4.0.Final/data" ]

USER hornetq
ENV HOME /opt/hornetq
ENV USER hornetq

WORKDIR ./hornetq-2.4.0.Final

I commented out the ONBUILD and RUN find... commands, and I still get errors, but the file ownership somehow changes?!

[hornetq@4ea24d53d0e1 hornetq-2.4.0.Final]$ ll
total 36
drwx------  4 hornetq hornetq 4096 Aug 20 06:37 bin
drwx------ 12 hornetq hornetq 4096 Dec 14  2013 config
drwxr-xr-x  2 hornetq hornetq 4096 Aug 20 06:42 data
drwxr-xr-x  8 hornetq hornetq 4096 Aug 14 09:10 docs
drwx------ 12 hornetq hornetq 4096 Dec 14  2013 examples
drwxr-xr-x  2 hornetq hornetq 4096 Aug 14 09:10 lib
drwx------  2 hornetq hornetq 4096 Dec 14  2013 licenses
drwxr-xr-x  2 hornetq hornetq 4096 Aug 20 06:42 logs
drwxr-xr-x  2 hornetq hornetq 4096 Aug 14 09:10 schema
[hornetq@4ea24d53d0e1 hornetq-2.4.0.Final]$ ll bin
ls: cannot open directory bin: Permission denied
[hornetq@4ea24d53d0e1 hornetq-2.4.0.Final]$ ll config
ls: cannot open directory config: Permission denied
[hornetq@4ea24d53d0e1 hornetq-2.4.0.Final]$ ll docs
total 12
drwxr-xr-x 10 hornetq hornetq 4096 Aug 14 09:10 api
drwxrwxr-x  8 hornetq hornetq 4096 Dec 16  2013 quickstart-guide
drwxrwxr-x  8 hornetq hornetq 4096 Dec 16  2013 user-manual

EDIT: I added some Dockerfile steps above that I had accidentally cut out the first time around.

[jdef]

I should add that I don't have any permission problems when running a container from this image on the build machine (using devicemapper-thin), only on the cloud VM (which uses aufs).

I dumped the directory listings for each relevant branch of the aufs graph -- it's pretty verbose so I'm putting it in this gist : https://gist.github.com/jdef/bdab1acec40338e6ddc1

[abevoelker]

@jdef What is the docker run statement that is starting container hornetq@bf1079c2c372? If you're mounting those directories from the host machine, they will set ownership to the UID of the host machine owner, which may cause issues and inconsistencies between hosts.

[jdef]

@abevoelker , the only thing I'm mounting is a syslog UNIX domain socket, and that doesn't change the outcome.

docker run --rm -ti -v /dev/log:/dev/log --entrypoint bash myrepo:hornetq-latest -il

[jdef]

Looks like aufs3 supports a dirperm1 option, which may help to alleviate problems with permissions. Thoughts on incorporating this into the aufs storage driver?
http://aufs.sourceforge.net/aufs3/man.html

[duglin]

Was looking at this one, and to add to the weirdness, the following Dockerfile shows that there's a difference when the chmod and chown are on the same line vs on different lines - this could be an indication of it being a aufs issue:

FROM ubuntu
RUN adduser --disabled-password --gecos "" joe
RUN mkdir /good /bad
RUN chmod 700 /good ; chown -R joe. /good
RUN chmod 700 /bad
RUN chown -R joe. /bad
USER joe
RUN echo > /good/foo ; echo > /bad/foo

In the above Dockerfile, the "good" path works just fine, but the "bad" gets an error:
/bin/sh: 1: cannot create /bad/foo: Permission denied

[fazy]

I have Docker on three different host machines, two exhibit this problem and one doesn't.

Config details of all three: http://pastebin.com/zUqnJrDj

The issue I'm facing is not just that I can't change the permissions; it seems like I "can" change the permissions, but the change has no effect (silent fail).

I've been trying to build an image based on my work in progress here: https://github.com/fazy/docker-apache-symfony/tree/permissions-issue/web

When running, Apache is giving "403 Forbidden" because it can't access the directory to check for a .htaccess file. If I create the right permissions before running docker build it's fine, but if I start with restrictive permissions and then try to open them up, the change has no effect.

Files used to build the container:

Contents of the working directory within my context when building:

$ ls -lR
.:
total 8
drwx------ 2 lars lars 4096 Nov 21 10:24 bar
drwxr-xr-x 2 lars lars 4096 Nov 21 10:24 foo

./bar:
total 4
-rw------- 1 lars lars 207 Nov 21 10:24 index.html

./foo:
total 4
-rw-r--r-- 1 lars lars 207 Nov 21 08:53 index.html

After building I run like this:

docker run -d -p 8000:80 --name hello fazy/symfony

Requests for http://localhost:8000/foo/ correctly serve the index file, requests for http://localhost:8000/bar/ fail with a 403 error.

Files inside the container:

Here's what I see when I nsenter into the container and cd to /var/www/app/web:

# ls -lR
.:
total 8
drwx------ 2 root root 4096 Nov 21 10:24 bar
drwxr-xr-x 2 root root 4096 Nov 21 10:24 foo

./bar:
total 4
-rw------- 1 root root 207 Nov 21 10:24 index.html

./foo:
total 4
-rw-r--r-- 1 root root 207 Nov 21 08:53 index.html

Then I try to change the permissions and view again:

# chmod 0755 bar && chmod 0644 bar/index.html
# ls -lR
.:
total 8
drwxr-xr-x 2 root root 4096 Nov 21 10:34 bar
drwxr-xr-x 2 root root 4096 Nov 21 10:24 foo

./bar:
total 4
-rw-r--r-- 1 root root 207 Nov 21 10:24 index.html

./foo:
total 4
-rw-r--r-- 1 root root 207 Nov 21 08:53 index.html

It looks like everything is correct now; but I still get the 403 forbidden error.

Obviously I wouldn't normally go in with nsenter, but do it in Dockerfile (build time) or /start.sh (run time).

This in Dockerfile didn't work (same behaviour as above):

ADD . /var/www/app
RUN chmod -R go+rX /var/www/app

The Dockerfile ends with:

CMD ["/start.sh"]

Adding that chmod command to /start.sh also doesn't resolve the problem.

The only way I've managed it is using a temp directory:

ADD . /tmp/app
RUN    cp -r /tmp/app /var/www \
    && chmod -R go+rX /var/www/app \
    && rm -rf /tmp/app

Aside from performance costs with a real world app, I'm a bit worried about the fact that this behaviour is easy to overlook (maybe sometimes when I ADD the files the permissions are already correct, but then another time they're not).

Because this limitation is so counter-intuitive, can there be a big, bold warning at the top of this section please? http://docs.docker.com/reference/builder/#add

Also, I note the discussion in #783. If I understand correctly it seems Docker is moving towards DeviceMapper, but AUFS is still the default when present. I'm not sure why two of my Debian-based systems have AUFS. One of them is running inside VMWare on Windows, and the other is installed directly on an Intel-based PC.

Maybe more should be communicated about the choice of filesystems? I can't see any mention of DeviceMapper in the Ubuntu or Debian install guides for example.

For now, I will try and move over to DeviceMapper (this article looks like a good starting point).

[GertBurger]

Same problem. Swapping the chmod/chown statements seems to work fine.

Ubuntu 14.04.1 LTS
Linux test 3.13.0-40-generic #69-Ubuntu SMP Thu Nov 13 17:53:56 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Client version: 1.3.2
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): 39fa2fa
OS/Arch (client): linux/amd64
Server version: 1.3.2
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): 39fa2fa

[SvenDowideit]

i presume this is the aufs issue?

[larrycai]

@ricardobranco777 , problem exist still in Docker 1.4.1, Verified in boot2docker 1.4.1. aufs problem.

[SvenDowideit]

@ricardobranco777 using which graphdrivers (docker info) ?

[mitchcapper]

I can also confirm this is still happening
Containers: 27
Images: 140
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Dirs: 194
Execution Driver: native-0.2
Kernel Version: 3.16.0-28-generic
Operating System: Ubuntu 14.04.1 LTS
CPUs: 1

[mitchcapper]

Certainly only with aufs. The main problem with this is with file build complexity. If a dockerfile up the chain does a chmod (or say an package install command does) you cannot fix the permissions on the file/folder by any means (minus a move and move back as root which works). In addition even without a chroot involved a chmod after chmod does not work (so chmod a folder 700 then back to 755 and no one but root can ls it). Below are two examples, a very simple one of the original problem and a second one showing all the oddities (for example if the folder was chmodded 711 then a chown is done the user can write to the folder they own but not list the contents).
Here is a simpler example of the problem (on aufs you will get permission denied on ls'ing the contents of test):

FROM busybox
RUN adduser -D bob
RUN mkdir /tmp/test
RUN chmod 700 /tmp/test
RUN chown bob.bob /tmp/test
USER bob
CMD ls -l /tmp /tmp/test

docker build -t test . && docker run -t -i test

/tmp:
total 8
drwxr-xr-x    2 root     root          4096 May 22  2014 ldconfig
drwx------    2 bob      bob           4096 Jan  3 19:37 test

/tmp/test:
ls: can't open '/tmp/test': Permission denied

Now to show the real craziness:

FROM busybox
RUN adduser -D bob
RUN mkdir /tmp/chmod_fail /tmp/write_allowed_no_ls /tmp/write_fail
RUN chmod 700 /tmp/chmod_fail /tmp/write_fail
RUN chmod 755 /tmp/chmod_fail
RUN chmod 711 /tmp/write_allowed_no_ls
RUN chown bob.bob /tmp/write_allowed_no_ls /tmp/write_fail
USER bob
CMD touch /tmp/write_allowed_no_ls/file;touch /tmp/write_fail/file;ls -l /tmp /tmp/chmod_fail /tmp/write_allowed_no_ls /tmp/write_allowed_no_ls/file

docker build -t test . && docker run -t -i test

touch: /tmp/write_fail/file: Permission denied
-rw-r--r--    1 bob      bob              0 Jan  3 20:27 /tmp/write_allowed_no_ls/file

/tmp:
total 16
drwxr-xr-x    2 root     root          4096 Jan  3 20:26 chmod_fail
drwxr-xr-x    2 root     root          4096 May 22  2014 ldconfig
drwx--x--x    2 bob      bob           4096 Jan  3 20:27 write_allowed_no_ls
drwx------    2 bob      bob           4096 Jan  3 20:26 write_fail

/tmp/chmod_fail:
ls: can't open '/tmp/chmod_fail': Permission denied
total 0

/tmp/write_allowed_no_ls:
ls: can't open '/tmp/write_allowed_no_ls': Permission denied

As mentioned a simple move and move back however works, take the first example and add one line to the docker file:

FROM busybox
RUN adduser -D bob
RUN mkdir /tmp/test
RUN chmod 700 /tmp/test
RUN chown bob.bob /tmp/test
RUN mv /tmp/test /tmp/test2 && mv /tmp/test2 /tmp/test
USER bob
CMD ls -l /tmp /tmp/test

and now it works just fine.

[mitchcapper]

and while I posted the info here is the version:
Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 5bc2ff8
OS/Arch (client): linux/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8

[jessfraz]

@mitchcapper can you please try with the latest version of docker

[jessfraz]

could this be related to #783

[jessfraz]

it seems to have the same comments and problems as #783 https://docs.docker.com/release-notes/#known-issues

[mitchcapper]

@jfrazelle yes it still occurs. I assume once #10390 gets merged in and the dirperm1 opt is available it will go away. Is there a reason this is thought to be fixed otherwise?

[tomaspinho]

This is still an issue on v1.12.1 with ubuntu 16.04 as both host and container.

chmod THEN chown:

jenkins@2caebbe69010:~$ ls -la
total 32
drwxr-xr-x 12 jenkins jenkins 4096 Jan  3 16:08 .
drwxr-xr-x 11 root    root    4096 Jan  3 16:07 ..
-rw-r--r--  1 jenkins jenkins  220 Aug 31  2015 .bash_logout
-rw-r--r--  1 jenkins jenkins 3771 Aug 31  2015 .bashrc
drwx------  2 jenkins jenkins 4096 Jan  3 16:07 .cache
-rw-r--r--  1 jenkins jenkins  655 Jun 24  2016 .profile
drwx------  2 jenkins jenkins 4096 Jan  3 16:08 .ssh
jenkins@2caebbe69010:~$ touch .cache/test
jenkins@2caebbe69010:~$ touch .ssh/test  
touch: cannot touch '.ssh/test': Permission denied

chown THEN chmod:

jenkins@f45e9a4b3a7a:~$ ls -la
total 28
drwxr-xr-x 10 jenkins jenkins 4096 Jan  3 16:27 .
drwxr-xr-x 11 root    root    4096 Jan  3 16:27 ..
-rw-r--r--  1 jenkins jenkins  220 Aug 31  2015 .bash_logout
-rw-r--r--  1 jenkins jenkins 3771 Aug 31  2015 .bashrc
drwx------  2 jenkins jenkins 4096 Jan  3 16:27 .cache
-rw-r--r--  1 jenkins jenkins  655 Jun 24  2016 .profile
drwx------  2 jenkins jenkins 4096 Jan  3 16:24 .ssh
jenkins@f45e9a4b3a7a:~$ touch .cache/test
jenkins@f45e9a4b3a7a:~$ touch .ssh/test

[thaJeztah]

@tomaspinho possibly an issue in the aufs kernel module; see #20240 (comment), and #24660

[tomaspinho]

@thaJeztah thanks, but it's easier to just change the Dockerfile, as monkeypatching kernel modules is not an option in my environment. Just left this as docs for other people that may be facing the same