"docker exec -itu root <c> bash" and WORKDIR set to some user home directory results in permission error

[Wz5m165c]

Description

With Docker version 18.06.0-ce, build 0ffa825, seems that docker exec with user set to root and working directory being inside a directory where only some other user than root has permissions (drwx------) results in an error message about permissions.

Steps to reproduce the issue:

1. Create Dockerfile:

FROM centos:7
ENV container docker
RUN useradd --system --create-home --home-dir /home/test --no-log-init test
USER test
WORKDIR /home/test

2. Build image with docker build -t test-img .
3. Start container: docker run -dt --name test test-img
4. Then try docker exec -itu root test bash to get the error message.

Describe the results you received:

OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "chdir to cwd (\"/home/test\") set in config.json failed: permission denied": unknown

Describe the results you expected:

This behavior was not present in Docker version 17.12.0-ce, build c97c6d6; docker exec -itu root test bash worked fine.

I would expect root to be able to chdir into any directory, regardless of the directory's permissions.

Additional information you deem important (e.g. issue happens only occasionally):

Commands docker exec -itu root -w / test bash and docker exec -it test bash work. Also docker exec -itu root -w / test bash -c "cd /home/test && pwd" works

Note that running the docker with docker run -itu root --name test test-img bash does not produce the error.

Output of docker version:

Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 19:11:02 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.0-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       0ffa825
  Built:            Wed Jul 18 19:09:05 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

Containers: 11
 Running: 5
 Paused: 0
 Stopped: 6
Images: 169
Server Version: 18.06.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d64c661f1d51c48782c9cec8fda7604785f93587
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-131-generic
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 6
Total Memory: 21.22GiB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

[cwgem]

I tried this basing off Ubuntu instead of CentOS 7 and it worked fine. Looking at their official images their Dockerfile is pretty much copying over a .xz tarball and FROM scratch. This tarball gets generated in what looks to be this build script which has SELinux enforcing set. I'd try either a different distro base or look into how to get SELinux integration with dockerd up and running .

[Wz5m165c]

Thank you for the suggestions! I will try to look into the SELinux issue. Due to business reasons changing the distro is out of the question, sadly.

However, the behavior of docker has changed between versions 18.03.1-ce, build 9ee9f40, where this issue is not reproducible, and 18.06.0-ce, build 0ffa825 where this issue is present.

Can you say which version behaves more correctly?

(I tested the older version by first stopping everything incl. the dockerd, purging all images, installing sudo apt install docker-ce=18.03.1~ce-0~ubuntu and running the reproduction steps above which pulled a fresh copy of CentOS; with this version, everything worked. Then I upgraded docker-ce to version 18.06.0~ce~3-0~ubuntu and did the same purge & install again; with this version I got again the error message.)

Versions used:

 docker-ce | 18.06.0~ce~3-0~ubuntu | https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages
 docker-ce | 18.03.1~ce-0~ubuntu | https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages

[cwgem]

Can you provide docker info for the version where you could reproduce this issue and the version where you couldn't? Tasks around container creation and execution are often offloaded to containerd and runc. During the dockerd build those are pinpointed at specific versions as you can see from the build scripts. Getting docker info from both indicates what commit hash both of those were pinned to in the specific docker builds.

[Wz5m165c]

Here's the docker info output of the version 18.03.1-ce, build 9ee9f40 where exec gives no error:

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 10
Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-131-generic
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 6
Total Memory: 21.22GiB
ID: IBWI:5TNL:NHCS:NWVJ:SFV7:UUR6:AKZY:6UDF:TK4G:CUHH:DDBN:DBFH
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

And for 18.06.0-ce, build 0ffa825 where exec gives an error:

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 10
Server Version: 18.06.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d64c661f1d51c48782c9cec8fda7604785f93587
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-131-generic
Operating System: Ubuntu 16.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 6
Total Memory: 21.22GiB
ID: IBWI:5TNL:NHCS:NWVJ:SFV7:UUR6:AKZY:6UDF:TK4G:CUHH:DDBN:DBFH
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additionally, I ran the dockerd with debug enabled. (Note that in this test I created /test-dir owned by user test and changed the WORKDIR to that, going on a fishing expedition and see if SELinux home directory context would have any effect.) You can find the debug output (for commands run & exec) as attachments. There's a stream copy error: reading from a closed fifo debug message in 18.06.
docker-18.03-debug.txt
docker-18.06-debug.txt

[jhfrontz]

Was there any update/workaround for this (aside from putting -w /root on all the docker exec -u root invocations)? I'm hitting the same issue -- I suspect it's something quirky about the contents of the centos Dockerfile (but I'm not savvy enough to know what).