docker run --mount creates a directory owned by root if directory is removed at a wrong time

[agoulti]

Description

If a "docker run" is started using "--mount" and the mounted directory is removed at a certain point during the invocation, it will create a directory owned by root.
This seems a race condition, since the timing or directory removal matters

Steps to reproduce the issue:

The following script has a good chance of reproducing the issue. Changing the timing of "sleep 0.5" changes the possibility of problems.

sudo rm -rf /tmp/try/nnn
mkdir -p /tmp/try/nnn
echo "before start:"  `ls -ld /tmp/try/nnn`
docker run -u 277174:89939 --mount type=bind,src=/tmp/try/nnn,dst=/test debian  sleep 3 &
sleep 0.5
rm -rf /tmp/try/nnn
echo "before sleep:"  `ls -ld /tmp/try/nnn`
sleep 3
echo "after sleep:"  `ls -ld /tmp/try/nnn`

Describe the results you received:

More often than not, this results in the following messages:

before start: drwxr-x--- 2 agoulti primarygroup 4096 May 16 14:48 /tmp/try/nnn
after start: drwxr-x--- 2 agoulti primarygroup 4096 May 16 14:48 /tmp/try/nnn
ls: cannot access '/tmp/try/nnn': No such file or directory
before sleep:
after sleep: drwxr-xr-x 2 root root 4096 May 16 14:48 /tmp/try/nnn

Describe the results you expected:

--mount is said not to create a directory if it doesn't exist.
I am creating bad running conditions, so I expect docker to fail.
However, I expect /tmp/try/nnn to never be created.

Additional information you deem important (e.g. issue happens only occasionally):

This is a race condition, so the results are not guaranteed.
On my machine I get a problem around half the time.
Changing "sleep 0.5" duration changes the behaviour, you might need to tune it to reproduce on another machine.

Too short and it correctly executes with:
docker: Error response from daemon: invalid mount config for type "bind": bind source path does not exist.

Too long and it correctly executes with:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"rootfs_linux.go:58: mounting \\\"/tmp/try/nnn\\\" to rootfs \\\"/usr/[...]merged\\\" at \\\"/test\\\" caused \\\"stat /tmp/try/nnn: no such file or directory\\\"\"": unknown.

Output of docker version:

Client:
 Version:       18.03.0-ce
 API version:   1.37
 Go version:    go1.9.4
 Git commit:    0520e24
 Built: Wed Mar 21 23:10:06 2018
 OS/Arch:       linux/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:      18.03.0-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   0520e24
  Built:        Wed Mar 21 23:08:35 2018
  OS/Arch:      linux/amd64
  Experimental: false

Output of docker info:

Containers: 813
 Running: 0
 Paused: 0
 Stopped: 813
Images: 26
Server Version: 18.03.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfd04396dc68220d1cecbe686a6cc3aa5ce3667c
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.9.0-6-amd64
Operating System: Debian GNU/Linux buster/sid
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 31.35GiB
Name: agoulti.wat.corp.google.com
ID: MMPA:UN32:XDPC:C6LU:IQ4F:WXSG:S2TS:23E4:GW73:4WSB:PKW5:RMWW
Docker Root Dir: /usr/local/google/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):
Desktop Linux machine

[thaJeztah]

/cc @cpuguy83

[agoulti]

Do we know if there's a plan for someone to look at this?

[cpuguy83]

I did look into this. There's definitely a race here with how bind mounts are handled. I held off fixing this until after some major refactoring was merged... which it now is. I'm currently on vacation.

…

On Mon, Jun 11, 2018 at 10:14 AM agoulti ***@***.***> wrote: Do we know if there's a plan for someone to look at this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#37083 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAwxZkdmNCb-ryifxMxJ_WJeolUPhtnQks5t7qV-gaJpZM4UB3Bq> .

[agoulti]

Great, thanks for a quick reply!
Have a great vacation, and please keep me posted on the fix when you're back!

[cpuguy83]

@agoulti Can you test #37378?

[agoulti]

Thanks for fixing this!
I've played with the new version and I cannot recreate the race condition, so it seems fixed!

Any ideas on which Docker version this might be included in?

Thanks!
-Alexandra

[cpuguy83]

Assuming it gets merged soon, it will be 18.06+1 (since 18.06 RC is already cut).