Mounting volumes fails with "error setting label on mount source".

[bardiharborow]

Description

The following started failing on my system after a (presumably CoreOS update induced) reboot ~10 hours ago:

$ docker run --rm --name test --mount source=test,destination=/test alpine
/run/torcx/bin/docker: Error response from daemon: error setting label on mount source '': no such file or directory.
ERRO[0000] error waiting for container: context canceled

Output of docker version:

Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 18:54:54 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.0-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       0ffa825
  Built:            Wed Jul 18 18:54:54 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

Containers: 4
 Running: 3
 Paused: 0
 Stopped: 1
Images: 8
Server Version: 18.06.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d64c661f1d51c48782c9cec8fda7604785f93587
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: v0.13.2 (expected: fec3683b971d9c3ef73f284f176672c44b448662)
Security Options:
 seccomp
  Profile: default
 selinux
Kernel Version: 4.17.9-coreos
Operating System: Container Linux by CoreOS 1849.0.0 (Rhyolite)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 988.2MiB
Name: <REDACTED>
ID: <REDACTED>
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: true

[holgum]

We are seeing the same error on RHEL 7.5 (selinux in permissive mode) with 18.06.
Curiously, it only happens with Swarm services and named volumes. If we use the same named volume with a standalone docker-compose instead of service, the volume mounts fine.

[mikee]

I'm seeing the same issue @holgum
Swarm deploy fails with error
Aug 5 21:59:39 ip-10-109-15-109 dockerd: time="2018-08-05T21:59:39.851169023Z" level=error msg="fatal task error" error="starting container failed: error setting label on mount source '': no such file or directory" module=node/agent/taskmanager node.id=ccrdyiatrs04gm419ktjhdngp service.id=bw6pq8 manfo44uzlynpmxb5t0 task.id=pykcv4cfdx2beeoh8o9ve9qaw`

docker version output

Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 19:08:18 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.0-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       0ffa825
  Built:            Wed Jul 18 19:10:42 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Docker Info

Containers: 6
 Running: 0
 Paused: 0
 Stopped: 6
Images: 2
Server Version: 18.06.0-ce
Storage Driver: devicemapper
 Pool Name: docker-thinpool
 Pool Blocksize: 524.3kB
 Base Device Size: 10.74GB
 Backing Filesystem: xfs
 Udev Sync Supported: true
 Data Space Used: 3.576GB
 Data Space Total: 42.95GB
 Data Space Available: 39.37GB
 Metadata Space Used: 1.815MB
 Metadata Space Total: 5.369GB
 Metadata Space Available: 5.367GB
 Thin Pool Minimum Free Space: 4.295GB
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.146-RHEL7 (2018-01-22)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: ccrdyiatrs04gm419ktjhdngp
 Is Manager: false
 Node Address: XXX.XXX.XXX.XXX
 Manager Addresses:
  XXX.XXX.XXX.XXX:2377
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: docker-init
containerd version: d64c661f1d51c48782c9cec8fda7604785f93587
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 seccomp
  Profile: default
 selinux
Kernel Version: 3.10.0-327.10.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.26GiB
Name: ip-XXXXXXXXXX.ap-southeast-2.compute.internal
ID: M3HR:ZU5D:YFYH:B6QS:FCYY:LM2V:5FK2:BX5Z:2LJU:RNEK:WZMC:3ISI
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: bridge-nf-call-ip6tables is disabled

Starting the container with docker-compose up -d rather than swarm works fine.

[mikee]

I've tested with permissive and enforcing SELinux mode with no difference

[holgum]

We’re working around it for the time being by using bind mounts on our services instead of named volumes. Hoping to see a fix soon (or at least confirmation of a bug).

[mikee]

Downgrading to 18.03 gets me past this issue

[michaeldayCA]

We are suffering critical failures in docker 18.06 because of this issue also.

The error is

Aug 16 16:00:32 localhost.localdomain dockerd[23857]: time="2018-08-16T16:00:32.901702931-07:00" level=error msg="fatal task error" error="starting container failed: error setting label on mount source '': no such file or directory" module=node/agent/taskmanager node.id=fwx50v7r1iiqkw0o3rmdbua28 service.id=tc2kzlg2cujplgmgicn5x2sog task.id=shndqduy7plkvvia52rkkm1nb

This manifests in the extremely strange behavior of some of our services automatically scaling themselves down to 0/0 copies.

ID                  NAME                        MODE                REPLICAS            IMAGE                                             
30wgn20age0v        product_service1            global              1/1                 ourdockerregistry.com/subdirectory/service1:latest              
mvrvczw16997        product_service2            global              0/1                 ourdockerregistry.com/subdirectory/service2:latest                       
7o6f7sq8q2vf        product_service3            global              0/1                 ourdockerregistry.com/subdirectory/service3:latest                 
oafm5f492aaf        product_service4            global              0/0                 ourdockerregistry.com/subdirectory/service4:latest                    

Downgrading to docker 18.03 solves the issue.

[icedream]

Confirming this issue, this is preventing me from firing up containers with named volumes in a Swarm setup with CoreOS instances (alpha and beta, and this may affect stable too as soon as it is updated to Docker 18.06).

Can't say yet about non-CoreOS setups but I'll try it out soon.

[sixcorners]

I don't think this is a CLI issue. Are we all using swarm? It looks like OP has swarm set to inactive.
I'm probably going to create an issue for this in the moby project.

[thaJeztah]

Looks like this should be resolved through moby/moby#37739 /cc @kolyshkin to confirm