New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker service and volume mount with selinux flag: Mount path for relabel is empty, triggering task error. #31137

Closed
CyrilPeponnet opened this Issue Feb 18, 2017 · 4 comments

Comments

Projects
None yet
5 participants
@CyrilPeponnet

CyrilPeponnet commented Feb 18, 2017

Description

When the docker daemon is using the --selinux-enabled accessing a volume from a service is leading to a task error like

Feb 14 17:56:10 localhost dockerd[12924]: time="2017-02-14T17:56:10.737990729Z" level=error msg="fatal task error" error="SELinux relabeling of  is not allowed: \"no such file or directory\"" module="node/agent/taskmanager" task.id=3wk491fp4m2qy1s4pjkuoiwoo`

Steps to reproduce the issue:

  1. Install docker > 1.13 enabled --selinux-enabled flag
  2. docker volume create --name foo
  3. docker service create --replicas 1 --mount type=volume,src=foo,dst=/test --restart-condition none --name test busybox ls /test/

Describe the results you received:

Task is rejected.

Describe the results you expected:

Task is accepted.

Additional information you deem important (e.g. issue happens only occasionally):

It works fine with 1.12.X, if I disable the --selinux-enabled flag it works fine with both 1.12.X, 1.13.X

Selinux on my host is permissive. Nothing is avc denied while auditing.

Output of docker version:

Client:
 Version:      1.13.1
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 08:47:51 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.1
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 08:47:51 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 1.13.1
Storage Driver: overlay
 Backing Filesystem: extfs
 Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1
runc version: 9df8b306d01f59d3a8029be411de015b7304dd8f
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
 selinux
Kernel Version: 4.7.3-coreos-r2
Operating System: Container Linux by CoreOS 1235.9.0 (Ladybug)
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.958 GiB
Name: localhost
ID: HAUR:EMA2:NHQK:JLQU:DINB:SFNU:HX3F:YYRT:HG5F:DIYU:SMFG:SP6Y
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

Virtual Box Core OS (stable, beta, alpha same issue).

@CyrilPeponnet CyrilPeponnet changed the title from Docker service and volume mount: Mount path for relabel is empty, triggering task error. to Docker service and volume mount with selinux flag: Mount path for relabel is empty, triggering task error. Feb 18, 2017

@CyrilPeponnet

This comment has been minimized.

Show comment
Hide comment
@CyrilPeponnet

CyrilPeponnet Feb 18, 2017

Adding full debug log:

Feb 18 01:42:44 localhost dockerd[4214]: time="2017-02-18T01:42:44.899742635Z" level=debug msg="Calling GET /_ping"
Feb 18 01:42:44 localhost dockerd[4214]: time="2017-02-18T01:42:44.900866905Z" level=debug msg="Calling POST /v1.26/services/create"
Feb 18 01:42:44 localhost dockerd[4214]: time="2017-02-18T01:42:44.901196529Z" level=debug msg="form data: {\"EndpointSpec\":{},\"Mode\":{\"Replicated\":{\"Replicas\":1}},\"Name\":\"test\",\"TaskTemplate\":{\"ContainerSpec\":{\"Args\":[\"tail\",\"-f\",\"/dev/null\"],\"DNSConfig\":{},\"Image\":\"busybox\",\"Mounts\":[{\"Source\":\"foo\",\"Target\":\"/test\",\"Type\":\"volume\"}]},\"ForceUpdate\":0,\"Placement\":{},\"Resources\":{\"Limits\":{},\"Reservations\":{}},\"RestartPolicy\":{\"Condition\":\"none\"}},\"UpdateConfig\":{\"FailureAction\":\"pause\",\"MaxFailureRatio\":0,\"Parallelism\":1}}"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.033921595Z" level=debug msg="pinning image busybox by digest: busybox:latest@sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.036841447Z" level=debug msg="Service p5am04jardrwj5qo0lw6muk2f was scaled up from 0 to 1 instances"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.090383005Z" level=debug msg="assigning to node qlxsnppn9kp8voqkc0o4v34uo" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.193901295Z" level=debug msg="(*worker).Update" len(assignments)=1 module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.194558204Z" level=debug msg="(*worker).reconcileSecrets" len(removedSecrets)=0 len(updatedSecrets)=0 module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.194939190Z" level=debug msg="(*worker).reconcileTaskState" len(removedTasks)=0 len(updatedTasks)=1 module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.195226034Z" level=debug msg=assigned module="node/agent" task.desiredstate=RUNNING task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.195602835Z" level=debug msg="state changed" module="node/agent" state.desired=RUNNING state.transition="ASSIGNED->ACCEPTED" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.196430812Z" level=debug msg="state changed" module="node/agent/taskmanager" state.desired=RUNNING state.transition="ASSIGNED->ACCEPTED" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.197159280Z" level=debug msg="(*Agent).UpdateTaskStatus" module="node/agent" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.199232102Z" level=debug msg="task status updated" method="(*Dispatcher).processUpdates" module=dispatcher state.transition="ASSIGNED->ACCEPTED" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.199929996Z" level=debug msg="state changed" module="node/agent/taskmanager" state.desired=RUNNING state.transition="ACCEPTED->PREPARING" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.200614069Z" level=debug msg="task status reported" module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.201408694Z" level=debug msg="(*Agent).UpdateTaskStatus" module="node/agent" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.202019360Z" level=debug msg="task status reported" module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.216422426Z" level=error msg="fatal task error" error="SELinux relabeling of  is not allowed: \"no such file or directory\"" module="node/agent/taskmanager" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.216769353Z" level=debug msg="state changed" module="node/agent/taskmanager" state.desired=RUNNING state.transition="PREPARING->REJECTED" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.217602920Z" level=debug msg="(*Agent).UpdateTaskStatus" module="node/agent" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.218375898Z" level=debug msg="task status reported" module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.219535947Z" level=debug msg="(*Agent).UpdateTaskStatus" module="node/agent" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.220127785Z" level=debug msg="task status reported" module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.301127807Z" level=debug msg="task status updated" method="(*Dispatcher).processUpdates" module=dispatcher state.transition="ACCEPTED->REJECTED" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.405989576Z" level=debug msg="(*worker).Update" len(assignments)=1 module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.406430447Z" level=debug msg="(*worker).reconcileSecrets" len(removedSecrets)=0 len(updatedSecrets)=0 module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.406707066Z" level=debug msg="(*worker).reconcileTaskState" len(removedTasks)=0 len(updatedTasks)=1 module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.406989229Z" level=debug msg=assigned module="node/agent" task.desiredstate=SHUTDOWN task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.409536206Z" level=debug msg="(*Agent).UpdateTaskStatus" module="node/agent" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.410607165Z" level=debug msg="task status reported" module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.505207252Z" level=debug msg="task status updated" method="(*Dispatcher).processUpdates" module=dispatcher state.transition="REJECTED->REJECTED" task.id=pdik9o71kjhdc3p98lpci8n1o

Adding full debug log:

Feb 18 01:42:44 localhost dockerd[4214]: time="2017-02-18T01:42:44.899742635Z" level=debug msg="Calling GET /_ping"
Feb 18 01:42:44 localhost dockerd[4214]: time="2017-02-18T01:42:44.900866905Z" level=debug msg="Calling POST /v1.26/services/create"
Feb 18 01:42:44 localhost dockerd[4214]: time="2017-02-18T01:42:44.901196529Z" level=debug msg="form data: {\"EndpointSpec\":{},\"Mode\":{\"Replicated\":{\"Replicas\":1}},\"Name\":\"test\",\"TaskTemplate\":{\"ContainerSpec\":{\"Args\":[\"tail\",\"-f\",\"/dev/null\"],\"DNSConfig\":{},\"Image\":\"busybox\",\"Mounts\":[{\"Source\":\"foo\",\"Target\":\"/test\",\"Type\":\"volume\"}]},\"ForceUpdate\":0,\"Placement\":{},\"Resources\":{\"Limits\":{},\"Reservations\":{}},\"RestartPolicy\":{\"Condition\":\"none\"}},\"UpdateConfig\":{\"FailureAction\":\"pause\",\"MaxFailureRatio\":0,\"Parallelism\":1}}"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.033921595Z" level=debug msg="pinning image busybox by digest: busybox:latest@sha256:817a12c32a39bbe394944ba49de563e085f1d3c5266eb8e9723256bc4448680e"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.036841447Z" level=debug msg="Service p5am04jardrwj5qo0lw6muk2f was scaled up from 0 to 1 instances"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.090383005Z" level=debug msg="assigning to node qlxsnppn9kp8voqkc0o4v34uo" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.193901295Z" level=debug msg="(*worker).Update" len(assignments)=1 module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.194558204Z" level=debug msg="(*worker).reconcileSecrets" len(removedSecrets)=0 len(updatedSecrets)=0 module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.194939190Z" level=debug msg="(*worker).reconcileTaskState" len(removedTasks)=0 len(updatedTasks)=1 module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.195226034Z" level=debug msg=assigned module="node/agent" task.desiredstate=RUNNING task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.195602835Z" level=debug msg="state changed" module="node/agent" state.desired=RUNNING state.transition="ASSIGNED->ACCEPTED" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.196430812Z" level=debug msg="state changed" module="node/agent/taskmanager" state.desired=RUNNING state.transition="ASSIGNED->ACCEPTED" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.197159280Z" level=debug msg="(*Agent).UpdateTaskStatus" module="node/agent" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.199232102Z" level=debug msg="task status updated" method="(*Dispatcher).processUpdates" module=dispatcher state.transition="ASSIGNED->ACCEPTED" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.199929996Z" level=debug msg="state changed" module="node/agent/taskmanager" state.desired=RUNNING state.transition="ACCEPTED->PREPARING" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.200614069Z" level=debug msg="task status reported" module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.201408694Z" level=debug msg="(*Agent).UpdateTaskStatus" module="node/agent" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.202019360Z" level=debug msg="task status reported" module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.216422426Z" level=error msg="fatal task error" error="SELinux relabeling of  is not allowed: \"no such file or directory\"" module="node/agent/taskmanager" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.216769353Z" level=debug msg="state changed" module="node/agent/taskmanager" state.desired=RUNNING state.transition="PREPARING->REJECTED" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.217602920Z" level=debug msg="(*Agent).UpdateTaskStatus" module="node/agent" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.218375898Z" level=debug msg="task status reported" module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.219535947Z" level=debug msg="(*Agent).UpdateTaskStatus" module="node/agent" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.220127785Z" level=debug msg="task status reported" module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.301127807Z" level=debug msg="task status updated" method="(*Dispatcher).processUpdates" module=dispatcher state.transition="ACCEPTED->REJECTED" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.405989576Z" level=debug msg="(*worker).Update" len(assignments)=1 module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.406430447Z" level=debug msg="(*worker).reconcileSecrets" len(removedSecrets)=0 len(updatedSecrets)=0 module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.406707066Z" level=debug msg="(*worker).reconcileTaskState" len(removedTasks)=0 len(updatedTasks)=1 module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.406989229Z" level=debug msg=assigned module="node/agent" task.desiredstate=SHUTDOWN task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.409536206Z" level=debug msg="(*Agent).UpdateTaskStatus" module="node/agent" task.id=pdik9o71kjhdc3p98lpci8n1o
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.410607165Z" level=debug msg="task status reported" module="node/agent"
Feb 18 01:42:47 localhost dockerd[4214]: time="2017-02-18T01:42:47.505207252Z" level=debug msg="task status updated" method="(*Dispatcher).processUpdates" module=dispatcher state.transition="REJECTED->REJECTED" task.id=pdik9o71kjhdc3p98lpci8n1o
@harsh-px

This comment has been minimized.

Show comment
Hide comment
@harsh-px

harsh-px May 24, 2017

We are hitting the same issue. Can we please know the plan to fix this?

Below command is creating a service using a local volume.

docker service create \
    --mount type=volume,src=foo,dst=/test \
    --name testvol \
    busybox /bin/sh -c "while true; do touch /test/bar; sleep 5; done"

Service tasks keep getting rejected.

# docker service ps testvol
ID            NAME           IMAGE           NODE            DESIRED STATE  CURRENT STATE            ERROR                        PORTS
pophye7gmczk  testvol.1      busybox:latest  harsh-centos-2  Ready          Rejected 1 second ago    "no such file or directory"  
kdm8ysqt761a   \_ testvol.1  busybox:latest  harsh-centos-2  Shutdown       Rejected 6 seconds ago   "no such file or directory"  
p81r9rcs7o7t   \_ testvol.1  busybox:latest  harsh-centos-2  Shutdown       Rejected 11 seconds ago  "no such file or directory"  
0tshoswffhgr   \_ testvol.1  busybox:latest  harsh-centos-3  Shutdown       Rejected 11 seconds ago  "no such file or directory"  

On removing the --selinux-enabled flag from dockerd, the issue gets resolved.

# docker info
Containers: 15
 Running: 14
 Paused: 0
 Stopped: 1
Images: 34
Server Version: 17.03.1-ce
Storage Driver: overlay
 Backing Filesystem: extfs
 Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local pxd
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: ypzqu8qah9cx7vxxf51hp7aad
 Is Manager: true
 ClusterID: yfdza43bz45gj4zyp0v477se9
 Managers: 3
 Nodes: 5
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  External CAs:
    cfssl: https://147.75.69.57:12381/api/v1/cfssl/sign
    cfssl: https://147.75.69.37:12381/api/v1/cfssl/sign
    cfssl: https://147.75.108.9:12381/api/v1/cfssl/sign
 Node Address: 147.75.108.9
 Manager Addresses:
  147.75.108.9:2377
  147.75.69.37:2377
  147.75.69.57:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
 selinux
Kernel Version: 3.10.0-327.22.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.785 GiB
Name: harsh-centos-1
ID: HXI4:EKPI:K2XK:LN6U:3JWY:QXLF:X4UT:TH4J:ZA3J:A4MX:3J3Z:YBL5
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

journalctl -r shows the following related entries.

May 24 00:15:19 harsh-centos-2 dockerd[24299]: time="2017-05-24T00:15:19.055734746Z" level=error msg="fatal task error" error="no such file or directory" module="node/agent/taskmanager" task.id=ychezsxverp4s2ak6njn35ow2
May 24 00:15:19 harsh-centos-2 kernel: SELinux: initialized (dev overlay, type overlay), uses xattr
May 24 00:15:14 harsh-centos-2 dockerd[24299]: time="2017-05-24T00:15:14.052600635Z" level=error msg="fatal task error" error="no such file or directory" module="node/agent/taskmanager" task.id=pophye7gmczkv430ppj6jnew1
May 24 00:15:14 harsh-centos-2 kernel: SELinux: initialized (dev overlay, type overlay), uses xattr
May 24 00:15:09 harsh-centos-2 dockerd[24299]: time="2017-05-24T00:15:09.047270845Z" level=error msg="fatal task error" error="no such file or directory" module="node/agent/taskmanager" task.id=kdm8ysqt761a38zbkeqk37yrn
May 24 00:15:09 harsh-centos-2 kernel: SELinux: initialized (dev overlay, type overlay), uses xattr
May 24 00:15:04 harsh-centos-2 dockerd[24299]: time="2017-05-24T00:15:04.043340874Z" level=error msg="fatal task error" error="no such file or directory" module="node/agent/taskmanager" task.id=p81r9rcs7o7ti53yenee1p79c
May 24 00:15:04 harsh-centos-2 kernel: SELinux: initialized (dev overlay, type overlay), uses xattr

harsh-px commented May 24, 2017

We are hitting the same issue. Can we please know the plan to fix this?

Below command is creating a service using a local volume.

docker service create \
    --mount type=volume,src=foo,dst=/test \
    --name testvol \
    busybox /bin/sh -c "while true; do touch /test/bar; sleep 5; done"

Service tasks keep getting rejected.

# docker service ps testvol
ID            NAME           IMAGE           NODE            DESIRED STATE  CURRENT STATE            ERROR                        PORTS
pophye7gmczk  testvol.1      busybox:latest  harsh-centos-2  Ready          Rejected 1 second ago    "no such file or directory"  
kdm8ysqt761a   \_ testvol.1  busybox:latest  harsh-centos-2  Shutdown       Rejected 6 seconds ago   "no such file or directory"  
p81r9rcs7o7t   \_ testvol.1  busybox:latest  harsh-centos-2  Shutdown       Rejected 11 seconds ago  "no such file or directory"  
0tshoswffhgr   \_ testvol.1  busybox:latest  harsh-centos-3  Shutdown       Rejected 11 seconds ago  "no such file or directory"  

On removing the --selinux-enabled flag from dockerd, the issue gets resolved.

# docker info
Containers: 15
 Running: 14
 Paused: 0
 Stopped: 1
Images: 34
Server Version: 17.03.1-ce
Storage Driver: overlay
 Backing Filesystem: extfs
 Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local pxd
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: ypzqu8qah9cx7vxxf51hp7aad
 Is Manager: true
 ClusterID: yfdza43bz45gj4zyp0v477se9
 Managers: 3
 Nodes: 5
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  External CAs:
    cfssl: https://147.75.69.57:12381/api/v1/cfssl/sign
    cfssl: https://147.75.69.37:12381/api/v1/cfssl/sign
    cfssl: https://147.75.108.9:12381/api/v1/cfssl/sign
 Node Address: 147.75.108.9
 Manager Addresses:
  147.75.108.9:2377
  147.75.69.37:2377
  147.75.69.57:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
 selinux
Kernel Version: 3.10.0-327.22.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.785 GiB
Name: harsh-centos-1
ID: HXI4:EKPI:K2XK:LN6U:3JWY:QXLF:X4UT:TH4J:ZA3J:A4MX:3J3Z:YBL5
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

journalctl -r shows the following related entries.

May 24 00:15:19 harsh-centos-2 dockerd[24299]: time="2017-05-24T00:15:19.055734746Z" level=error msg="fatal task error" error="no such file or directory" module="node/agent/taskmanager" task.id=ychezsxverp4s2ak6njn35ow2
May 24 00:15:19 harsh-centos-2 kernel: SELinux: initialized (dev overlay, type overlay), uses xattr
May 24 00:15:14 harsh-centos-2 dockerd[24299]: time="2017-05-24T00:15:14.052600635Z" level=error msg="fatal task error" error="no such file or directory" module="node/agent/taskmanager" task.id=pophye7gmczkv430ppj6jnew1
May 24 00:15:14 harsh-centos-2 kernel: SELinux: initialized (dev overlay, type overlay), uses xattr
May 24 00:15:09 harsh-centos-2 dockerd[24299]: time="2017-05-24T00:15:09.047270845Z" level=error msg="fatal task error" error="no such file or directory" module="node/agent/taskmanager" task.id=kdm8ysqt761a38zbkeqk37yrn
May 24 00:15:09 harsh-centos-2 kernel: SELinux: initialized (dev overlay, type overlay), uses xattr
May 24 00:15:04 harsh-centos-2 dockerd[24299]: time="2017-05-24T00:15:04.043340874Z" level=error msg="fatal task error" error="no such file or directory" module="node/agent/taskmanager" task.id=p81r9rcs7o7ti53yenee1p79c
May 24 00:15:04 harsh-centos-2 kernel: SELinux: initialized (dev overlay, type overlay), uses xattr
@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 May 24, 2017

Contributor

Thanks for reporting.
This is resolved in the upcoming 17.03.2 release.
It is also resolved in the 17.05 edge release.

Thanks again!

Contributor

cpuguy83 commented May 24, 2017

Thanks for reporting.
This is resolved in the upcoming 17.03.2 release.
It is also resolved in the 17.05 edge release.

Thanks again!

@cpuguy83 cpuguy83 closed this May 24, 2017

@thaJeztah thaJeztah added this to the 17.05.0 milestone May 24, 2017

@harsh-px

This comment has been minimized.

Show comment
Hide comment
@harsh-px

harsh-px May 24, 2017

Verified on docker 17.05.0-ce, Thanks !

harsh-px commented May 24, 2017

Verified on docker 17.05.0-ce, Thanks !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment