Failed to bind-mount /sys/fs/cgroup when user namespace is enabled. #27629

yoshiokatsuneo · 2016-10-21T09:40:31Z

Description
Starting a container with mouting /sys/fs/cgroup failed if user namespace is enabled.
There is no problem if I disabled the user namespace.

Steps to reproduce the issue:

Start docker daemon with user namespace like:

# /usr/bin/docker daemon -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver zfs --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=generic --userns-remap=default

Run following command to start a container.

# docker run  -v /sys/fs/cgroup:/sys/fs/cgroup:ro hello-world

Describe the results you received:
Error messages:
(operation not permitted)

docker: Error response from daemon: invalid header field value "oci runtime error: container_linux.go:247: starting container process caused \"process_linux.go:359: container init caused \\\"rootfs_linux.go:53: mounting \\\\\\\"/sys/fs/cgroup\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/231072.231072/zfs/graph/6b22b274858c2f27e74d9de6af68f97f6c9830d091b20d64b083903a8fce25fe\\\\\\\" at \\\\\\\"/var/lib/docker/231072.231072/zfs/graph/6b22b274858c2f27e74d9de6af68f97f6c9830d091b20d64b083903a8fce25fe/sys/fs/cgroup\\\\\\\" caused \\\\\\\"operation not permitted\\\\\\\"\\\"\"\n".

Describe the results you expected:
Container starts.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

# docker version
Client:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 18:29:41 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 18:29:41 2016
 OS/Arch:      linux/amd64

Output of docker info:

# docker info
Containers: 14
 Running: 3
 Paused: 0
 Stopped: 11
Images: 4
Server Version: 1.12.2
Storage Driver: zfs
 Zpool: zpool-docker
 Zpool Health: ONLINE
 Parent Dataset: zpool-docker/docker
 Space Used By Parent: 19456
 Space Available: 103498376192
 Parent Quota: no
 Compression: off
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null overlay host bridge
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.4.0-43-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.858 GiB
Name: paizaterm-prod2
ID: OR4Z:NLM5:7E5H:KYTH:SB2T:TUSF:6Y4T:YL2S:PBBN:UKLB:LSVZ:GU2R
Docker Root Dir: /var/lib/docker/231072.231072
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Labels:
 provider=generic
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):

The text was updated successfully, but these errors were encountered:

justincormack · 2016-10-21T10:16:16Z

Can you test on 1.12.1?

yoshiokatsuneo · 2016-10-21T13:47:02Z

@justincormack How can I downgrade to 1.12.1 ?

justincormack · 2016-10-21T14:44:49Z

@yoshiokatsuneo something like apt-get install docker-engine=1.12.1-0~xenial should work (guessing the exact package name, I think thats right)

yoshiokatsuneo · 2016-10-21T15:36:18Z

@justincormack Thanks. So, I tried to downgrade to 1.12.1, but I got the similar error again.

# docker run  -v /sys/fs/cgroup:/sys/fs/cgroup:ro hello-world
docker: Error response from daemon: oci runtime error: rootfs_linux.go:53: mounting "/var/lib/docker/231072.231072/zfs/graph/6af1698465f563c0752d5a384bd53718ea955752f981ba6099fc14966c25a3ea/sys/fs/cgroup" to rootfs "/var/lib/docker/231072.231072/zfs/graph/6af1698465f563c0752d5a384bd53718ea955752f981ba6099fc14966c25a3ea" caused "operation not permitted".
# docker version
Client:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:33:38 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.1
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   23cf638
 Built:        Thu Aug 18 05:33:38 2016
 OS/Arch:      linux/amd64

justincormack · 2016-10-21T18:06:30Z

ok, unrelated to another issue then.

Guessing this is around the differences in mount permissions in userns? @estesp ?

estesp · 2016-11-01T12:40:11Z

I cannot reproduce on the latest xenial kernel (4.4.0-45-generic), which is not very far from yours, so I'm not sure it has any relation to that. However, I wonder if zfs is somehow involved. I do not have a zfs disk to try, but maybe you could try running the daemon on a separate non-zfs mount with aufs or overlay2 graphdrivers?

Specifically, I can run (on docker 1.12.3 with user namespaces enabled, and aufs driver):

docker run -v /sys/fs/cgroup:/sys/fs/cgroup:ro hello-world

and I get the Hello from Docker! message with no errors.

yoshiokatsuneo · 2016-11-11T07:53:50Z

@estesp I can reproduce the issue on the aufs, too...
I tested on:

Ubuntu 16.05.1
Docker 1.12.3
Linux 4.4.0-43-generic

jiangytcn · 2017-09-20T04:27:15Z

Solved by upgrade the host kernel to Linux ubuntu-xenial 4.4.0-96-generic

ubuntu@ubuntu-xenial:~$ sudo docker version
sudo: unable to resolve host ubuntu-xenial
Client:
Version: 17.06.2-ce
API version: 1.30
Go version: go1.8.3
Git commit: cec0b72
Built: Tue Sep 5 20:00:17 2017
OS/Arch: linux/amd64

Server:
Version: 17.06.2-ce
API version: 1.30 (minimum version 1.12)
Go version: go1.8.3
Git commit: cec0b72
Built: Tue Sep 5 19:59:11 2017
OS/Arch: linux/amd64
Experimental: false

vrosales · 2017-10-04T16:46:23Z

I can reproduce:

$ uname -a
Linux victor-dev 4.4.0-96-generic #119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

$ docker version
Client:
Version: 17.09.0-ce
API version: 1.32
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:42:18 2017
OS/Arch: linux/amd64

Server:
Version: 17.09.0-ce
API version: 1.32 (minimum version 1.12)
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:40:56 2017
OS/Arch: linux/amd64
Experimental: false

$ docker run -v /sys:/sys:ro hello-world
container_linux.go:265: starting container process caused "process_linux.go:368: container init caused "rootfs_linux.go:57: mounting \"/sys\" to rootfs \"/var/lib/docker/165536.165536/overlay2/8e9bb70c12e61dc7e2e818c09605a5dee7d2bd7248e1b3562734411e3f080ab0/merged\" at \"/var/lib/docker/165536.165536/overlay2/8e9bb70c12e61dc7e2e818c09605a5dee7d2bd7248e1b3562734411e3f080ab0/merged/sys\" caused \"operation not permitted\"""
docker: Error response from daemon: oci runtime error: container_linux.go:265: starting container process caused "process_linux.go:368: container init caused "rootfs_linux.go:57: mounting \"/sys\" to rootfs \"/var/lib/docker/165536.165536/overlay2/8e9bb70c12e61dc7e2e818c09605a5dee7d2bd7248e1b3562734411e3f080ab0/merged\" at \"/var/lib/docker/165536.165536/overlay2/8e9bb70c12e61dc7e2e818c09605a5dee7d2bd7248e1b3562734411e3f080ab0/merged/sys\" caused \"operation not permitted\""".
ERRO[0000] error waiting for container: context canceled

thaJeztah · 2023-09-18T11:03:33Z

Let me close this ticket for now, as it looks like it went stale.

GordonTheTurtle added the version/1.12 label Oct 21, 2016

akalipetis mentioned this issue Nov 1, 2016

Cannot mount /run/docker/plugins when in userns enabled Docker daemon #27935

Closed

thaJeztah added the area/security/userns label Jun 28, 2017

thaJeztah changed the title ~~Failed to mount /sys/fs/cgroup when user namespace is enabled.~~ Failed to bind-mount /sys/fs/cgroup when user namespace is enabled. Sep 18, 2023

thaJeztah closed this as not planned Won't fix, can't repro, duplicate, stale Sep 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to bind-mount /sys/fs/cgroup when user namespace is enabled. #27629

Failed to bind-mount /sys/fs/cgroup when user namespace is enabled. #27629

yoshiokatsuneo commented Oct 21, 2016 •

edited

justincormack commented Oct 21, 2016

yoshiokatsuneo commented Oct 21, 2016

justincormack commented Oct 21, 2016

yoshiokatsuneo commented Oct 21, 2016

justincormack commented Oct 21, 2016

estesp commented Nov 1, 2016

yoshiokatsuneo commented Nov 11, 2016

jiangytcn commented Sep 20, 2017

vrosales commented Oct 4, 2017 •

edited

thaJeztah commented Sep 18, 2023

Failed to bind-mount /sys/fs/cgroup when user namespace is enabled. #27629

Failed to bind-mount /sys/fs/cgroup when user namespace is enabled. #27629

Comments

yoshiokatsuneo commented Oct 21, 2016 • edited

justincormack commented Oct 21, 2016

yoshiokatsuneo commented Oct 21, 2016

justincormack commented Oct 21, 2016

yoshiokatsuneo commented Oct 21, 2016

justincormack commented Oct 21, 2016

estesp commented Nov 1, 2016

yoshiokatsuneo commented Nov 11, 2016

jiangytcn commented Sep 20, 2017

vrosales commented Oct 4, 2017 • edited

thaJeztah commented Sep 18, 2023

yoshiokatsuneo commented Oct 21, 2016 •

edited

vrosales commented Oct 4, 2017 •

edited