cgroup permission error?

[sentryrook]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

Expected behavior

Start kubernetes container

Actual behavior

Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:279: applying cgroup configuration for process caused "open /sys/fs/cgroup/cpuset/kubepods/cpuset.cpus: permission denied"": unknown
Error: failed to start containers: 42380d843697

Steps to reproduce the behavior

This happened while trying to install a kubernetes cluster install through virtualbox using a default ubuntu image.
I found many references to "not found", but now regarding permissions error.

Output of docker version:

docker version
Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 19:09:54 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       e68fc7a
  Built:            Mon Oct  1 14:25:33 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info:

docker info
Containers: 45
 Running: 0
 Paused: 0
 Stopped: 45
Images: 7
Server Version: 18.06.1-ce
Storage Driver: aufs
 Root Dir: /var/snap/docker/common/var-lib-docker/aufs
 Backing Filesystem: extfs
 Dirs: 112
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: N/A (expected: 69663f0bd4b60df09991c08812a60108003fa340)
init version: 949e6fa (expected: fec3683)
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-43-generic
Operating System: Ubuntu Core 16
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.947GiB
Name: kubemaster
ID: S4RA:EJSA:Z7PK:WQ77:YDSJ:HETX:OFX7:COJM:YN5E:IL2P:VMPY:3WDG
Docker Root Dir: /var/snap/docker/common/var-lib-docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 33
 Goroutines: 55
 System Time: 2018-12-26T22:05:55.167930015Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.)
kubelet --version
Kubernetes v1.13.1

uname -a
Linux kubemaster 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

ls -la /sys/fs/cgroup/cpuset/kubepods/cpuset.cpus
-rw-r--r-- 1 root root 0 Dec 26 22:04 /sys/fs/cgroup/cpuset/kubepods/cpuset.cpus

[geoko86]

I am running to the same error:

Output of docker version:

 Version:           18.09.0
 API version:       1.38 (downgraded from 1.39)
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:49:01 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       e68fc7a
  Built:            Mon Oct  1 14:25:33 2018
  OS/Arch:          linux/amd64
  Experimental:     false

and Output of docker info:

Containers: 184
 Running: 0
 Paused: 0
 Stopped: 184
Images: 10
Server Version: 18.06.1-ce
Storage Driver: aufs
 Root Dir: /var/snap/docker/common/var-lib-docker/aufs
 Backing Filesystem: extfs
 Dirs: 389
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: N/A (expected: 69663f0bd4b60df09991c08812a60108003fa340)
init version: 949e6fa (expected: fec3683)
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-42-generic
Operating System: Ubuntu Core 16
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 31.11GiB
Name: my-hostname
ID: TGYY:IXDX:HVPR:T64O:4456:4AN7:UAQF:LQK4:CSMP:ELRZ:7Z6Q:RPHD
Docker Root Dir: /var/snap/docker/common/var-lib-docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 30
 Goroutines: 55
 System Time: 2019-01-03T18:32:55.437510265Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

I am running bare metal server, not virtualization involved.

No SELinux running, AppArmor has been (temporarily) disabled, but still getting same issue:

Jan 03 18:38:30 my-hostname kubelet[28472]: E0103 18:38:30.462089   28472 pod_workers.go:190] Error syncing pod 44b569a35761491825f4e7253fbf0543 ("kube-scheduler-my-hostname_kube-system(44b569a35761491825f4e7253fbf0543)"), skipping: failed to "CreatePodSandbox" for "kube-scheduler-my-hostname_kube-system(44b569a35761491825f4e7253fbf0543)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-scheduler-my-hostname_kube-system(44b569a35761491825f4e7253fbf0543)\" failed: rpc error: code = Unknown desc = failed to start sandbox container for pod \"kube-scheduler-my-hostname\": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:279: applying cgroup configuration for process caused \\\"open /sys/fs/cgroup/cpuset/kubepods/cpuset.cpus: permission denied\\\"\": unknown"

running docker run hello-world gives me the expected results and runs without errors.

[thaJeztah]

Looking at; /var/snap/docker/common/var-lib-docker/aufs - wondering; did you use the official packages from download.docker.com to install, or are those packages provided by your distro (Ubuntu)?

[thaJeztah]

The versions of runc and init also don't seem to be on the expected version;

runc version: N/A (expected: 69663f0bd4b60df09991c08812a60108003fa340)
init version: 949e6fa (expected: fec3683)

[vojtechmares]

Hi, I am running into the same issue. Except I am not running Kubernetes, only single docker container, but in the end is is the same problem with cgroup.

Output of docker run hello-world:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "process_linux.go:311: applying cgroup configuration for process caused \"failed to write 1761 to cgroup.procs: write /sys/fs/cgroup/cpuset/docker/53bb393578bec09221f75048fa0206fad96ee8c81ff3d66b71bc0b5eab92e440/cgroup.procs: permission denied\"": unknown.
ERRO[0000] error waiting for container: context canceled

Output of docker info:

Containers: 9
 Running: 0
 Paused: 0
 Stopped: 9
Images: 3
Server Version: 18.09.1-ce
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 6635b4f0c6af3810594d2770f662f34ddc15b40d
init version: v0.18.0 (expected: fec3683b971d9c3ef73f284f176672c44b448662)
Security Options:
 userns
Kernel Version: 5.0.3
Operating System: Alpine Linux v3.9
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 4GiB
Name: vpsfree
ID: FMMO:BCET:VXAZ:QYBP:NK7L:NONR:IDYA:PFNT:PW2C:36SP:ERGJ:Q6PI
Docker Root Dir: /var/lib/docker/100.65533
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: true

Output of uname -a:

Linux vpsfree 5.0.3 #1-NixOS SMP Tue Mar 19 12:10:58 UTC 2019 x86_64 Linux

Output of ls /sys/fs/cgroup:

blkio             cpu,cpuacct       devices           memory            openrc            systemd
cglimit           cpuset            freezer           net_cls,net_prio  pids              unified

I have added mount to /etc/fstab as mentioned at Alpine wiki - Docker

cgroup /sys/fs/cgroup cgroup defaults 0 0

The mount helped that I am no longer getting an error that mount destination is unknow:

docker: Error response from daemon: cgroups: cannot find cgroup mount destination: unknown

(This error was discussed in other issues)

I am running Alpine Linux 3.9 in virtualized environment inside LXC on vpsAdminOS at vpsFree.

Edit:
Resolved by not using Alpine :(

[HowellBP]

Has anyone come across a fix for this yet?

[ghost]

echo "0-7" > /sys/fs/cgroup/cpuset/docker/cpuset.cpus
moby/moby#29496

[nalakafernando]

Im also facing a similar kind of an issue when i try to execute runc tests.

ok 70 runc exec [tty consolesize]
ok 71 runc create [terminal=false]
ok 72 runc run [terminal=false]
ok 73 runc run -d [terminal=false]
ok 74 update # skip
not ok 75 update rt period and runtime
# (in test file tests/integration/update.bats, line 271)
#   `[ "$status" -eq 0 ]' failed
# runc list (status=0):
# ID          PID         STATUS      BUNDLE      CREATED     OWNER
# runc list (status=0):
# ID          PID         STATUS      BUNDLE      CREATED     OWNER
# runc list (status=0):
# ID          PID         STATUS      BUNDLE      CREATED     OWNER
# runc spec (status=0):
# 
# ------------------
# ------------------/tmp/console.sock
# runc run -d --console-socket /tmp/console.sock test_update_rt (status=1):
# time="2022-11-22T05:40:51Z" level=warning msg="signal: killed" 
# time="2022-11-22T05:40:51Z" level=error msg="container_linux.go:344: starting container process caused \"process_linux.go:424: container init caused \\"process_linux.go:390: setting cgroup config for procHooks process caused \\\\"failed to write 1000 to blkio.weight: open /sys/fs/cgroup/blkio/runc-cgroups-integration-test/test-cgroup/blkio.weight: permission denied\\\\"\\"\"
# " 
# container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"process_linux.go:390: setting cgroup config for procHooks process caused \\\"failed to write 1000 to blkio.weight: open /sys/fs/cgroup/blkio/runc-cgroups-integration-test/test-cgroup/blkio.weight: permission denied\\\"\""
# runc list (status=0):
# ID          PID         STATUS      BUNDLE      CREATED     OWNER
# runc list (status=0):
# ID          PID         STATUS      BUNDLE      CREATED     OWNER
# runc list (status=0):
# ID          PID         STATUS      BUNDLE      CREATED     OWNER
ok 76 runc version
make: *** [Makefile:82: localintegration] Error 1