System error on run: stat /sys/fs/cgroup/systemd: no such file or directory

[gdm85]

The system error:

# docker run busybox-nonroot ls
Timestamp: 2015-12-27 16:31:52.402344267 +0100 CET
Code: System error

Message: stat /sys/fs/cgroup/systemd: no such file or directory

Frames:

---
0: setupRootfs
Package: github.com/opencontainers/runc/libcontainer
File: rootfs_linux.go@40

---
1: Init
Package: github.com/opencontainers/runc/libcontainer.(*linuxStandardInit)
File: standard_init_linux.go@57

---
2: StartInitialization
Package: github.com/opencontainers/runc/libcontainer.(*LinuxFactory)
File: factory_linux.go@242

---
3: initializer
Package: github.com/docker/docker/daemon/execdriver/native
File: init.go@35

---
4: Init
Package: github.com/docker/docker/pkg/reexec
File: reexec.go@26

---
5: main
Package: main
File: docker.go@18

---
6: main
Package: runtime
File: proc.go@63

---
7: goexit
Package: runtime
File: asm_amd64.s@2232
Error response from daemon: Cannot start container 8aac8437dfb50042fb6db7ff97627b443ddb92e4ec92fa0fbd17e6ed751b9a03: [8] System error: stat /sys/fs/cgroup/systemd: no such file or directory

Some environment details:

# docker version
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64
# docker info
Containers: 6
Images: 11
Server Version: 1.9.1
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.19.0-30-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 2
Total Memory: 5.829 GiB
Name: something
WARNING: No swap limit support

This is how cgroup is setup here:

# mount|grep cgroup
cgroup on /sys/fs/cgroup type tmpfs (rw)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,relatime,cpuset,release_agent=/run/cgmanager/agents/cgm-release-agent.cpuset,clone_children)
cgroup on /sys/fs/cgroup/cpu type cgroup (rw,relatime,cpu,release_agent=/run/cgmanager/agents/cgm-release-agent.cpu)
cgroup on /sys/fs/cgroup/cpuacct type cgroup (rw,relatime,cpuacct,release_agent=/run/cgmanager/agents/cgm-release-agent.cpuacct)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,relatime,memory,release_agent=/run/cgmanager/agents/cgm-release-agent.memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices,release_agent=/run/cgmanager/agents/cgm-release-agent.devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,relatime,freezer,release_agent=/run/cgmanager/agents/cgm-release-agent.freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,relatime,net_cls,release_agent=/run/cgmanager/agents/cgm-release-agent.net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,relatime,blkio,release_agent=/run/cgmanager/agents/cgm-release-agent.blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,relatime,perf_event,release_agent=/run/cgmanager/agents/cgm-release-agent.perf_event)
cgroup on /sys/fs/cgroup/net_prio type cgroup (rw,relatime,net_prio,release_agent=/run/cgmanager/agents/cgm-release-agent.net_prio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,relatime,hugetlb,release_agent=/run/cgmanager/agents/cgm-release-agent.hugetlb)
none on /sys/fs/cgroup type tmpfs (rw,uid=0,gid=0,mode=0755,size=1024)
systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd)
cgroup on /sys/fs/cgroup type tmpfs (rw,uid=0,gid=0,mode=0755)
cgroup on /sys/fs/cgroup type tmpfs (rw,uid=0,gid=0,mode=0755)

[GordonTheTurtle]

Hi!

Please read this important information about creating issues.

If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.

This is an automated, informational response.

Thank you.

For more information about reporting issues, see https://github.com/docker/docker/blob/master/CONTRIBUTING.md#reporting-other-issues

---

BUG REPORT INFORMATION

Use the commands below to provide key information from your environment:

docker version:
docker info:
uname -a:

Provide additional environment details (AWS, VirtualBox, physical, etc.):

List the steps to reproduce the issue:
1.
2.
3.

Describe the results you received:

Describe the results you expected:

Provide additional info you think is important:

----------END REPORT ---------

#ENEEDMOREINFO

[gdm85]

This issue has same root cause as rkt/rkt#1076 and I can workaround it by running the daemon in foreground in a tmux/screen:

         mkdir -p /sys/fs/cgroup/systemd
         mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd
         mkdir -p /sys/fs/cgroup/systemd/user
         echo $$ > /sys/fs/cgroup/systemd/user/cgroup.procs 
         docker daemon

Obviously the cgroupfs setup stage in the init script for the daemon is not taking care of non-systemd systems properly.

[gdm85]

@GordonTheTurtle you don't need uname -a when you have the docker version output already.

[thaJeztah]

@gdm85 created a PR for that; docker-archive/leeroy#41. Reason we kept it still around is that we sometimes receive reports from people running an older version of docker, which didn't yet include that information. It's probably safe by now to assume the majority of people have a version of docker that does.

[gdm85]

@thaJeztah oh, thanks - now I understand why it was there :)

As per this issue, I think the change should go in the LSB init script, I can attempt a PR if it's acknowledged as a legit bug

[thaJeztah]

@gdm85 opening a PR won't hurt I think (at the chance it gets rejected 😇); It may be easier for maintainers to discuss the code change when looked at

[gdm85]

@thaJeztah sure, I am not conservative about the time needed to arrange a PR but rather: am I on a real issue or is this a blunder on my side? 😄

I think I nailed it with that PR; I found also a missing mountpoint function definition in the upstart script (upstart scripts cannot "see" LSB functions, AFAIK)

[thaJeztah]

@gdm85 lol, I'll happy defer that to @LK4D4 or @anusha-ragunathan, I'm trying to keep up with the cgroups issues we encountered, but failing at times 😇

[gdm85]

@thaJeztah ok thanks. I also Cc'ed in the PR @tianon and @jfrazelle as they are listed in REVIEWERS there.

[thaJeztah]

Yup, I saw, thanks!

[gdm85]

This is not happening anymore, tested with Docker 17/18; however, the bug in #36169 happens for any arbitrary directory created under /sys/fs/cgroup/.

I closed that issue, but perhaps it would be best for Docker to ignore such unknown directories in there.