Can not run any docker container (fresh install after reboot)

[bialasjaroslaw]

I have installed docker using script provided at https://get.docker.com/. I was able to test docker, but after a while I have to reboot system. After that I could not start any container (even after complete removal and reinstall). I've got the same error each time, and I was unable to find any solution:

docker run -it ubuntu:16.04 /bin/sh

Handler for POST /v1.24/containers
/38a049b5e8da8a720250856ce0e09a4c5a86cf8f1679da05aa79b787869df309/start 
returned error: invalid header field value "oci runtime error: container_linux.go:247: 
starting container process caused \"process_linux.go:359: container init caused 
\\\"rootfs_linux.go:53: mounting \\\\\\\"cgroup\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/devicemapper
/mnt/8c26d2f3ae0b4d69b4375705d3c5b03386e64c9cec69dd012e972f2055acf820/rootfs\\\\\\\" 
at \\\\\\\"/sys/fs/cgroup\\\\\\\" caused \\\\\\\"no subsystem for mount\\\\\\\"\\\"\"\n" 

Linux 4.6.0-1-amd64 #1 SMP Debian 4.6.4-1 (2016-07-18) x86_64 GNU/Linux
Docker version 1.12.3, build 6b644ec

[brennovich]

I'm getting the same error here:

[brennovich@archpad ~]$ sudo docker run --rm hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c04b14da8d14: Pull complete 
Digest: sha256:0256e8a36e2070f7bf2d0b0763dbabdd67798512411de4cdcf9431a1feb60fd9
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: oci runtime error: rootfs_linux.go:53: mounting "/sys/fs/cgroup" to rootfs "/var/lib/docker/devicemapper/mnt/caba129d5183f6cd0b5c933f71aa8699ae593e87c83cadd8a943c403a12f544c/rootfs" caused "no subsystem for mount".

Look at my system details below:

docker info

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.12.3
Storage Driver: devicemapper
 Pool Name: docker-254:1-6818821-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 14.16 MB
 Data Space Total: 107.4 GB
 Data Space Available: 94.14 GB
 Metadata Space Used: 585.7 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.136 (2016-11-05)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host bridge overlay null
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.8.6-1-ARCH
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.671 GiB
Name: archpad
ID: Z56X:XKW7:6266:64S6:JLGB:6OX4:IAXL:PVGZ:MJ26:OOXW:DGOG:7LPX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8

Linux archpad 4.8.6-1-ARCH #1 SMP PREEMPT Mon Oct 31 18:51:30 CET 2016 x86_64 GNU/Linux

Although, I'm able to run containers if I manually mount /sys/fs/cgroup:

docker run --rm -v /sys/fs/cgroup:/sys/fs/cgroup:ro hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker Hub account:
 https://hub.docker.com

For more examples and ideas, visit:
 https://docs.docker.com/engine/userguide/

[bialasjaroslaw]

If I mount /sys/fs/cgroup manually it solves the problem I had.
I have also tested this behavior on Arch Linux with kernel 4.7.4 (system is a little bit outdated) and with the same Docker version it works fine without any manual mounting.

[mithrandi]

Possibly caused by opencontainers/runc#1175 ?

[francois2metz]

Yup, opencontainers/runc#1175 is the root cause (I got his error via docker initially).

[francois2metz]

It happens on systems which have systemd 232.

[qzio]

Just want to report that I got this on machines running debian unstable.

[brennovich]

Which version of systemd are you using?

On Mon, Nov 7, 2016, 2:26 PM joel hansson notifications@github.com wrote:

Just want to report that I got this on machines running debian unstable.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#28109 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAXL9gJ7KcT0qT-n1EnD3s0MQBJ55OUGks5q71EegaJpZM4Kqord
.

[mithrandi]

systemd 232 was very recently uploaded to Debian Unstable: https://tracker.debian.org/news/812332

[wieczorek1990]

opencontainers/runc#1175 mentions such solution:

Change GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub so that it looks like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet systemd.legacy_systemd_cgroup_controller=yes"

quiet is from default Debian grub install.

Then update grub and reboot:

sudo update-grub
sudo reboot

[scoquelin]

@wieczorek1990 error message is not showing up anymore after the fix you provided but now I have some containers exiting with status 139 and nothing is showing up using docker logs mycontainer ...

Linux localhost 4.8.0-1-amd64 #1 SMP Debian 4.8.5-1 (2016-10-28) x86_64 GNU/Linux
Docker version 1.12.3, build 6b644ec
ii  systemd                                       232-2                                amd64        system and service manager

Update : switching back to kernel 4.4 allowed me to workaround the "Exited (139)" issue, no idea why though...

Linux localhost 4.4.0-1-amd64 #1 SMP Debian 4.4.6-1 (2016-03-17) x86_64 GNU/Linux

[sej7278]

Getting this on Debian Sid, assumed it was a 4.8 kernel thing, but it seems its systemd 232-2

changing grub params is not a fix, its a workaround, has anyone reported to debian bugs?

edit: seems to be fixed in systemd 232-3

[bialasjaroslaw]

edit: seems to be fixed in systemd 232-3

Confirmed. It is working with systemd 232-3 like before previous systemd update

[martinpitt]

systemd/systemd#4628 reverted this for now, but you can still boot with systemd.legacy_systemd_cgroup_controller=0 to get the unified behaviour; that is useful for developers to make runc/docker/lxc/etc. work with the unified hierarchy eventually.

[jamiethermo]

I'm getting this on centos 7.2, systemd 219, docker 1.12.2.

Would this be related to the -H fd:// changes in 1.12 or is that a red herring?

[lrepolho]

More people seems to be having the issue here https://forums.docker.com/t/oci-runtime-error-container-init-caused-not-a-directory/23451/8

I could run my container by setting the problematic volume as READ ONLY to share a single file.
From:
volumes:

• "./config/php-xdebug.ini:/usr/local/etc/php/conf.d/php-xdebug.ini"

To:
volumes:

• "./config/php-xdebug.ini:/usr/local/etc/php/conf.d/php-xdebug.ini:ro"

[thaJeztah]

@leandrocr that's not related; using a relative path (./) or bind mounting a file to a remote host won't work, therefore the docker daemon creates an empty directory with the name you provided and mounts it in the container (which fails because there's a file on that location).

Please don't put every "daemon failed to start" or "container failed to start" in this thread, as it makes it quickly into a kitchen sink of unrelated issues

[Majkl578]

This re-appears with Systemd 233:

$ uname -v
#1 SMP Debian 4.9.13-1 (2017-02-27)

$ aptitude versions docker-engine | grep ^i
i  17.03.0~ce-0~debian-stretch debian-stretch 500

$ aptitude versions systemd | grep ^i
i  233-3 experimental 850

$ docker run --rm -it debian:jessie
docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"rootfs_linux.go:54: mounting \\\"cgroup\\\" to rootfs \\\"/var/lib/docker/devicemapper/mnt/675c1e80fb22bb74c28a52843dec71d82615f4a541a5028d1f171d85889d8bba/rootfs\\\" at \\\"/sys/fs/cgroup\\\" caused \\\"no subsystem for mount\\\"\"".

[thaJeztah]

ping @crosbymichael @mlaventure ^^

[martinpitt]

@Majkl578 : This got fixed by opencontainers/runc#1266, there just hasn't been a new runc release since then. I backported the fix to to Ubuntu already, but it is not yet fixed in Debian, see the bug report

[erickvermot]

I had the same issue using Arch with systemd 232-3.

docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:258: applying cgroup configuration for process caused "mkdir /sys/fs/cgroup/cpuset: no such file or directory"".

upgrading the system got me systemd 232-8, and docker seem to be working again.

[blacktop]

How would I fix this for Docker for Mac?

[cpuguy83]

@blacktop Do you you fix what exactly? I'd recommend opening a new issue on github.com/docker/for-mac and give a detailed explanation of what you are seeing.

Thanks!

[thaJeztah]

Locking the conversation on this issue, because the original issue reported here was resolved in #31667. If you're still seeing this on an up-to-date docker, please open a new issue with details