Can not run any docker container (fresh install after reboot)

[bialasjaroslaw]

I have installed docker using script provided at https://get.docker.com/. I was able to test docker, but after a while I have to reboot system. After that I could not start any container (even after complete removal and reinstall). I've got the same error each time, and I was unable to find any solution:

docker run -it ubuntu:16.04 /bin/sh

Handler for POST /v1.24/containers
/38a049b5e8da8a720250856ce0e09a4c5a86cf8f1679da05aa79b787869df309/start 
returned error: invalid header field value "oci runtime error: container_linux.go:247: 
starting container process caused \"process_linux.go:359: container init caused 
\\\"rootfs_linux.go:53: mounting \\\\\\\"cgroup\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/devicemapper
/mnt/8c26d2f3ae0b4d69b4375705d3c5b03386e64c9cec69dd012e972f2055acf820/rootfs\\\\\\\" 
at \\\\\\\"/sys/fs/cgroup\\\\\\\" caused \\\\\\\"no subsystem for mount\\\\\\\"\\\"\"\n" 

Linux 4.6.0-1-amd64 #1 SMP Debian 4.6.4-1 (2016-07-18) x86_64 GNU/Linux
Docker version 1.12.3, build 6b644ec

[brennovich]

I'm getting the same error here:

[brennovich@archpad ~]$ sudo docker run --rm hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c04b14da8d14: Pull complete 
Digest: sha256:0256e8a36e2070f7bf2d0b0763dbabdd67798512411de4cdcf9431a1feb60fd9
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: oci runtime error: rootfs_linux.go:53: mounting "/sys/fs/cgroup" to rootfs "/var/lib/docker/devicemapper/mnt/caba129d5183f6cd0b5c933f71aa8699ae593e87c83cadd8a943c403a12f544c/rootfs" caused "no subsystem for mount".

Look at my system details below:

docker info

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.12.3
Storage Driver: devicemapper
 Pool Name: docker-254:1-6818821-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 14.16 MB
 Data Space Total: 107.4 GB
 Data Space Available: 94.14 GB
 Metadata Space Used: 585.7 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.136 (2016-11-05)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: host bridge overlay null
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.8.6-1-ARCH
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.671 GiB
Name: archpad
ID: Z56X:XKW7:6266:64S6:JLGB:6OX4:IAXL:PVGZ:MJ26:OOXW:DGOG:7LPX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8

Linux archpad 4.8.6-1-ARCH #1 SMP PREEMPT Mon Oct 31 18:51:30 CET 2016 x86_64 GNU/Linux

Although, I'm able to run containers if I manually mount /sys/fs/cgroup:

docker run --rm -v /sys/fs/cgroup:/sys/fs/cgroup:ro hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker Hub account:
 https://hub.docker.com

For more examples and ideas, visit:
 https://docs.docker.com/engine/userguide/

[bialasjaroslaw]

If I mount /sys/fs/cgroup manually it solves the problem I had.
I have also tested this behavior on Arch Linux with kernel 4.7.4 (system is a little bit outdated) and with the same Docker version it works fine without any manual mounting.

[mithrandi]

Possibly caused by opencontainers/runc#1175 ?

[francois2metz]

Yup, opencontainers/runc#1175 is the root cause (I got his error via docker initially).

[francois2metz]

It happens on systems which have systemd 232.

[qzio]

Just want to report that I got this on machines running debian unstable.

[brennovich]

Which version of systemd are you using?

On Mon, Nov 7, 2016, 2:26 PM joel hansson notifications@github.com wrote:

Just want to report that I got this on machines running debian unstable.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#28109 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAXL9gJ7KcT0qT-n1EnD3s0MQBJ55OUGks5q71EegaJpZM4Kqord
.

[mithrandi]

systemd 232 was very recently uploaded to Debian Unstable: https://tracker.debian.org/news/812332

[wieczorek1990]

opencontainers/runc#1175 mentions such solution:

Change GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub so that it looks like this:

GRUB_CMDLINE_LINUX_DEFAULT="quiet systemd.legacy_systemd_cgroup_controller=yes"

quiet is from default Debian grub install.

Then update grub and reboot:

sudo update-grub
sudo reboot

[scoquelin]

@wieczorek1990 error message is not showing up anymore after the fix you provided but now I have some containers exiting with status 139 and nothing is showing up using docker logs mycontainer ...

Linux localhost 4.8.0-1-amd64 #1 SMP Debian 4.8.5-1 (2016-10-28) x86_64 GNU/Linux
Docker version 1.12.3, build 6b644ec
ii  systemd                                       232-2                                amd64        system and service manager

Update : switching back to kernel 4.4 allowed me to workaround the "Exited (139)" issue, no idea why though...

Linux localhost 4.4.0-1-amd64 #1 SMP Debian 4.4.6-1 (2016-03-17) x86_64 GNU/Linux

[sej7278]

Getting this on Debian Sid, assumed it was a 4.8 kernel thing, but it seems its systemd 232-2

changing grub params is not a fix, its a workaround, has anyone reported to debian bugs?

edit: seems to be fixed in systemd 232-3