Failed to mount API filesystems, freezing.

[sirkkalap]

I get freeze at startup on OS X 10.11.4. Any idea what went wrong?

~ ❯❯❯ docker run -i -t --stop-signal=SIGRTMIN+3 --tmpfs /run -v /sys/fs/cgroup:/sys/fs/cgroup:ro solita/centos-systemd:latest
[!!!!!!] Failed to mount API filesystems, freezing.

[noidi]

I managed to reproduce issue on Windows with the latest Docker Toolbox. Will investigate.

[nschoe]

I confirmed it too, on Ubuntu 14.04.

Sometimes after running the docker run command above, I get this error:

Failed to mount tmpfs at /run: Permission denied
Failed to mount tmpfs at /run/lock: Permission denied
[!!!!!!] Failed to mount API filesystems, freezing.
Freezing execution.

And sometimes, it starts alright. This is particularly weird because I don't change anything, and so I don't understand why I have the permission denied error.

[noidi]

@nschoe That's a different issue, caused by a bug in Docker that will be fixed in 1.12.0

[nschoe]

@noidi thanks for the link.
By the way, inspired from here, I modified the Dockerfile to add VOLUME ["/sys/fs/cgroup", "/tmp", "/run", "/run/lock"] and even after several tries, I havent' add neither the permission error nor the "Failed to mount API filesystems, freezing" problem.

[noidi]

Thanks, good to know!

[noidi]

The error can also be reproduced on Ubuntu 16.04. It's possible that this error happens when the Docker host runs Linux 4.

[noidi]

On Ubuntu 16.04 the container starts up correctly with the addition of the following flags:

--cap-add SYS_ADMIN --security-opt seccomp=unconfined

I'm guessing it could also run with a much stricter security profile.

Unfortunately this is not enough to start the container in boot2docker. I'll keep investigating.

[noidi]

I think I've managed to come up with a fix. I've tested it with Docker Machine + boot2docker on Linux, but it's still untested on OS X or Windows.

To test the fix, follow the updated README: First run the new script systemd-container-host-config, and then pass the new option --security-opt seccomp=unconfined to docker create or docker run.

[noidi]

Fix verified with Docker Toolbox on Windows 7.

[nschoe]

A bit more info here:

I've switched to Ubuntu 16.04, and running original docker run command I get errors:

Failed to determine whether /sys is a mount point: Operation not permitted
Failed to determine whether /proc is a mount point: Operation not permitted
Failed to determine whether /dev is a mount point: Operation not permitted
Failed to determine whether /dev/shm is a mount point: Operation not permitted
Failed to determine whether /run is a mount point: Operation not permitted
Failed to determine whether /run/lock is a mount point: Operation not permitted
Failed to determine whether /sys/fs/cgroup is a mount point: Operation not permitted
Failed to determine whether /sys/fs/cgroup/systemd is a mount point: Operation not permitted
[!!!!!!] Failed to mount API filesystems, freezing.
Freezing execution.

I then added --security-opt seccomp=unconfined and it cleared the first messages, only to leave

Failed to mount cgroup at /sys/fs/cgroup/systemd: Permission denied
[!!!!!!] Failed to mount API filesystems, freezing.
Freezing execution.

I tried adding --cap-add SYS_ADMIN to the docker run command but it did not change anything: I still have the last error message.

I don't understand why I've got a permission denied on /sys/fs/cgroup/systemd and the failed to mount API filesystem

[nschoe]

@noidi I executed the shell script in the readme, it said that the cgroup was already mounted.
Then I still have the error. Smth specific I'm missing?

[noidi]

@nschoe Did you use all the options described in the README? Here's a full command line that works for me (apart from occasional problems due to the tmpfs bug in Docker 1.11) for me both directly on a Linux host as well as on a boot2docker machine in VirtualBox:

docker run --name systemd -it --rm --security-opt seccomp=unconfined --stop-signal=SIGRTMIN+3 --tmpfs /run --tmpfs /run/lock -v /sys/fs/cgroup:/sys/fs/cgroup:ro solita/ubuntu-systemd:16.04

[nschoe]

@noidi actually I have just confirmed that your commands work. I can successfully create and start a container form the solita/ubuntu-systemd image with docker run.

However, and this is very strange to me, I can't have it work with docker-compose files.

See here.

I am pretty sure this is not a bug of yours, but I still can't explain it... If you have any idea?

Thanks for your reactivity btw!

[nschoe]

Actually maybe it is?
I have been using docker-compose fine with other images, but I can't make it work with image solita/ubuntu-systemd. Is there something specific I'm missing?

[noidi]

@nschoe I suspect it's the tmpfs bug biting here.

I made the following changes to your docker-compose.yml:

• Changed seccomp:unconfined to seccomp=unconfined
• Added tty: yes, to make systemd print what it's doing.
• Added command: bash -c 'mount | grep /run; exec /sbin/init', to print the tmpfs mount order before starting systemd.

This typically gives me the following output:

timomi@melanthos /tmp> docker-compose up
Starting tmp_test_1
Attaching to tmp_test_1
test_1  | tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
test_1  | tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
test_1  | Failed to mount tmpfs at /run/lock: Permission denied
test_1  | [!!!!!!] Failed to mount API filesystems, freezing.
test_1  | Freezing execution.

As you can see, /run is mounted after /run/lock, thus shadowing it, which causes the failure. But very, very rarely I managed to get a run where the order was correct and systemd started correctly:

timomi@melanthos /tmp> docker-compose up
Starting tmp_test_1
Attaching to tmp_test_1
test_1  | tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
test_1  | tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
test_1  | systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN)
test_1  | Detected virtualization docker.
test_1  | Detected architecture x86-64.
test_1  | 
test_1  | Welcome to Ubuntu 16.04 LTS!
test_1  | 
test_1  | Set hostname to <aa27e213fbe5>.
test_1  | Failed to install release agent, ignoring: File exists
test_1  | [  OK  ] Created slice System Slice.
test_1  | [  OK  ] Reached target Slices.
test_1  | [  OK  ] Listening on Journal Socket (/dev/log).
test_1  | [  OK  ] Reached target Swap.
test_1  | [  OK  ] Listening on Journal Socket.
test_1  |          Starting Journal Service...
test_1  | [  OK  ] Reached target Sockets.
test_1  | [  OK  ] Reached target Local File Systems.
test_1  | [  OK  ] Reached target Paths.
test_1  |          Starting Create Volatile Files and Directories...
test_1  | [  OK  ] Started Journal Service.
test_1  | [  OK  ] Started Create Volatile Files and Directories.
test_1  | [  OK  ] Reached target System Initialization.
test_1  | [  OK  ] Started Daily Cleanup of Temporary Directories.
test_1  | [  OK  ] Reached target Timers.
test_1  | [  OK  ] Reached target Basic System.
test_1  |          Starting LSB: Set the CPU Frequency Scaling governor to "ondemand"...
test_1  |          Starting /etc/rc.local Compatibility...
test_1  |          Starting Permit User Sessions...
test_1  | [  OK  ] Started /etc/rc.local Compatibility.
test_1  | [  OK  ] Started Permit User Sessions.
test_1  | [  OK  ] Started LSB: Set the CPU Frequency Scaling governor to "ondemand".
test_1  | [  OK  ] Reached target Multi-User System.

[noidi]

If you don't care so much about having Ubuntu, and any Linux distribution with systemd will do, you could use solita/centos-systemd:7 instead of solita/ubuntu. It only needs one tmpfs, /run, so it does not have this problem. Or you could try compiling Docker from Git, as the tmpfs bug is fixed on master.

[sirkkalap]

Connecting to Docker Beta xhyve virtual machine (host) is actually possible. Thus it seems to be possible to modify the cgroups (maybe).

Here is the topic explaining how to dig inside the Docker Beta virtual machine: https://forums.docker.com/t/how-to-make-changes-to-xhyve-host/11820

[sirkkalap]

I have now confirmed that creating systemd cgroup was possible in Docker Beta xhyve hypervisor provided Docker host. Here is the pretty screenshot: https://www.dropbox.com/s/v9ijh6y1uxo0tiw/Screenshot%202016-05-11%2023.13.01.png?dl=0

The command to enter the xhyve docker host in Docker Beta was:

~ ❯❯❯ screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

The commands I used inside the docker host box were:

docker:~# mkdir -p /sys/fs/cgroup/systemd
docker:~# mount -t cgroup cgroup -o none,name=systemd /sys/fs/cgroup/systemd

Finally I ran the container as described in the README and ran another bash inside the container using:

~ ❯❯❯ docker ps -a
CONTAINER ID        IMAGE                   COMMAND             CREATED             STATUS              PORTS               NAMES
6cf6cb029583        solita/ubuntu-systemd   "/sbin/init"        27 seconds ago      Up 26 seconds                           furious_shirley
~ ❯❯❯ docker exec -i -t 6c bash
root@6cf6cb029583:/# ps axufww
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root        33  0.0  0.0  18212  3216 ?        Ss   20:06   0:00 bash
root        44  0.0  0.0  34424  2892 ?        R+   20:06   0:00  \_ ps axufww
root         1  0.0  0.1  36824  4652 ?        Ss   20:04   0:00 /sbin/init
root        18  0.0  0.0  27904  1352 ?        Ss   20:04   0:00 /lib/systemd/systemd-journald
root@6cf6cb029583:/#

[keithyau]

Same issue on latest centos7 after following this guide

https://forums.docker.com/t/systemctl-status-is-not-working-in-my-docker-container/9075/4

[deiga]

@sirkkalap This works, but can it be scripted somehow?

[noidi]

@deiga I think we might be able to support Docker for Mac by doing the setup in a privileged container instead of running commands in a host shell. Unfortunately I don't have a Mac to test on. If I manage to get the setup change working on Linux, would you be willing to try it out with Docker for Mac?

[deiga]

@noidi Yeah, sure

[thaJeztah]

We've been looking into this issue (because previously disabling seccomp was not required), and may have found the cause of this; there's a pull request opened that changes the default seccomp profile and hopefully no longer requires completely disabling seccomp; moby/moby#25567

[cleverlzc]

I am also facing this problem.
Luckily, now I have solved it by add --privileged option， when execute the docker run xxx.

[thaJeztah]

This issue should be fixed now. Please, don't use --privileged; using --privileged disables all protection that containers provide. Using --cap-add SYS_ADMIN is sufficient in current versions of Docker;

docker run -it \
  --tmpfs /run \
  -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
  --cap-add SYS_ADMIN \
  solita/centos-systemd:latest

systemd 219 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
Detected virtualization docker.
Detected architecture x86-64.

Welcome to CentOS Linux 7 (Core)!

Set hostname to <6e964a8388ed>.
Initializing machine ID from random generator.
[  OK  ] Created slice Root Slice.
[  OK  ] Created slice System Slice.
[  OK  ] Reached target Slices.
[  OK  ] Reached target Swap.
[  OK  ] Reached target Paths.
[  OK  ] Listening on Journal Socket.
         Starting Journal Service...
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
[  OK  ] Started Journal Service.
[  OK  ] Started Create Volatile Files and Directories.
[  OK  ] Reached target System Initialization.
[  OK  ] Reached target Basic System.
         Starting Permit User Sessions...
[  OK  ] Reached target Timers.
         Starting Cleanup of Temporary Directories...
[  OK  ] Started Permit User Sessions.
[  OK  ] Started Cleanup of Temporary Directories.
[  OK  ] Reached target Multi-User System.

[cleverlzc]

I have tried to use --cap-add SYS_ADMIN，and all things seems to be ok. I'll use --cap-add SYS_ADMIN instead of --privileged in ther later days. By the way, please follow the advice of @thaJeztah .Thank you, thaJeztah !

[mani1soni]

when i trying to use --cap-add SYS_ADMIN, then i got following error--
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused "rootfs_linux.go:58: mounting \"/sys/fs/cgroup\" to rootfs \"/var/lib/docker/overlay2/1bda6365428f268eca0fdc1fb151fa47c3caaccc3578a9e140e2346bb9fd086d/merged\" at \"/var/lib/docker/overlay2/1bda6365428f268eca0fdc1fb151fa47c3caaccc3578a9e140e2346bb9fd086d/merged/sys/fs/cgroups\" caused \"mkdir /var/lib/docker/overlay2/1bda6365428f268eca0fdc1fb151fa47c3caaccc3578a9e140e2346bb9fd086d/merged/sys/fs/cgroups: read-only file system\""": unknown.