systemd cgroup scope somehow gets left behind not allowing containers to start

[ibuildthecloud]

I ran into a situation in which I got:

[error] container.go:468 Error running container: Unit docker-d3ca1668bff4e74d22113886ba1433ecae920ece02c76ce1a9344409b68903bc.scope already exists.

It seems the scope did not get cleaned up by systemd. The scope file was still present in /run/systemd/system/... but systemd-cgls did not show it. In the end I had to delete the container because I couldn't start it.

The way I got into this situation is I had a container that would die on startup and I had a script that was auto-restarting it. So it was starting this container every 5 seconds and it would die every 5 seconds. Eventually it must have hit some race condition, and then it just started to always fail to start in docker because of this cgroup issue.

This is systemd 212 and docker 1.0.1 running CoreOS 367.1.0

[ibuildthecloud]

Upon further inspection I found the following cgroups still exist.

/sys/fs/cgroup/blkio/system.slice/docker-d3ca1668bff4e74d22113886ba1433ecae920ece02c76ce1a9344409b68903bc.scope
/sys/fs/cgroup/memory/system.slice/docker-d3ca1668bff4e74d22113886ba1433ecae920ece02c76ce1a9344409b68903bc.scope
/sys/fs/cgroup/cpu,cpuacct/system.slice/docker-d3ca1668bff4e74d22113886ba1433ecae920ece02c76ce1a9344409b68903bc.scope
/sys/fs/cgroup/systemd/system.slice/docker-d3ca1668bff4e74d22113886ba1433ecae920ece02c76ce1a9344409b68903bc.scope

For all cgroups, cgroup.procs is empty. I suspect this is a systemd issue.

[crosbymichael]

Yes, this is a systemd cgroup issue

[crosbymichael]

Look around this part for a fix

https://github.com/docker/libcontainer/blob/master/cgroups/systemd/apply_systemd.go#L323

[ibuildthecloud]

Looking into this, the docker systemd cgroup code is cleaning up properly. It's deleting all cgroups that it created, that systemd doesn't support. The issue seems to be that the cgroups that systemd manages are supposed to be deleted automatically and they are not. So, still seems like a specific systemd issue.

Since systemd is out there in the wild and this issue exists, it seems as though docker should use the cgroups if they already exist, and not fail. This does mean that systemd will be orphaning cgroups, but that will have to be address separately.

I will try to look into what the fix may be for this

[joschi127]

I have this issue when using manjaro linux as host system. When stopping a container, the cgroup cleanup does not seem to work, the following .scope folders still exist after stopping the container:

/sys/fs/cgroup/blkio/system.slice/docker-663a7122c20d75e6c34b94bac7dafa144ac06e6ce1d9c09c71d6dbb4fe94be4c.scope
/sys/fs/cgroup/memory/system.slice/docker-663a7122c20d75e6c34b94bac7dafa144ac06e6ce1d9c09c71d6dbb4fe94be4c.scope
/sys/fs/cgroup/cpu,cpuacct/system.slice/docker-663a7122c20d75e6c34b94bac7dafa144ac06e6ce1d9c09c71d6dbb4fe94be4c.scope
/sys/fs/cgroup/systemd/system.slice/docker-663a7122c20d75e6c34b94bac7dafa144ac06e6ce1d9c09c71d6dbb4fe94be4c.scope

As far as I can see this happens every time for me, not only from time to time.

[djmaze]

Same problem on Arch Linux. Is there a way to fix this state, apart from rebooting?

[joschi127]

@djmaze For me the only real fix is rebooting. As a workaround you can save the container to an image like this:

docker commit containername imagename

Then you can remove the old container and run a new container from the saved image:

docker rm containername
docker run --name containername [...] -t imagename [...]

[mastercactapus]

workaround in the meantime:

#systemctl stop docker-FULL_CONTAINER_ID.scope
systemctl stop docker-663a7122c20d75e6c34b94bac7dafa144ac06e6ce1d9c09c71d6dbb4fe94be4c.scope

[gegere]

@mastercactapus great, thank you for this post it solved my issue, for now!

[guilhermebr]

Hi, this happened many times.

core@coreos03 ~ $ docker start containername
Error response from daemon: Cannot start container containername: Unit docker-93408c6f3c1bb4a944b04d39bb06509aa1b5159f715d3ce157b34a4874cb5109.scope already exists.

$ docker --version
Docker version 1.1.2, build d84a070

[gegere]

@guilhermebr ok... run the following command.

$ systemctl stop docker-93408c6f3c1bb4a944b04d39bb06509aa1b5159f715d3ce157b34a4874cb5109.scope