Containers fail to start when using --memory in docker 17.09.0-ce

[TimWolla]

Note: This already happened with 17.06, but was fixed with 17.07, it reappears with 17.09, see #34616

Description

Containers fail to start with --memory set.

Steps to reproduce the issue:

1. docker run -it --rm --memory 1G debian bash

Describe the results you received:

docker: Error response from daemon: oci runtime error: container_linux.go:265: starting container process caused "process_linux.go:284: applying cgroup configuration for process caused \"No such device or address\"".

Describe the results you expected:
A bash running inside a Debian container.

Additional information you deem important (e.g. issue happens only occasionally):
Happens even after a reboot of the complete machine.

Output of docker version:

Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:09 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:48 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 24
 Running: 24
 Paused: 0
 Stopped: 0
Images: 138
Server Version: 17.09.0-ce
Storage Driver: zfs
 Zpool: rpool
 Zpool Health: ONLINE
 Parent Dataset: rpool/docker
 Space Used By Parent: 3726835712
 Space Available: 770689671168
 Parent Quota: no
 Compression: on
Logging Driver: json-file
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.9.0-4-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.57GiB
Name: *snip*
ID: *snip*
Docker Root Dir: /docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: true

Additional environment details (AWS, VirtualBox, physical, etc.):
Dedicated bare metal host running Debian Stretch.

[thaJeztah]

ping @tiborvass

[tiny1990]

it's works in cgroupfs mode , you can change to test, but i do not know why

[huzhengchuan]

I meet the same question in centos7.2.

[root@k8s1-8-master ~]# docker version
Client:
Version: 17.09.0-ce
API version: 1.32
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:41:23 2017
OS/Arch: linux/amd64

Server:
Version: 17.09.0-ce
API version: 1.32 (minimum version 1.12)
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:42:49 2017
OS/Arch: linux/amd64
Experimental: false

docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc --runtime-args --systemd-cgroup=true
use systemd as group manage

[huzhengchuan]

upgrade to docker 17.10.0-ce-rc2 work well.
But I didnot know the issue in moby. who know, can ref it.

[gyliu513]

Yes, seems using Cgroup Driver: cgroupfs works fine, can anyone explain why does this works?

[thaJeztah]

The systemd cgroups implementation is known to be troublesome, and frequently breaks between systemd versions. For that reason, the default is to use cgroupfs (see #17704, #16238, #16256, and various other issues). Unless you have a very specific reason to require systemd cgroups, I'd recommend using the cgroupfs option.

@TimWolla does switching to cgroupfs resolve your issue as well?

[TimWolla]

@TimWolla does switching to `cgroupfs` resolve your issue as well?

Yes, it does. Am 25.10.2017 um 13:44 schrieb Sebastiaan van Stijn:

…

The systemd cgroups implementation is known to be troublesome, and frequently breaks between systemd versions. For that reason, the default is to use `cgroupfs` (see #17704, #16238, #16256, and various other issues). Unless you have a very specific reason to require systemd cgroups, I'd recommend using the cgroupfs option. @TimWolla does switching to `cgroupfs` resolve your issue as well?

[tiny1990]

@thaJeztah what's difference between cgroupfs and systemd

[tossmilestone]

@thaJeztah Sorry to bother, I want to make sure that if this issue is fixed on 17.12, because we may upgrade to this version if it was fixed.

[darshan-karia]

@tossmilestone I tried 17.12 in past and it did not work. The version which worked for me was 17.10

[thaJeztah]

@darshan-karia what problem did you run into? Testing this on a 17.12 machine:

$ docker run -it --rm --memory 1G alpine echo hello world
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
hello world

That warning indicates that the machine doesn't support swap accounting. The container started, but limiting swap is not applied; after enabling (see Your kernel does not support cgroup swap limit capabilities);

$ docker run -it --rm --memory 1G alpine echo hello world
hello world

[darshan-karia]

@thaJeztah The issue I was encountering was when running a systemd container with memory option.
But, I tried running systemd container after your comment, and that seems to be working.

I must have confused the versions, as I was trying many for my experiment, sorry about confusion.

[thaJeztah]

The fix for this should be in Docker 17.12.1, and a test is added in #36172. I'll go ahead and close this 👍