New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Containers fail to start when using --memory in docker 17.09.0-ce #35123

Closed
TimWolla opened this Issue Oct 7, 2017 · 13 comments

Comments

Projects
None yet
8 participants
@TimWolla
Contributor

TimWolla commented Oct 7, 2017

Note: This already happened with 17.06, but was fixed with 17.07, it reappears with 17.09, see #34616

Description

Containers fail to start with --memory set.

Steps to reproduce the issue:

  1. docker run -it --rm --memory 1G debian bash

Describe the results you received:

docker: Error response from daemon: oci runtime error: container_linux.go:265: starting container process caused "process_linux.go:284: applying cgroup configuration for process caused \"No such device or address\"".

Describe the results you expected:
A bash running inside a Debian container.

Additional information you deem important (e.g. issue happens only occasionally):
Happens even after a reboot of the complete machine.

Output of docker version:

Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:09 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:48 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 24
 Running: 24
 Paused: 0
 Stopped: 0
Images: 138
Server Version: 17.09.0-ce
Storage Driver: zfs
 Zpool: rpool
 Zpool Health: ONLINE
 Parent Dataset: rpool/docker
 Space Used By Parent: 3726835712
 Space Available: 770689671168
 Parent Quota: no
 Compression: on
Logging Driver: json-file
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.9.0-4-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.57GiB
Name: *snip*
ID: *snip*
Docker Root Dir: /docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: true

Additional environment details (AWS, VirtualBox, physical, etc.):
Dedicated bare metal host running Debian Stretch.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah
Member

thaJeztah commented Oct 7, 2017

ping @tiborvass

@tiny1990

This comment has been minimized.

Show comment
Hide comment
@tiny1990

tiny1990 Oct 13, 2017

it's works in cgroupfs mode , you can change to test, but i do not know why

tiny1990 commented Oct 13, 2017

it's works in cgroupfs mode , you can change to test, but i do not know why

@huzhengchuan

This comment has been minimized.

Show comment
Hide comment
@huzhengchuan

huzhengchuan Oct 16, 2017

I meet the same question in centos7.2.

[root@k8s1-8-master ~]# docker version
Client:
Version: 17.09.0-ce
API version: 1.32
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:41:23 2017
OS/Arch: linux/amd64

Server:
Version: 17.09.0-ce
API version: 1.32 (minimum version 1.12)
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:42:49 2017
OS/Arch: linux/amd64
Experimental: false

docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc --runtime-args --systemd-cgroup=true
use systemd as group manage

huzhengchuan commented Oct 16, 2017

I meet the same question in centos7.2.

[root@k8s1-8-master ~]# docker version
Client:
Version: 17.09.0-ce
API version: 1.32
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:41:23 2017
OS/Arch: linux/amd64

Server:
Version: 17.09.0-ce
API version: 1.32 (minimum version 1.12)
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:42:49 2017
OS/Arch: linux/amd64
Experimental: false

docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc --runtime-args --systemd-cgroup=true
use systemd as group manage

@huzhengchuan

This comment has been minimized.

Show comment
Hide comment
@huzhengchuan

huzhengchuan Oct 16, 2017

upgrade to docker 17.10.0-ce-rc2 work well.
But I didnot know the issue in moby. who know, can ref it.

huzhengchuan commented Oct 16, 2017

upgrade to docker 17.10.0-ce-rc2 work well.
But I didnot know the issue in moby. who know, can ref it.

@gyliu513

This comment has been minimized.

Show comment
Hide comment
@gyliu513

gyliu513 Oct 24, 2017

Yes, seems using Cgroup Driver: cgroupfs works fine, can anyone explain why does this works?

gyliu513 commented Oct 24, 2017

Yes, seems using Cgroup Driver: cgroupfs works fine, can anyone explain why does this works?

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Oct 25, 2017

Member

The systemd cgroups implementation is known to be troublesome, and frequently breaks between systemd versions. For that reason, the default is to use cgroupfs (see #17704, #16238, #16256, and various other issues). Unless you have a very specific reason to require systemd cgroups, I'd recommend using the cgroupfs option.

@TimWolla does switching to cgroupfs resolve your issue as well?

Member

thaJeztah commented Oct 25, 2017

The systemd cgroups implementation is known to be troublesome, and frequently breaks between systemd versions. For that reason, the default is to use cgroupfs (see #17704, #16238, #16256, and various other issues). Unless you have a very specific reason to require systemd cgroups, I'd recommend using the cgroupfs option.

@TimWolla does switching to cgroupfs resolve your issue as well?

@TimWolla

This comment has been minimized.

Show comment
Hide comment
@TimWolla

TimWolla Oct 26, 2017

Contributor
Contributor

TimWolla commented Oct 26, 2017

@thaJeztah thaJeztah added this to backlog in maintainers-session Oct 26, 2017

@thaJeztah thaJeztah removed this from backlog in maintainers-session Oct 26, 2017

@tiny1990

This comment has been minimized.

Show comment
Hide comment
@tiny1990

tiny1990 Oct 27, 2017

@thaJeztah what's difference between cgroupfs and systemd

tiny1990 commented Oct 27, 2017

@thaJeztah what's difference between cgroupfs and systemd

kolyshkin added a commit to kolyshkin/moby that referenced this issue Jan 31, 2018

Test for systemd cgroupdriver memory setting
This is a test case for issue moby#35123,
making sure we can set container's memory limit when using
`native.cgroupdriver=systemd`.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

kolyshkin added a commit to kolyshkin/moby that referenced this issue Feb 1, 2018

Test for systemd cgroupdriver memory setting
This is a test case for issue moby#35123,
making sure we can set container's memory limit when using
`native.cgroupdriver=systemd`.

[v2: skip if no systemd present]

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
@tossmilestone

This comment has been minimized.

Show comment
Hide comment
@tossmilestone

tossmilestone Feb 6, 2018

Contributor

@thaJeztah Sorry to bother, I want to make sure that if this issue is fixed on 17.12, because we may upgrade to this version if it was fixed.

Contributor

tossmilestone commented Feb 6, 2018

@thaJeztah Sorry to bother, I want to make sure that if this issue is fixed on 17.12, because we may upgrade to this version if it was fixed.

@darshan-karia

This comment has been minimized.

Show comment
Hide comment
@darshan-karia

darshan-karia Feb 9, 2018

@tossmilestone I tried 17.12 in past and it did not work. The version which worked for me was 17.10

darshan-karia commented Feb 9, 2018

@tossmilestone I tried 17.12 in past and it did not work. The version which worked for me was 17.10

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Feb 10, 2018

Member

@darshan-karia what problem did you run into? Testing this on a 17.12 machine:

$ docker run -it --rm --memory 1G alpine echo hello world
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
hello world

That warning indicates that the machine doesn't support swap accounting. The container started, but limiting swap is not applied; after enabling (see Your kernel does not support cgroup swap limit capabilities);

$ docker run -it --rm --memory 1G alpine echo hello world
hello world
Member

thaJeztah commented Feb 10, 2018

@darshan-karia what problem did you run into? Testing this on a 17.12 machine:

$ docker run -it --rm --memory 1G alpine echo hello world
WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
hello world

That warning indicates that the machine doesn't support swap accounting. The container started, but limiting swap is not applied; after enabling (see Your kernel does not support cgroup swap limit capabilities);

$ docker run -it --rm --memory 1G alpine echo hello world
hello world
@darshan-karia

This comment has been minimized.

Show comment
Hide comment
@darshan-karia

darshan-karia Mar 6, 2018

@thaJeztah The issue I was encountering was when running a systemd container with memory option.
But, I tried running systemd container after your comment, and that seems to be working.

I must have confused the versions, as I was trying many for my experiment, sorry about confusion.

darshan-karia commented Mar 6, 2018

@thaJeztah The issue I was encountering was when running a systemd container with memory option.
But, I tried running systemd container after your comment, and that seems to be working.

I must have confused the versions, as I was trying many for my experiment, sorry about confusion.

kolyshkin added a commit to kolyshkin/moby that referenced this issue Mar 12, 2018

Test for systemd cgroupdriver memory setting
This is a test case for issue moby#35123,
making sure we can set container's memory limit when using
`native.cgroupdriver=systemd`.

[v2: skip if no systemd present]
[v3: add --iptables=false to avoid flaky tests with t.Parallel()]

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

kolyshkin added a commit to kolyshkin/moby that referenced this issue Mar 12, 2018

Test for systemd cgroupdriver memory setting
This is a test case for issue moby#35123,
making sure we can set container's memory limit when using
`native.cgroupdriver=systemd`.

[v2: skip if no systemd present]
[v3: add --iptables=false to avoid flaky tests with t.Parallel()]

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

kolyshkin added a commit to kolyshkin/moby that referenced this issue Mar 15, 2018

Test for systemd cgroupdriver memory setting
This is a test case for issue moby#35123,
making sure we can set container's memory limit when using
`native.cgroupdriver=systemd`.

[v2: skip if no systemd present]
[v3: add --iptables=false to avoid flaky tests with t.Parallel()]

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

kolyshkin added a commit to kolyshkin/moby that referenced this issue Mar 19, 2018

Test for systemd cgroupdriver memory setting
This is a test case for issue moby#35123,
making sure we can set container's memory limit when using
`native.cgroupdriver=systemd`.

[v2: skip if no systemd present]
[v3: add --iptables=false to avoid flaky tests with t.Parallel()]
[v4: rebase after PR#36507 merge]

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>

thaJeztah added a commit to thaJeztah/docker-ce that referenced this issue Mar 22, 2018

Test for systemd cgroupdriver memory setting
This is a test case for issue moby/moby#35123,
making sure we can set container's memory limit when using
`native.cgroupdriver=systemd`.

[v2: skip if no systemd present]
[v3: add --iptables=false to avoid flaky tests with t.Parallel()]
[v4: rebase after PR#36507 merge]

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 4ca5c5361059e29ed31074ca5b96f8b2030b5f99)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Mar 22, 2018

Member

The fix for this should be in Docker 17.12.1, and a test is added in #36172. I'll go ahead and close this 👍

Member

thaJeztah commented Mar 22, 2018

The fix for this should be in Docker 17.12.1, and a test is added in #36172. I'll go ahead and close this 👍

@thaJeztah thaJeztah closed this Mar 22, 2018

docker-jenkins pushed a commit to docker/docker-ce that referenced this issue Mar 22, 2018

Test for systemd cgroupdriver memory setting
This is a test case for issue moby/moby#35123,
making sure we can set container's memory limit when using
`native.cgroupdriver=systemd`.

[v2: skip if no systemd present]
[v3: add --iptables=false to avoid flaky tests with t.Parallel()]
[v4: rebase after PR#36507 merge]

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 4ca5c5361059e29ed31074ca5b96f8b2030b5f99
Component: engine
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment