Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to start container using low memory limit #2551

Open
chrischdi opened this Issue Feb 18, 2019 · 3 comments

Comments

Projects
None yet
3 participants
@chrischdi
Copy link

chrischdi commented Feb 18, 2019

Issue Report

Bug

Container Linux Version

NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1967.6.0
VERSION_ID=1967.6.0
BUILD_ID=2019-02-12-2138
PRETTY_NAME="Container Linux by CoreOS 1967.6.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr

Environment

We're running in Openstack / VMWare Integrated Openstack:

$ cat /proc/meminfo  | grep MemTotal
MemTotal:        8171244 kB

Expected Behavior

Docker container starts

Actual Behavior

Docker fails to start a container

Reproduction Steps

  1. Start docker container including memory limit:
    docker run --rm --memory 10485760 -ti jimmidyson/configmap-reload:v0.2.2
    

/run/torcx/bin/docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused "process_linux.go:367: setting cgroup config for procHooks process caused \"failed to write 10485760 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/docker/aa95f28a0ecfc178c4fb52e361281f936f2e556679f55ae193aacd5b150a6416/memory.limit_in_bytes: device or resource busy\""": unknown.


When using a higher memory limit, starting the container is no problem.
When using a slightly higher memory limit, the result is varying:

$ docker run --rm --memory 12534336 -ti jimmidyson/configmap-reload:v0.2.2
/run/torcx/bin/docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused "process_linux.go:367: setting cgroup config for procHooks process caused \"failed to write 12534336 to memory.limit_in_bytes: write /sys/fs/cgroup/memory/docker/5ea6580804d337da46c06f4c06b4df7dce62adb17362687ac3d0faabe966f302/memory.limit_in_bytes: device or resource busy\""": unknown.
$ docker run --rm --memory 12534336 -ti jimmidyson/configmap-reload:v0.2.2
/run/torcx/bin/docker: Error response from daemon: OCI runtime create failed: docker-runc did not terminate sucessfully: panic: No error following JSON procError payload. [recovered]
panic: No error following JSON procError payload.

goroutine 1 [running]:
github.com/opencontainers/runc/vendor/github.com/urfave/cli.HandleAction.func1(0xc420145870)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:478 +0x23d
panic(0x55f49ba9ce40, 0x55f49bb1d5c0)
/usr/lib/go1.10/src/runtime/panic.go:502 +0x22d
github.com/opencontainers/runc/libcontainer.parseSync(0x55f49bb1f500, 0xc4200da108, 0xc420144a50, 0x3, 0x0)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/libcontainer/sync.go:99 +0x2f3
github.com/opencontainers/runc/libcontainer.(*initProcess).start(0xc4200e0700, 0x0, 0x0)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/libcontainer/process_linux.go:323 +0x3b4
github.com/opencontainers/runc/libcontainer.(*linuxContainer).start(0xc4201ae000, 0xc4200e6d80, 0x1, 0x0, 0xc420144c50)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/libcontainer/container_linux.go:343 +0x95
github.com/opencontainers/runc/libcontainer.(*linuxContainer).Start(0xc4201ae000, 0xc4200e6d80, 0x0, 0x0)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/libcontainer/container_linux.go:237 +0xc5
main.(*runner).run(0xc420145398, 0xc4200e49c0, 0x0, 0x0, 0x0)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/utils_linux.go:303 +0x90c
main.startContainer(0xc420124140, 0xc4201260e0, 0x1, 0x0, 0x0, 0x20, 0xc4200d2cf0)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/utils_linux.go:428 +0x3c4
main.glob..func2(0xc420124140, 0x0, 0x0)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/create.go:65 +0xb0
reflect.Value.call(0x55f49baa85c0, 0x55f49bb1cb10, 0x13, 0x55f49b7d9e57, 0x4, 0xc420055830, 0x1, 0x1, 0xc4200600c0, 0x55f49bb10e20, ...)
/usr/lib/go1.10/src/reflect/value.go:447 +0x96b
reflect.Value.Call(0x55f49baa85c0, 0x55f49bb1cb10, 0x13, 0xc420055830, 0x1, 0x1, 0x55f49b79d4b8, 0x55f49b7d9f77, 0x4)
/usr/lib/go1.10/src/reflect/value.go:308 +0xa6
github.com/opencontainers/runc/vendor/github.com/urfave/cli.HandleAction(0x55f49baa85c0, 0x55f49bb1cb10, 0xc420124140, 0x0, 0x0)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:487 +0x16f
github.com/opencontainers/runc/vendor/github.com/urfave/cli.Command.Run(0x55f49b7da66b, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x55f49b7de64e, 0x12, 0x0, ...)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/command.go:191 +0x9ac
github.com/opencontainers/runc/vendor/github.com/urfave/cli.(*App).Run(0xc420102180, 0xc4200100f0, 0xf, 0xf, 0x0, 0x0)
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/vendor/github.com/urfave/cli/app.go:240 +0x548
main.main()
/build/amd64-usr/var/tmp/portage/app-emulation/docker-runc-1.0.0_rc5_p19-r1/work/docker-runc-1.0.0_rc5_p19/src/github.com/opencontainers/runc/main.go:147 +0xaea
: unknown.
$ docker run --rm --memory 12534336 -ti jimmidyson/configmap-reload:v0.2.2
2019/02/18 07:16:16 Missing volume-dir
2019/02/18 07:16:16
Usage of /configmap-reload:
-volume-dir value
the config map volume directory to watch for updates; may be used multiple times
-webhook-method string
the HTTP method url to use to send the webhook (default "POST")
-webhook-status-code int
the HTTP status code indicating successful triggering of reload (default 200)
-webhook-url string
the url to send a request to when the specified config map volume directory has been updated


### Other Information ###

Unable to reproduce the problem on same os configuration but using the following CoreOS Release:

$ cat /etc/os-release
NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1967.3.0
VERSION_ID=1967.3.0
BUILD_ID=2019-01-08-0044
PRETTY_NAME="Container Linux by CoreOS 1967.3.0 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

We did not test `1967.5.0`
@chrischdi

This comment has been minimized.

Copy link
Author

chrischdi commented Feb 18, 2019

Also tested 1967.5.0: so it looks to me that the runc fix is producing this bug.
Because I can also reproduce the problem on 1967.5.0

One more note: we are using docker 18.06.1-ce, installed via torcx.

@dongsupark

This comment has been minimized.

Copy link

dongsupark commented Feb 18, 2019

You would probably want to look into the upstream issue, which is still in discussions: opencontainers/runc#1980

@chrischdi

This comment has been minimized.

Copy link
Author

chrischdi commented Mar 12, 2019

Hi again,

the problem should be resolved by opencontainers/runc#1984 and opencontainers/runc#2006 .
Will these patches also get into a new CoreOS Release?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.