Swap limit capabilities failure while trying to set memory limit #847

Closed
flaviamissi opened this Issue Jun 6, 2013 · 22 comments

Comments

Projects
None yet
10 participants
@flaviamissi

Hi guys,

I'm trying to set memory limit on docker commit using the -run option, e.g.:

docker commit  -run='{"Memory": 128}' 0b5e63a76024 flaviamissi/test

This works as expected, but when I attempt to run a container using the obtained image I get the following:

$ docker run -t -i tsuru/test /bin/bash
WARNING:  Your kernel does not support memory swap capabilities. Limitation discarded.
lxc-start: Device or resource busy - write /sys/fs/cgroup/memory/lxc/5e29ddc69751e982fd1895e5d8b8f3159184ab9859c2bc3747b6c16d17a9e1ff/memory.limit_in_bytes : Device or resource busy
lxc-start: Error setting memory.limit_in_bytes to 128 for lxc/5e29ddc69751e982fd1895e5d8b8f3159184ab9859c2bc3747b6c16d17a9e1ff

lxc-start: failed to setup the cgroups for '5e29ddc69751e982fd1895e5d8b8f3159184ab9859c2bc3747b6c16d17a9e1ff'
lxc-start: failed to spawn '5e29ddc69751e982fd1895e5d8b8f3159184ab9859c2bc3747b6c16d17a9e1ff'
lxc-start: Device or resource busy - failed to remove cgroup '/sys/fs/cgroup/cpuset/lxc/5e29ddc69751e982fd1895e5d8b8f3159184ab9859c2bc3747b6c16d17a9e1ff'
lxc-start: Device or resource busy - failed to remove cgroup '/sys/fs/cgroup/cpu/lxc/5e29ddc69751e982fd1895e5d8b8f3159184ab9859c2bc3747b6c16d17a9e1ff'
lxc-start: Device or resource busy - failed to remove cgroup '/sys/fs/cgroup/cpuacct/lxc/5e29ddc69751e982fd1895e5d8b8f3159184ab9859c2bc3747b6c16d17a9e1ff'
lxc-start: Device or resource busy - failed to remove cgroup '/sys/fs/cgroup/memory/lxc/5e29ddc69751e982fd1895e5d8b8f3159184ab9859c2bc3747b6c16d17a9e1ff'
lxc-start: Device or resource busy - failed to remove cgroup '/sys/fs/cgroup/devices/lxc/5e29ddc69751e982fd1895e5d8b8f3159184ab9859c2bc3747b6c16d17a9e1ff'
lxc-start: Device or resource busy - failed to remove cgroup '/sys/fs/cgroup/freezer/lxc/5e29ddc69751e982fd1895e5d8b8f3159184ab9859c2bc3747b6c16d17a9e1ff'

What is puzzling me is the warning printed by docker:

WARNING:  Your kernel does not support memory swap capabilities. Limitation discarded.

If you pay attention, I'm not setting the MemorySwap option, but the Memory opt.

I've found the code that logs it, and it seems that the check is wrong, see:

if container.Config.Memory > 0 && !container.runtime.capabilities.SwapLimit {
...

What should be really checked here isn't the container.Config.MemorySwap attribute?

Would someone mind to clarify this to me?

Thanks!

@unclejack

This comment has been minimized.

Show comment
Hide comment
@unclejack

unclejack Jun 6, 2013

Contributor

@flaviamissi The memory limit is in bytes. You'd have to use 134217728 for the memory limit if you want to set it to 128MB.

I'm getting the exact same error you're getting if I use 128 as the memory limit. Passing 134217728 works.

docker run -m 128 ubuntu true
lxc-start: Device or resource busy - write /sys/fs/cgroup/memory/lxc/bbbc00f25479d111449d3d117ca322cfe97b2fc988b26967f5371c1bda674bd7/memory.limit_in_bytes : Device or resource busy
lxc-start: Error setting memory.limit_in_bytes to 128 for lxc/bbbc00f25479d111449d3d117ca322cfe97b2fc988b26967f5371c1bda674bd7

lxc-start: failed to setup the cgroups for 'bbbc00f25479d111449d3d117ca322cfe97b2fc988b26967f5371c1bda674bd7'
lxc-start: failed to spawn 'bbbc00f25479d111449d3d117ca322cfe97b2fc988b26967f5371c1bda674bd7'

# let's set the limit in bytes
docker run -m 134217728 ubuntu true
# execution ends without any issue
Contributor

unclejack commented Jun 6, 2013

@flaviamissi The memory limit is in bytes. You'd have to use 134217728 for the memory limit if you want to set it to 128MB.

I'm getting the exact same error you're getting if I use 128 as the memory limit. Passing 134217728 works.

docker run -m 128 ubuntu true
lxc-start: Device or resource busy - write /sys/fs/cgroup/memory/lxc/bbbc00f25479d111449d3d117ca322cfe97b2fc988b26967f5371c1bda674bd7/memory.limit_in_bytes : Device or resource busy
lxc-start: Error setting memory.limit_in_bytes to 128 for lxc/bbbc00f25479d111449d3d117ca322cfe97b2fc988b26967f5371c1bda674bd7

lxc-start: failed to setup the cgroups for 'bbbc00f25479d111449d3d117ca322cfe97b2fc988b26967f5371c1bda674bd7'
lxc-start: failed to spawn 'bbbc00f25479d111449d3d117ca322cfe97b2fc988b26967f5371c1bda674bd7'

# let's set the limit in bytes
docker run -m 134217728 ubuntu true
# execution ends without any issue
@flaviamissi

This comment has been minimized.

Show comment
Hide comment
@flaviamissi

flaviamissi Jun 6, 2013

Yes, I just figured that out, thanks!

But the warning still seems wrong...

Yes, I just figured that out, thanks!

But the warning still seems wrong...

@unclejack

This comment has been minimized.

Show comment
Hide comment
@unclejack

unclejack Jun 6, 2013

Contributor

What distro and what kernel are you running docker on? I didn't get that warning on Ubuntu 13.04 with kernel 3.9 when passing 128 as the memory limit.

Contributor

unclejack commented Jun 6, 2013

What distro and what kernel are you running docker on? I didn't get that warning on Ubuntu 13.04 with kernel 3.9 when passing 128 as the memory limit.

@flaviamissi

This comment has been minimized.

Show comment
Hide comment
@flaviamissi

flaviamissi Jun 6, 2013

ubuntu 13.04 with kernel 3.8

ubuntu 13.04 with kernel 3.8

@flaviamissi

This comment has been minimized.

Show comment
Hide comment
@flaviamissi

flaviamissi Jun 6, 2013

The 3.9 kernel might have the swap capability, but what I'm trying to say is that the if that checks it is wrong:

The 3.9 kernel might have the swap capability, but what I'm trying to say is that the if that checks it is wrong:

@flaviamissi

This comment has been minimized.

Show comment
Hide comment
@flaviamissi

flaviamissi Jun 6, 2013

pressed commend before finishing, I meant this if:

if container.Config.Memory > 0 && !container.runtime.capabilities.SwapLimit {

pressed commend before finishing, I meant this if:

if container.Config.Memory > 0 && !container.runtime.capabilities.SwapLimit {
@flaviamissi

This comment has been minimized.

Show comment
Hide comment
@flaviamissi

flaviamissi Jun 6, 2013

It's probably a typo...

It's probably a typo...

@shykes

This comment has been minimized.

Show comment
Hide comment
@shykes

shykes Jun 6, 2013

Collaborator

Flavia, under a certain memory allocation the system can barely get
anything done, and will fail at a very early stage - in this case it can't
even complete setting up the process group. In theory we should chase down
all cases where an error is caused by insufficient memory. In practice this
seemed like more work than it was worth.

On Thu, Jun 6, 2013 at 11:11 PM, unclejack notifications@github.com wrote:

What distro and what kernel are you running docker on? I didn't get that
warning on Ubuntu 13.04 with kernel 3.9 when passing 128 as the memory
limit.


Reply to this email directly or view it on GitHubhttps://github.com/dotcloud/docker/issues/847#issuecomment-19074434
.

Collaborator

shykes commented Jun 6, 2013

Flavia, under a certain memory allocation the system can barely get
anything done, and will fail at a very early stage - in this case it can't
even complete setting up the process group. In theory we should chase down
all cases where an error is caused by insufficient memory. In practice this
seemed like more work than it was worth.

On Thu, Jun 6, 2013 at 11:11 PM, unclejack notifications@github.com wrote:

What distro and what kernel are you running docker on? I didn't get that
warning on Ubuntu 13.04 with kernel 3.9 when passing 128 as the memory
limit.


Reply to this email directly or view it on GitHubhttps://github.com/dotcloud/docker/issues/847#issuecomment-19074434
.

@unclejack

This comment has been minimized.

Show comment
Hide comment
@unclejack

unclejack Jun 6, 2013

Contributor

That piece of code is correct. It's meant to let us know that we lack swap limiting capabilities when we tell docker that we want to limit memory.

Swap limits are automatically computed based on the memory limits.

Contributor

unclejack commented Jun 6, 2013

That piece of code is correct. It's meant to let us know that we lack swap limiting capabilities when we tell docker that we want to limit memory.

Swap limits are automatically computed based on the memory limits.

@flaviamissi

This comment has been minimized.

Show comment
Hide comment
@flaviamissi

flaviamissi Jun 6, 2013

Thanks, @unclejack, that clarifies the question :)

Since it was not an error, I'll close the issue.

Thanks everybody o/

Thanks, @unclejack, that clarifies the question :)

Since it was not an error, I'll close the issue.

Thanks everybody o/

@flaviamissi flaviamissi closed this Jun 6, 2013

@shykes

This comment has been minimized.

Show comment
Hide comment
@shykes

shykes Jun 6, 2013

Collaborator

Sorry, I missed that part. the '-m' flag actually sets a certain blend of
swap and real memory, assuming a certain preset ratio. Currently you can't
control swap and actual memory limits separately in docker (although you
can in the underlying lxc config).

On Thu, Jun 6, 2013 at 11:16 PM, unclejack notifications@github.com wrote:

That piece of code is correct. It's meant to let us know that we lack swap
limiting capabilities when we tell docker that we want to limit memory.

Swap limits are automatically computed based on the memory limits.


Reply to this email directly or view it on GitHubhttps://github.com/dotcloud/docker/issues/847#issuecomment-19074728
.

Collaborator

shykes commented Jun 6, 2013

Sorry, I missed that part. the '-m' flag actually sets a certain blend of
swap and real memory, assuming a certain preset ratio. Currently you can't
control swap and actual memory limits separately in docker (although you
can in the underlying lxc config).

On Thu, Jun 6, 2013 at 11:16 PM, unclejack notifications@github.com wrote:

That piece of code is correct. It's meant to let us know that we lack swap
limiting capabilities when we tell docker that we want to limit memory.

Swap limits are automatically computed based on the memory limits.


Reply to this email directly or view it on GitHubhttps://github.com/dotcloud/docker/issues/847#issuecomment-19074728
.

@samof76

This comment has been minimized.

Show comment
Hide comment
@samof76

samof76 Aug 27, 2013

But so that you are not worried about it any long you need to boot the kernel with these parameters cgroup_enable=memory swapaccount=1.

To do that you would need to edit your /etc/default/grub file and set this

GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"

samof76 commented Aug 27, 2013

But so that you are not worried about it any long you need to boot the kernel with these parameters cgroup_enable=memory swapaccount=1.

To do that you would need to edit your /etc/default/grub file and set this

GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"
@matthewmueller

This comment has been minimized.

Show comment
Hide comment
@matthewmueller

matthewmueller Sep 23, 2013

Contributor

running into same warning while running 13.04 and with -m 5242880. @samof76 what does that command do?

Contributor

matthewmueller commented Sep 23, 2013

running into same warning while running 13.04 and with -m 5242880. @samof76 what does that command do?

@matthewmueller matthewmueller referenced this issue in matthewmueller/coderunner Sep 23, 2013

Open

add memory constraints #17

@samof76

This comment has been minimized.

Show comment
Hide comment
@samof76

samof76 Sep 30, 2013

@matthewmueller this is to enable memory and swap accounting. But this also enables more control features for cgroups. Please read more here... https://www.kernel.org/doc/Documentation/cgroups/memory.txt

samof76 commented Sep 30, 2013

@matthewmueller this is to enable memory and swap accounting. But this also enables more control features for cgroups. Please read more here... https://www.kernel.org/doc/Documentation/cgroups/memory.txt

@petri

This comment has been minimized.

Show comment
Hide comment
@petri

petri Oct 22, 2013

Sounds like this should be included in the docker installation docs? Ie. memory limits don't work unless swap limiting capabilities are enabled.

petri commented Oct 22, 2013

Sounds like this should be included in the docker installation docs? Ie. memory limits don't work unless swap limiting capabilities are enabled.

@rca

This comment has been minimized.

Show comment
Hide comment
@rca

rca Oct 30, 2013

@samof76 thanks for the GRUB config line!

rca commented Oct 30, 2013

@samof76 thanks for the GRUB config line!

@thebyrd

This comment has been minimized.

Show comment
Hide comment
@thebyrd

thebyrd May 13, 2014

@unclejack
I ran into this error when starting the daemon with sudo docker -d and then the daemon was unresponsive. The server had plenty of resources with no spikes (see graph below). I don't know exactly how many containers were running, but I'd guess around 1000. Restarting the daemon with upstart also failed.

  1. What's the best way to remove the containers without the daemon running?
  2. What's causing this error?

screen shot 2014-05-13 at 12 31 44 pm

thebyrd commented May 13, 2014

@unclejack
I ran into this error when starting the daemon with sudo docker -d and then the daemon was unresponsive. The server had plenty of resources with no spikes (see graph below). I don't know exactly how many containers were running, but I'd guess around 1000. Restarting the daemon with upstart also failed.

  1. What's the best way to remove the containers without the daemon running?
  2. What's causing this error?

screen shot 2014-05-13 at 12 31 44 pm

@scadgek

This comment has been minimized.

Show comment
Hide comment
@scadgek

scadgek Jul 15, 2015

Sorry for necroposting, but I think this still should go here.
So I'm on Ubuntu 14.04, I updated GRUB options with GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1" and GRUB_CMDLINE_LINUX_DEFAULT="cgroup_enable=memory swapaccount=1", done update-grub, rebooted and still receive
WARNING: Your kernel does not support swap limit capabilities. Limitation discarded.
By the way, I also can see these options set in /boot/grub/grub.cfg.
I have no idea what I am missing

scadgek commented Jul 15, 2015

Sorry for necroposting, but I think this still should go here.
So I'm on Ubuntu 14.04, I updated GRUB options with GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1" and GRUB_CMDLINE_LINUX_DEFAULT="cgroup_enable=memory swapaccount=1", done update-grub, rebooted and still receive
WARNING: Your kernel does not support swap limit capabilities. Limitation discarded.
By the way, I also can see these options set in /boot/grub/grub.cfg.
I have no idea what I am missing

@shredding

This comment has been minimized.

Show comment
Hide comment
@shredding

shredding Jan 26, 2016

@scadgek : Did you got it fixed? I fail to enable it as well.

@scadgek : Did you got it fixed? I fail to enable it as well.

@scadgek

This comment has been minimized.

Show comment
Hide comment
@scadgek

scadgek Jan 26, 2016

@shredding No, I didn't. The task became not important and I stopped looking for the ways to solve.

scadgek commented Jan 26, 2016

@shredding No, I didn't. The task became not important and I stopped looking for the ways to solve.

@shredding

This comment has been minimized.

Show comment
Hide comment
@shredding

shredding Jan 26, 2016

I tracked it down. I assume you are on a VPS like Digital Oceans Droplets? You'd need to use 15.05 or 15.10 then because 14.04 does not support internal kernel configuration from within a vps.

I tracked it down. I assume you are on a VPS like Digital Oceans Droplets? You'd need to use 15.05 or 15.10 then because 14.04 does not support internal kernel configuration from within a vps.

@scadgek

This comment has been minimized.

Show comment
Hide comment
@scadgek

scadgek Jan 26, 2016

@shredding No it wasn't Digital Ocean, it was my local machine - not even a virtual one

scadgek commented Jan 26, 2016

@shredding No it wasn't Digital Ocean, it was my local machine - not even a virtual one

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment