Cannot start container <..>: write <..>memory.memsw.limit_in_bytes: invalid argument #10280

ernetas · 2015-01-22T17:46:42Z

[root@local ernestas]# docker run -m="2G" -t -i slave /sbin/my_init -- bash -l
2015/01/22 19:15:26 Error response from daemon: Cannot start container 8096197f79ccc8277f0a8482fc148bd3207e615c038a2bf6422556bcf2db3782: write /sys/fs/cgroup/memory/system.slice/docker-8096197f79ccc8277f0a8482fc148bd3207e615c038a2bf6422556bcf2db3782.scope/memory.memsw.limit_in_bytes: invalid argument
[root@local ernestas]# 

[root@local ernestas]# docker start bd9041d95adc
Error response from daemon: Cannot start container bd9041d95adc: write /sys/fs/cgroup/memory/system.slice/docker-bd9041d95adcdf46ff49b9220b99deb6aa4dd414d204a62e87b65fd29be6bd56.scope/memory.memsw.limit_in_bytes: invalid argument
FATA[0000] Error: failed to start one or more containers 
[root@local ernestas]#

Exactly the same container worked fine on another installation of Fedora 21 in VirtualBox. This is running latest Docker from official RPM repos:

Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 5bc2ff8/1.4.1
OS/Arch (client): linux/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8/1.4.1

journalctl ends with:

Jan 22 19:17:30 local docker[6160]: time="2015-01-22T19:17:30+02:00" level="error" msg="Warning: error unmounting device bd9041d95adcdf46ff49b9220b99deb6aa4dd414d204a62e87b65fd29be6bd56: UnmountDevice: device not-mounted id bd9041d95adcdf46ff49b9220b99deb6aa4dd414d204a62e87b65fd29be6bd56\n"
Jan 22 19:17:30 local docker[6160]: time="2015-01-22T19:17:30+02:00" level="info" msg="+job log(die, bd9041d95adcdf46ff49b9220b99deb6aa4dd414d204a62e87b65fd29be6bd56, slave:latest)"
Jan 22 19:17:30 local docker[6160]: time="2015-01-22T19:17:30+02:00" level="info" msg="-job log(die, bd9041d95adcdf46ff49b9220b99deb6aa4dd414d204a62e87b65fd29be6bd56, slave:latest) = OK (0)"
Jan 22 19:17:30 local docker[6160]: Cannot start container bd9041d95adc: write /sys/fs/cgroup/memory/system.slice/docker-bd9041d95adcdf46ff49b9220b99deb6aa4dd414d204a62e87b65fd29be6bd56.scope/memory.memsw.limit_in_bytes: invalid argument
Jan 22 19:17:30 local docker[6160]: time="2015-01-22T19:17:30+02:00" level="info" msg="-job start(bd9041d95adc) = ERR (1)"
Jan 22 19:17:30 local docker[6160]: time="2015-01-22T19:17:30+02:00" level="error" msg="Handler for POST /containers/{name:.*}/start returned error: Cannot start container bd9041d95adc: write /sys/fs/cgroup/memory/system.slice/docker-bd9041d95adcdf46ff49b9220b99deb6aa4dd414d204a62e87b65fd29be6bd56.scope/memory.memsw.limit_in_bytes: invalid argument"
Jan 22 19:17:30 local docker[6160]: time="2015-01-22T19:17:30+02:00" level="error" msg="HTTP Error: statusCode=500 Cannot start container bd9041d95adc: write /sys/fs/cgroup/memory/system.slice/docker-bd9041d95adcdf46ff49b9220b99deb6aa4dd414d204a62e87b65fd29be6bd56.scope/memory.memsw.limit_in_bytes: invalid argument"

Which could be somehow related to #4036, but doesn't seem to be an exact replica of this issue. Full logs here: https://gist.github.com/ernetas/7b1ad29db74cd4f093df

What could be the issue here? The containers fail to start only when started with memory limitations (-m="x").

The text was updated successfully, but these errors were encountered:

estesp · 2015-01-22T21:45:19Z

Can you provide uname -a and cat /proc/cmdline output from the working and failing cases? I would say to check you have the proper memory limits/swap accounting turned on in the case of the failing system, but I assume you do as I don't see the warnings in the logs that they aren't available (see http://docs.docker.com/installation/ubuntulinux/#memory-and-swap-accounting)

ernetas · 2015-01-22T22:02:06Z

Of course!

This is from the working machine:

[root@localhost ~]# uname -a
Linux localhost.localdomain 3.17.8-300.fc21.x86_64 #1 SMP Thu Jan 8 23:32:49 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.17.8-300.fc21.x86_64 root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/swap rd.lvm.lv=fedora/root rhgb quiet LANG=en_US.UTF-8
[root@localhost ~]#

The same commands on the failing host:

[root@local]~# uname -a
Linux local 3.17.8-300.fc21.x86_64 #1 SMP Thu Jan 8 23:32:49 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@local]~# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-3.17.8-300.fc21.x86_64 root=/dev/mapper/fedora-root ro rd.lvm.lv=fedora/swap rd.lvm.lv=fedora/root rhgb quiet LANG=en_US.UTF-8
[root@local]~#

Both seem identical to me.

Yes, the memory/swap accounting seem to be on, as no errors are shown regarding them on both machines.

ernetas · 2015-01-22T22:57:26Z

I launched a simple container with no memory limitations and tried to change it on the fly with simple echo (what seems to be an alternative to what Docker does, right?):

[root@local ~]# docker run -d -t -i phusion/baseimage /sbin/my_init -- bash -l
5b6e94aea7c1a47e8d418e5c66bc966527ea0b5b987e5d27d039d14be85f7bc2
[root@local ~]# docker ps
CONTAINER ID        IMAGE                      COMMAND                CREATED             STATUS              PORTS               NAMES
5b6e94aea7c1        phusion/baseimage:0.9.16   "/sbin/my_init -- ba   2 seconds ago       Up 1 seconds                            dreamy_bardeen      
[root@local ~]# echo 2000000000 > /sys/fs/cgroup/memory/system.slice/docker-5b6e94aea7c1a47e8d418e5c66bc966527ea0b5b987e5d27d039d14be85f7bc2.scope/memory.memsw.limit_in_bytes 
bash: echo: write error: Invalid argument
[root@local ~]# echo 2000000000 > /sys/fs/cgroup/memory/system.slice/docker-5b6e94aea7c1a47e8d418e5c66bc966527ea0b5b987e5d27d039d14be85f7bc2.scope/memory.limit_in_bytes 
[root@local ~]# echo 2000000000 > /sys/fs/cgroup/memory/system.slice/docker-5b6e94aea7c1a47e8d418e5c66bc966527ea0b5b987e5d27d039d14be85f7bc2.scope/memory.memsw.limit_in_bytes 
[root@local ~]#

This actually works:

root@5b6e94aea7c1:/# stress --vm 1 --vm-bytes 3G
stress: info: [146] dispatching hogs: 0 cpu, 0 io, 1 vm, 0 hdd
stress: FAIL: [146] (416) <-- worker 147 got signal 9
stress: WARN: [146] (418) now reaping child worker processes
stress: FAIL: [146] (452) failed run completed in 0s
root@5b6e94aea7c1:/#

Default values are:

[root@local ~]# cat /sys/fs/cgroup/memory/system.slice/docker-094e9dd329f1d1fe57a80f6e87a5db9bf7c8b1ec4de91669bfa3b570e34d7963.scope/memory.memsw.limit_in_bytes 
18446744073709551615
[root@local ~]# cat /sys/fs/cgroup/memory/system.slice/docker-094e9dd329f1d1fe57a80f6e87a5db9bf7c8b1ec4de91669bfa3b570e34d7963.scope/memory.limit_in_bytes 
18446744073709551615
[root@local ~]#

Obviously, one cannot change memory.memsw.limit_in_bytes first. memory.limit_in_bytes has to be smaller or equal to that first. However, the same is on the working (VirtualBox) machine, so it's not clear to me why Docker does this correctly on one host, but not on the other:

[root@localhost ~]# cat /sys/fs/cgroup/memory/system.slice/docker-dcd8ef236c92815fdc327e82539792a485b416f5e23f71558fd51cdd6b4e407d.scope/memory.memsw.limit_in_bytes 
18446744073709551615
[root@localhost ~]# cat /sys/fs/cgroup/memory/system.slice/docker-dcd8ef236c92815fdc327e82539792a485b416f5e23f71558fd51cdd6b4e407d.scope/memory.limit_in_bytes 18446744073709551615
[root@localhost ~]#

The above is without memory limitations, whereas the below is with:

[root@localhost ~]# cat /sys/fs/cgroup/memory/system.slice/docker-b3be58b835edfe7d7c6260c7bb9fad4f27c5f13b235ca82e4fd7d792d3ba8293.scope/memory.memsw.limit_in_bytes 
4000002048
[root@localhost ~]# cat /sys/fs/cgroup/memory/system.slice/docker-b3be58b835edfe7d7c6260c7bb9fad4f27c5f13b235ca82e4fd7d792d3ba8293.scope/memory.limit_in_bytes 
2000003072
[root@localhost ~]#

Also what's interesting on the working host:

Jan 23 00:41:31 localhost.localdomain systemd[1]: Failed to set memory.limit_in_bytes on : Invalid argument
Jan 23 00:41:31 localhost.localdomain systemd[1]: Failed to reset devices.list on /system.slice: Invalid argument

Whatsoever, on that host, both memory.limit_in_bytes and memory.memsw.limit_in_bytes are set correctly.

Once again, logs from a failing host: https://gist.github.com/ernetas/c1ac5cf5ee6e1913c611 Logs from a working host: https://gist.github.com/ernetas/6d0d40aa28e0f49c5103 They seem to be very different.

ernetas · 2015-01-23T10:53:46Z

Reinstalled fresh Fedora, using exactly the same steps as before... Now both hosts are failing.

setiseta · 2015-01-23T14:58:11Z

Same sing here. A new installed & updated Fedora 21 Server fails on setting memory limits in docker.
It seems to be an "new" update since 21.01.2015.
I got the last installation an this day ready. and this one works.

estesp · 2015-01-23T16:41:46Z

So, this seems to be potentially some interaction with systemd that has changed.. does anything from #7015 seem applicable?

ernetas · 2015-01-23T21:53:27Z

Can't confirm, nor deny. But yes - the working host has systemd upgrade pending, as it turns out. #7015 does not seem to be applicable for my case, though - there are no similar errors in the logs.

Aside - I tested it with vfs before realizing it's a systemd issue - it didn't work as well.

ernetas · 2015-01-24T18:53:10Z

Can confirm - upgrading systemd and libgudev1 is the cause of the issue:

Jan 24 20:46:16 Updated: systemd-libs-216-16.fc21.x86_64
Jan 24 20:46:31 Updated: systemd-216-16.fc21.x86_64
Jan 24 20:46:32 Updated: systemd-python3-216-16.fc21.x86_64
Jan 24 20:46:32 Updated: systemd-python-216-16.fc21.x86_64
Jan 24 20:46:33 Updated: libgudev1-216-16.fc21.x86_64
Jan 24 20:46:33 Updated: systemd-compat-libs-216-16.fc21.x86_64

ernetas · 2015-01-26T12:46:06Z

Should we report this to RedHat bugzilla or is this an issue of Docker (given that you can launch a container with Docker and then later successfully change memory limitations yourself)?

crosbymichael · 2015-01-26T23:57:36Z

@ernetas are you able to test this with the 1.5.0 RC? We did some work around memory and mem-swap and I would be interested in seeing if you are encountering the same issue with it.

You can find binary downloads that are easily replaceable on your system in this comment for the upcoming release.

#10286 (comment)

ernetas · 2015-01-27T10:09:21Z

Sadly, still encountering it:

[root@localhost ernestas]# docker version
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): 7e803ba
OS/Arch (client): linux/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): 7e803ba
[root@localhost ernestas]# docker run -d -m="2000000000" --cap-add=SYS_NICE -t -i slave nice -n -18 /sbin/my_init -- bash -l 
b27956ed3e2753153f210683d23010c94099047e8a0ce0f07326f71d1f44c8dd
FATA[0001] Error response from daemon: Cannot start container b27956ed3e2753153f210683d23010c94099047e8a0ce0f07326f71d1f44c8dd: write /sys/fs/cgroup/memory/system.slice/docker-b27956ed3e2753153f210683d23010c94099047e8a0ce0f07326f71d1f44c8dd.scope/memory.memsw.limit_in_bytes: invalid argument 
[root@localhost ernestas]#

estesp · 2015-01-27T22:10:33Z

I'm looking into this a bit deeper. I've reviewed any cgroups related changes in the systemd update from -14 to -16; there are a few that look interesting, not to mention there are over 100 patches to the latest systemd update on Fedora.

I have a hunch that something in the systemd changes causes libcontainer's systemd.Apply(..) to fail setting the memory.limit_in_bytes cgroup, which then, since systemd doesn't support the memsw cgroup directly, libcontainer tries to write that file directly and fails because memory.limit_in_bytes wasn't set properly first. I'm going to validate that given I now have a Fedora21 VM with both pre-.16 and .16 systemd builds available and can dump some more info before failing the container start.

If we can narrow it down and it seems to be a problem in Fedora's systemd, then it would make sense to open a Red Hat bugzilla for the issue.

setiseta · 2015-01-27T22:21:19Z

Ok, thx, there are some new updates of systemd in fed repos today. Sorry, i
don't know the Version number but it keeps failing with docker & memory Limit.

Gesendet mit AquaMail für Android
http://www.aqua-mail.com

ernetas · 2015-01-27T22:25:48Z

Jan 27 23:55:09 Updated: systemd-libs-216-17.fc21.x86_64
Jan 27 23:55:12 Updated: systemd-216-17.fc21.x86_64
Jan 27 23:55:50 Updated: systemd-python3-216-17.fc21.x86_64
Jan 27 23:55:50 Updated: systemd-python-216-17.fc21.x86_64
Jan 27 23:55:50 Updated: systemd-compat-libs-216-17.fc21.x86_64

Tested with these upgrades (-17). Even after reboot and with Docker 1.5, it still throws the same error.

LK4D4 · 2015-01-28T01:14:54Z

Works for me with systemd-218:

cat /sys/fs/cgroup/memory/system.slice/docker-2046530acac1f15f03399a4dc750fa42f0363a9a16b972abcec127deff7af7d1.scope/memory.memsw.limit_in_bytes
4294967296

estesp · 2015-01-28T16:57:08Z

So, I'm fairly confident that a single patch that is unique to the Fedora/RH build of systemd is the culprit. In a working systemd on Fedora (216-12, 216-13, 216-14), this message is output in the logs:

Jan 28 11:01:34 localhost systemd: Failed to set memory.limit_in_bytes on : Invalid argument

While that message comes from the src/core/cgroups.c code that is setting the memory limit, it does appear to work (potentially that code is called again successfully?); I don't have the systemd debug fu (nor time to dig into it), other than to say that clearly the limit_in_bytes setting is being called and does appear to set the value given we can check afterwards and see it is set properly.

That message no longer appears with later systemd builds on Fedora (216-16 and 216-17), and this patch also appeared in those builds which are now failing:

0159-cgroup-memory-limits-on-are-not-supported.patch

diff --git a/src/core/cgroup.c b/src/core/cgroup.c
 12 index 5f3d9907b7..37db3d5289 100644
 13 --- a/src/core/cgroup.c
 14 +++ b/src/core/cgroup.c
 15 @@ -382,7 +382,7 @@ void cgroup_context_apply(CGroupContext *c, CGroupControllerMask mask, const cha
 16                  }
 17          }
 18
 19 -        if (mask & CGROUP_MEMORY) {
 20 +        if ((mask & CGROUP_MEMORY) & !is_root) {
 21                  if (c->memory_limit != (uint64_t) -1) {
 22                          char buf[DECIMAL_STR_MAX(uint64_t) + 1];
 23

The is_root variable is defined as true if path is empty or /, and from the error message above we can see the failing call from prior systemd builds had a path of "" (because the error message ends with "on :") where path is emitted in the log. Therefore, it would seem that it assumes this is an attempt to write to root and disallows it. Maybe that patch is not namespace aware? It only exists in the Fedora/Red Hat version of systemd, which is why @LK4D4 sees systemd-218 working fine.

A workaround in Docker is possible by catching the error when we try the manual write of memory.memsw.limit_in_bytes and trying to manually write limit_in_bytes and then memsw limit again. But, I think this should be brought to the attention of the Fedora/RH folks as that patch is suspect and needs further investigation from what I can tell.

ernetas · 2015-01-28T18:15:27Z

Thanks, @estesp. I contacted zbyszek, who commited the patches. For me, it seems that we are missing additional patches. E. g. systemd/systemd@4ad4900, which reverts 0159-cgroup-memory-limits-on-are-not-supported.patch patch and, correct me if I'm wrong, is included in systemd-218. Anyways, I will close this issue and update status once everything is clear.

keszybz · 2015-01-31T21:13:38Z

When it fails, the bad systemd verions is used on the host, in the container, or both?

ernetas · 2015-01-31T21:16:32Z

For me - on the host only. I'm trying to launch an Ubuntu container, so it doesn't support/have systemd yet. Also, I doubt if systemd could run inside a container?

keszybz · 2015-01-31T21:17:33Z

Hm, so my first theory is wrong.

Yes, it can run in a container.

setiseta · 2015-01-31T23:01:28Z

Its only on the host in my case.

Gesendet mit AquaMail für Android
http://www.aqua-mail.com

Am 31. Jänner 2015 22:14:46 schrieb keszybz notifications@github.com:

When it fails, the bad systemd verions is used on the host, in the
container, or both?

Reply to this email directly or view it on GitHub:
#10280 (comment)

keszybz · 2015-02-01T04:11:51Z

Fixed in http://cgit.freedesktop.org/systemd/systemd/commit/?id=a3bd89ea99. I'll try to issue an update for systemd package tomorrow, but it'll be a while before it passes through QA. I made a build at http://koji.fedoraproject.org/koji/taskinfo?taskID=8788274 with this one patch added if you want to upgrade now.

dustymabe · 2015-02-01T16:31:26Z

I hit this using Fedora 21 atomic. @keszybz Thanks for fixing.. Is there a bz that tracks this issue that I can follow?

This bug (in systemd) isn't really hard to reproduce but if anyone is interested I wrote down some steps to recreate and recover on F21 atomic host:
http://dustymabe.com/2015/01/31/crisis-averted-im-using-atomic-host/

keszybz · 2015-02-01T16:43:42Z

On Sun, Feb 01, 2015 at 08:32:25AM -0800, Dusty Mabe wrote:

I hit this using Fedora 21 atomic. @keszybz Thanks for fixing.. Is there a bz that tracks this issue that I can follow?
Nope, feel free to open one (preferably in the POST state if you can).

dustymabe · 2015-02-01T17:13:35Z

https://bugzilla.redhat.com/show_bug.cgi?id=1188037

@keszybz I could only open in "NEW" or "ASSIGNED". Do you mind updating to "POST" state?

ernetas · 2015-02-15T18:54:21Z

The issue was fixed in systemd-216-20.fc21.

Based on 7539904 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Zbigniew=20J=C4=99drzejewski-Szmek?= <zbyszek@in.waw.pl> Date: Mon, 5 Jan 2015 19:03:08 -0500 Subject: [PATCH] cgroup: memory limits on / are not supported Includes a3bd89e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Zbigniew=20J=C4=99drzejewski-Szmek?= <zbyszek@in.waw.pl> Date: Sat, 31 Jan 2015 23:03:25 -0500 Subject: [PATCH] core/cgroup: fix embarrassing typo moby/moby#10280

ernetas closed this as completed Feb 15, 2015

jfoy mentioned this issue Mar 5, 2015

Failed to set memory.limit_in_bytes on : Invalid argument coreos/bugs#123

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot start container <..>: write <..>memory.memsw.limit_in_bytes: invalid argument #10280

Cannot start container <..>: write <..>memory.memsw.limit_in_bytes: invalid argument #10280

ernetas commented Jan 22, 2015

estesp commented Jan 22, 2015

ernetas commented Jan 22, 2015

ernetas commented Jan 22, 2015

ernetas commented Jan 23, 2015

setiseta commented Jan 23, 2015

estesp commented Jan 23, 2015

ernetas commented Jan 23, 2015

ernetas commented Jan 24, 2015

ernetas commented Jan 26, 2015

crosbymichael commented Jan 26, 2015

ernetas commented Jan 27, 2015

estesp commented Jan 27, 2015

setiseta commented Jan 27, 2015

ernetas commented Jan 27, 2015

LK4D4 commented Jan 28, 2015

estesp commented Jan 28, 2015

ernetas commented Jan 28, 2015

keszybz commented Jan 31, 2015

ernetas commented Jan 31, 2015

keszybz commented Jan 31, 2015

setiseta commented Jan 31, 2015

keszybz commented Feb 1, 2015

dustymabe commented Feb 1, 2015

keszybz commented Feb 1, 2015

dustymabe commented Feb 1, 2015

ernetas commented Feb 15, 2015

Cannot start container <..>: write <..>memory.memsw.limit_in_bytes: invalid argument #10280

Cannot start container <..>: write <..>memory.memsw.limit_in_bytes: invalid argument #10280

Comments

ernetas commented Jan 22, 2015

estesp commented Jan 22, 2015

ernetas commented Jan 22, 2015

ernetas commented Jan 22, 2015

ernetas commented Jan 23, 2015

setiseta commented Jan 23, 2015

estesp commented Jan 23, 2015

ernetas commented Jan 23, 2015

ernetas commented Jan 24, 2015

ernetas commented Jan 26, 2015

crosbymichael commented Jan 26, 2015

ernetas commented Jan 27, 2015

estesp commented Jan 27, 2015

setiseta commented Jan 27, 2015

ernetas commented Jan 27, 2015

LK4D4 commented Jan 28, 2015

estesp commented Jan 28, 2015

ernetas commented Jan 28, 2015

keszybz commented Jan 31, 2015

ernetas commented Jan 31, 2015

keszybz commented Jan 31, 2015

setiseta commented Jan 31, 2015

keszybz commented Feb 1, 2015

dustymabe commented Feb 1, 2015

keszybz commented Feb 1, 2015

dustymabe commented Feb 1, 2015

ernetas commented Feb 15, 2015