Kernel <3.8.0 panics on lxc-start with 1core/low memory VM #407
Comments
|
I managed to reproduce the issue on 2ee3db6, I need to update the test script to handle the old docker/dockerd scheme in order to go further |
|
Attaching screenshots from virtualbox console output. Unfortunately the output is incomplete. Steps to reproduce:
for i in $(seq 100); do docker run base echo hello world; doneThe command causing the crash will print the intended output ("hello world"), then crash before returning. Screenshot 1: visible immediately. Screenshot 2: appears 2 - 5 seconds after screenshot 1. Then every 3-5 seconds, it is re-printed. |
|
This is a blocker for 0.2. My best guess is some sort of interaction between aufs and lxc-start - maybe we unmount the rootfs too early for example? |
|
@creack can you share the exact steps to reproduce with maximum certainty? That way we can all help with debugging, by each trying different revisions. |
|
I pushed my script in contrib/crashTest.go You need to update the docket path and just 'go run crashTest.go' On Monday, April 15, 2013, Solomon Hykes wrote:
Guillaume J. Charmes |
|
Thanks. What is the current range of good / bad revisions that you On Mon, Apr 15, 2013 at 9:29 AM, Guillaume J. Charmes <
|
|
Things to try:
You mentioned that the problem happened on UP machines but not SMP. If |
|
Memory use increases after every docker run. It looks like aufs has some kind of problem or there's some other problem within the kernel. I've just tried the script posted above with 10000 runs and I was able to get 3.8.7 with aufs3 to start swapping with 3GB of RAM. Memory never got released after running this script, it just kept growing and growing forever. |
|
I installed a fresh ubuntu 13 with a kernel 3.8.0 and I wasn't able to reproduce (I let the script run for ~1h). |
|
After a lot of tests, I am pretty sure the leaks are due to #197 |
|
I've performed a few tests to try to reproduce this on 12.04 with stock kernels. docker was downloaded from docker.io to keep things simple memory after first test w/ 100 runs & before second test memory after the second test w/ 100 runs |
|
do you perform your test with @shykes command or from my script (in /contrib/crashTest.go) ? |
|
Getting the same issue, captured the output from VirtualBox here: https://gist.github.com/robknight/5430280 - it's pretty much the same thing reported by @shykes earlier. Running Docker inside the standard Vagrant box, OSX 10.8 host. For me this doesn't seem to have much to do with the length of time that the container is running for. I'm trying to build an image using docker-build, and my build succeeds maybe 25% of the time while the other 75% results in the above crash, after which the Vagrant box becomes unresponsive and has to be restarted. My docker-build changefile only has two lines: The file referenced here definitely exists, and the build does succeed sometimes. Identical behaviour occurs when using a different base image, e.g. centos. |
|
Also getting kernel panics running docker 0.1.5, 0.1.6, 0.1.7 on Ubuntu 12.10, Linux 3.5.0-27, bare metal Dell Latitude D830 w/ Intel Core 2 Duo and 4GB RAM. Reproduced by running the example multiple (<20) times: Screen photos (docker 0.1.7): |
|
It seems that for the time being Docker requires Linux >3.8. This is unfortunate, but it seems earlier versions just can't handle spawning too many short-lived namespaced processes. And we couldn't pinpoint the exact change which caused the bug to strike more frequently... Docker now issues a warning on Linux kernels <3.8. |
|
The screenshot posted by @barryaustin shows that it's exactly the same problem with bare metal. That's very useful, because it rules out many potential side effects caused by virtualization. Are we sure we want to close this issue? People running Ubuntu in production will very probably run 12.04 LTS rather than 12.10 or 13.04, and 12.04 LTS might not be supporting 3.8 ever. |
|
I don't mind keeping it open, but that would imply that there's something On Tue, Apr 23, 2013 at 10:03 AM, Jérôme Petazzoni <notifications@github.com
|
|
My plan would look like this:
On Tue, Apr 23, 2013 at 10:08 AM, Solomon Hykes notifications@github.comwrote:
|
|
I agree this would be great. I re-opened the issue and removed it from 0.2. Want to lead the charge? Let me know and I'll assign to you. |
|
Per @creack requests here what happens for me on https://gist.github.com/lopter/5449001#file-dmesg-log I'm running Docker in daemon mode on Ubuntu 12.04 in Virtualbox: |
|
I've just tried the exact same setup as @lopter. It didn't crash at all, not even with CPU limit set to 40%. However, I was able to make the system leak memory when the memory cgroup didn't get mounted. That seems to happen once every other boot on ubuntu in virtualbox. This didn't seem to break the system even after running the script @shykes posted above with 10000 runs. I've used the precise64 box. I've updated the system to use the latest kernel (3.2.0.40). |
|
It locked up by the time it reached the 1355th run with the script posted by @shykes. |
|
As another data point, I just ran the crashTest.go script on a Debian Wheezy VM running under Virtualbox on a dual-core i7 Macbook Air: The script has been running for an hour without crashing, and has done just over 10,000 runs. Memory usage remained constant throughout (with between 4 and 6MB free out of 250MB the whole time). I think this means either:
I hope some progress can be made on this issue. One of the things that I like about Docker is how easy it is to get started, requiring a 3.8 kernel makes that much harder in many environments. |
|
Broadening kernel support is going to be a major priority for us in June. I'm also frustrated by the current kernel situation! On Wed, May 22, 2013 at 7:09 PM, Paul Hammond notifications@github.com
|
|
Hi guys, sorry I wasn't aware of this issue when I did my write up for OStatic. I'll update the post with a link back here. I experienced what looks to be this bug on Ubuntu 12.04 LTS, 64bit. |
|
I had something very similar in the past and was related to the hardware/kernel pair. Please test it on different CPU families if you can to see correlations. |
|
I'm closing this since it's not immediately actionable. |
|
For the record, Josh Poimboeuf found something which might be related to this:
|
Allow the user to set DOCKER_NOWARN_KERNEL_VERSION=1 to disable the warning for RHEL 6.5 and other distributions that don't exhibit the panics described in moby#407.
|
can we keep this open? still verified with 0.11 and kernel 3.2.0 (2 processors) |
|
@gdm85 No, this will stay closed. Kernels older than 3.8 aren't supported. That means technical support isn't provided and you might run into unexpected behavior, even if it seems like it's working. The only exception is the kernel provided by RHEL6 (2.6.32xxxxxx) which was patched and improved to work properly with Docker. Docker on kernels older than 3.8 won't happen. Please upgrade your kernel. |
Allow the user to set DOCKER_NOWARN_KERNEL_VERSION=1 to disable the warning for RHEL 6.5 and other distributions that don't exhibit the panics described in moby/moby#407.
ghost
mentioned this issue
|
I know this is an old bug, has any one tried to reproduce this using the VFS driver instead of aufs ? |
…ainer_error [19.03 backport] Propagate GetContainer error from event processor
…9.03-6eeb9ec3d6517883995b327b2e92b12a3cfc3074 [19.03] sync to upstream 19.03 4bed012
The issue does not occur on a 8 core machine.
The lxc-start process causes a kernel panic while waiting for the child process to return.
It has something to do with the unmount/lock/namespaces. The output of the panic is difficult to catch.
On the last version, this happen after 5 hello world...
On older version, it takes more time.
I'll push a script that reproduce the issue.
The text was updated successfully, but these errors were encountered: