Kernel <3.8.0 panics on lxc-start with 1core/low memory VM

[creack]

The issue does not occur on a 8 core machine.

The lxc-start process causes a kernel panic while waiting for the child process to return.

It has something to do with the unmount/lock/namespaces. The output of the panic is difficult to catch.

On the last version, this happen after 5 hello world...
On older version, it takes more time.

I'll push a script that reproduce the issue.

[creack]

I managed to reproduce the issue on 2ee3db6, I need to update the test script to handle the old docker/dockerd scheme in order to go further

[shykes]

Attaching screenshots from virtualbox console output. Unfortunately the output is incomplete.

Steps to reproduce:

1. Run docker in daemon mode

for i in $(seq 100); do docker run base echo hello world; done

The command causing the crash will print the intended output ("hello world"), then crash before returning.

Screenshot 1: visible immediately.

Screenshot 2: appears 2 - 5 seconds after screenshot 1. Then every 3-5 seconds, it is re-printed.

[shykes]

This is a blocker for 0.2.

My best guess is some sort of interaction between aufs and lxc-start - maybe we unmount the rootfs too early for example?

[shykes]

@creack can you share the exact steps to reproduce with maximum certainty? That way we can all help with debugging, by each trying different revisions.

[creack]

I pushed my script in contrib/crashTest.go

You need to update the docket path and just 'go run crashTest.go'

On Monday, April 15, 2013, Solomon Hykes wrote:

@creack https://github.com/creack can you share the exact steps to
reproduce with maximum certainty? That way we can all help with debugging,
by each trying different revisions.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/407#issuecomment-16395640
.

Guillaume J. Charmes

[shykes]

Thanks. What is the current range of good / bad revisions that you
identified?

On Mon, Apr 15, 2013 at 9:29 AM, Guillaume J. Charmes <
notifications@github.com> wrote:

I pushed my script in contrib/crashTest.go

You need to update the docket path and just 'go run crashTest.go'

On Monday, April 15, 2013, Solomon Hykes wrote:

@creack https://github.com/creack can you share the exact steps to
reproduce with maximum certainty? That way we can all help with
debugging,
by each trying different revisions.

—
Reply to this email directly or view it on GitHub<
https://github.com/dotcloud/docker/issues/407#issuecomment-16395640>
.

Guillaume J. Charmes

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/407#issuecomment-16395821
.

[jpetazzo]

Things to try:

• reproduce with hardened kernel (s3://
  get.docker.io/kernels/linux-headers-3.2.40-grsec-dotcloud_42~tethys_amd64.deb
  )
• reproduce in such a way that we actually get the full backtrace (e.g. in
  a Xen VM on our test machines at the office :-))
• if the problem can be triggered in a Xen VM, extract the backtrace of the
  kernel (starting point: xenctx)

You mentioned that the problem happened on UP machines but not SMP. If
that's indeed the case, try with 1 core but with SMP code anyway (IIRC,
kernel option noreplace-smp).

[unclejack]

Memory use increases after every docker run. It looks like aufs has some kind of problem or there's some other problem within the kernel.

I've just tried the script posted above with 10000 runs and I was able to get 3.8.7 with aufs3 to start swapping with 3GB of RAM. Memory never got released after running this script, it just kept growing and growing forever.

[creack]

I installed a fresh ubuntu 13 with a kernel 3.8.0 and I wasn't able to reproduce (I let the script run for ~1h).
However, as @unclejack said, it leaks.

[creack]

After a lot of tests, I am pretty sure the leaks are due to #197

[unclejack]

I've performed a few tests to try to reproduce this on 12.04 with stock kernels.
It didn't crash, nor leak.

docker was downloaded from docker.io to keep things simple

docker version
Version:0.1.4
Git Commit:

uname -rv
3.2.0-40-generic #64-Ubuntu SMP Mon Mar 25 21:22:10 UTC 2013

cat /proc/cpuinfo | grep processor
processor       : 0

cat /proc/meminfo | grep Total
MemTotal:         496260 kB
SwapTotal:             0 kB

memory after first test w/ 100 runs & before second test

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0      0 333512  29196  54776    0    0   214    38   41  213  3  2 94  1

memory after the second test w/ 100 runs

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0      0 324360  30968  56548    0    0   182    46   47  283  4  2 93  1

[creack]

do you perform your test with @shykes command or from my script (in /contrib/crashTest.go) ?
What version of the kernel and lxc are you using?

[unclejack]

@creack I was trying the command @shykes has posted earlier.

I'll try your crashTest script as well.

lxc is the standard one from Ubuntu 12.04.