Bad memory management in IPython notebook

[gdonval]

Hi,

I'm using IPython2/3 1.0.0-3 on archlinux.

Issue:

Each time I launch IPython (2 or 3) notebook,

• Each IPython instance eats 140 MB of memory (I insist, each instance).
• Each single notebook launched from these instances also eats 140 MB each.

Usecase:

Imagine I have 2 notebooks in one instance and another notebook in another IPython instance. These are gonna eat 700 MB of RAM before doing anything with them.

Expected behaviour:

• Each notebook should take a few MBs at most at launch (as a usual Python/IPython instance).
• The memory footprint of the IPython notebook should be much lower.

[minrk]

This is surprising, because I see IPython using only 50MB.

Some questions:

1. How are you getting the memory usage reports?
2. Do you see the same behavior with terminal IPython?
3. If not, do you see the same behavior with ipython kernel?
4. Have you actually opened a few notebooks and seen 700 MB taken up? Unfortunately, 5 processes reporting 100MB in use does not always mean 500MB are occupied.

[jdfreder]

Tested on Windows, I only see it using 25MB and an additional 19MB per notebook

[jdfreder]

And on Linux, I see 40MB and an additional 22MB (roughly the same)

[gdonval]

Same version?

1. I got the memory usage report through top. But I know what you are gonna say. So I just tested using free (-/+ buffers/cache):
   • 1 IPython instance: 1400 MB free
   • 1 IPython instance + 1 active notebook: 1275 MB free
   • 1 IPython instance + notebook shut down: 1407 MB free
   • IPython killed: 1542 MB free
2. No, both IPython and Python takes a couple of MB if launched without the notebook.
3. When I do as suggested, the free memory drops from 1549 to 1444 MB.
4. Yes. edit: I "discovered" the problem because OOM_killer has been launched.

[ahmadia]

What web browser are you using? What are the memory allocations if you
don't have a web browser open?

On Thu, Sep 12, 2013 at 11:58 AM, Gaël notifications@github.com wrote:

Same version?

1. I got the memory usage report through top. But I know what you are
   gonna say. So I just tested using free (-/+ buffers/cache):
   • 1 IPython instance: 1400 MB free
   • 1 IPython instance + 1 active notebook: 1275 MB free
   • 1 IPython instance + notebook shut down: 1407 MB free
   • IPython killed: 1542 MB free
     1. No, both IPython and Python takes a couple of MB if launched
        without the notebook.
2. When I do as suggested, the free memory drops from 1549 to _1444_MB.
3. Yes.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/4199#issuecomment-24337543
.

[gdonval]

I'm using chromium at the moment.

For the figure, I'm still using free (even though top reports similar results) but I'll do the subtractions this time. These results are cumulative (raw):

• Launching IPython notebook : 159 MB
• Opening 1 notebook : 302 MB (+143 MB)
• Closing the two chromium tabs: 254 MB (-48 MB)
• Killing IPython: 9 MB (-245 MB)

I did almost nothing with my system on the time being, but stuff always occurs in memory (hence the residual 9 MB).

[minrk]

Do you have any IPython configuration, or code run at startup in your profile?

[gdonval]

Ok, this morning, update to IPython 1.1.0-1.

Still the same problem.

I checked for config file there:

~:

• .ipythonrc
• .ipython
• .config/ipython <- found something and deleted it.

/etc/ipython

No change. I also tried to use a random profile just in case: ipython notebook --profile=foo.

I still have the same problem. I'd like to stress that this problem does not exist in console IPython.

[minrk]

What do you get for memory usage (in the terminal) after each of the following imports:

from tornado import web, ioloop
import jinja2
import zmq
from IPython import kernel
from IPython.html import notebookapp

[gdonval]

I repeated the measurements 4 times and calculated the average

ipython

15 MB (+15 MB)

from tornado import web, ioloop

18 MB (+3 MB)

import jinja2

20 MB (+2 MB)

import zmq

21 MB (+1 MB)

from IPython import kernel

124 MB (+103 MB)

from IPython.html import notebookapp

134 MB (+10 MB)

Do the python instances launched specifically for each notebook import kernel from IPython?

[takluyver]

On Sep 14, 2013 8:44 AM, "Gaël" notifications@github.com wrote:

Do the python instances launched specifically for each notebook import
kernel from IPython?

Certainly. Those instances are what we call kernels, so that package exists
to support them. If you have time, can you dig further into the imports
within that package to see what is using memory?

[gdonval]

Ok, I understand. I thought a kernel was an IPython instance to which notebooks were connected somehow.

I dug...

The offending import is:

from IPython.kernel.connect import *

or

from .connect import * 

as it is stated in []/IPython/kernel/init.py.

But I really need to dig a bit more: when I execute all the lines one after the other from connect.py in a python shell, the memory usage increases only by 3 MB. More about that tomorrow.

[minrk]

Intriguing. One more thing to try is a utility that's been giving us trouble lately:

from IPython.utils import localinterfaces

[gdonval]

No problem with this utility.

I also confirm that:

from IPython.kernel.connect import *

increases the memory consumption by ~100 MB while

%loadpy /usr/lib/python3.3/site-packages/IPython/kernel/connect.py

followed by a couple of hits on enter increases memory consumption by ~3 MB just like when I copied and pasted the content of the file manually. The resource module gives me comparable results.

Do you know any decent memory profiler that I could use to follow and spot memory?

[gdonval]

Ok, I got it: numpy

When kernel is imported, kernel.connect is imported which in turn imports zmq which imports zmq.session which imports zmq.serialize which imports IPython.utils.pickleutil which finally imports numpy.

I'm sorry, I should have checked that first before submitting something here. Especially since I'm using numpy on a daily basis and should have spotted that the memory leak was there.

Thanks for your time and for your help anyway. This is has been great to interact with such a helpful and reactive community! Again, sorry for the noise.

[Carreau]

Still I think @minrk, @takluyver (and me) have numpy installed and do not see this behavior. IT woudl at least be nice to understand why you see it.

[gdonval]

Yes of course. My point was that it does not seem to be directly related to IPython, so I closed the bug report. But I planned to report the source of the memory leak here also as you might be interested and as other people might also think that this is due to IPython.

[Carreau]

OK, great.
Thanks.

[gdonval]

I tracked down the memory leak to numpy.core.umath. Since this is C code, I stopped there for now.
I'll try to find the offending parts, but nothing guaranteed.

[takluyver]

I'd definitely recommend pinging the numpy guys about it - they may have more information on potential problems, or ideas for how to debug it.

Also, I'm not convinced that we should be importing numpy wherever it's available, as even normally importing numpy takes a certain amount of time and memory (about 7 MB, on my system).