Memory leak even when cache_size = 0 and history_length = 0 or history_length = 1

[cpcloud]

i'm using git master ipython, python 2.7.5, arch linux 64-bit

using the following code as a starting point

from pandas import DataFrame, concat
from numpy.random import randint, randn
from string import ascii_letters as letters
ncols = 16
nrows = 1e7
letters = list(letters)
df = DataFrame(randn(nrows, ncols), columns=letters[:ncols])
col = DataFrame(randint(1, size=nrows), columns=[letters[ncols]])
big = concat([df, col], axis=1)

if you now repeatedly evaluate big.values (by hand not in a loop) then memory keeps on growing (open htop or top to watch it in action). same is true when i've set the output cache to zero and the history length to 0 or 1.

furthermore this doesn't happen in vanilla python which has just _ for history and if i e.g., evaluate big.values n times then i need to execute n other statements e.g,. x = 1 n times to reclaim the memory.

see issue at pandas-dev/pandas#3629 for a long discussion about this. am i missing some feature/quirk of the history or caching system?

[takluyver]

history_length should be irrelevant here, that's for command history.

Looking into it now. Anyone else testing, you might want to set nrows down to 1e6 to avoid hitting swap.

[takluyver]

It is something related to the output cache - running %reset out releases the memory again.

[takluyver]

Ah, the last three outputs are cached as _, __, ___, even if you set the output cache size to zero. I think that's intentional, but I agree that it's confusing. Maybe we should disable those as well if the output cache is disabled, or just fall back to the standard Python _.

[cpcloud]

i like completely disabled because that's what i'm asking for when i say InteractiveShell.cache_size = 0, i.e., store nothing, less trickery to me that way, but maybe there's a reason to keep those.

[takluyver]

Do you want to have a look at IPython.core.displayhook.DisplayHook.update_user_ns() and prepare a pull request? I think there's a good chance it would be accepted.

[cpcloud]

where are the relevant tests?

[cpcloud]

ah i see them interactiveshelltestcase

[cpcloud]

do you guys use travis or some other CI that I can hook into?

[ivanov]

@cpcloud yes, if you make a PR it will place it in our Travis queue

[cpcloud]

ok cool thanks

[cpcloud]

is there a way to set config variables at runtime?

[ivanov]

In [1]: get_ipython().config
Out[1]: 
{'InteractiveShellApp': {'extensions': ['storemagic',
   'memory_profiler',
   'django_notebook']},
 'ProfileDir': {},
 'TerminalInteractiveShell': {'banner1': '', 'colors': 'LightBG'}}

## and you can set them just by just accessing new keys as though they are attributes
In [2]: get_ipython().config.TerminalInteractiveShell.foo = 1

In [3]: get_ipython().config
Out[3]: 
{'InteractiveShellApp': {'extensions': ['storemagic',
   'memory_profiler',
   'django_notebook']},
 'ProfileDir': {},
 'TerminalInteractiveShell': {'banner1': '', 'colors': 'LightBG', 'foo': 1}}

[takluyver]

There's also a %config magic, which should work like this:

%config TerminalInteractiveShell.cache_size = 0

[ellisonbg]

@takluyver and @cpcloud any progress on this one. I am going to bump it to 3.0. If you plan on getting to this in the next few days, feel free to bump it back to 2.0.

[wjakob]

I've just stumbled across this issue. Even with

%config TerminalInteractiveShell.cache_size = 0

Jupyter notebook will not free objects referenced in cells. For instance, when writing

A + B

where (A and B could be NumPy arrays or tensors on a GPU), Jupyter will hold on to A+B for the rest of the session even though the object should be freed right after printing/visualizing it.

This is not a problem when working with small objects, but it does get rather annoying in interactive sessions that involve big data on a resource-constrained device (GPU memory..)

[mohammad7t]

It seems that c.InteractiveShell.cache_size is not properly read from ~/.jupyter/jupyter_notebook_config.py. As a workaround, I could prevent the memory leak by running the following command in the initial cell of the notebook:

import IPython
IPython.get_ipython().displayhook.cache_size = 0

[ivanov]

@mhsekhavat: that setting is a kernel or standalone interactive console option and does not apply to to notebooks, so it's not surprising that jupyter_notebook_config.py does not affect it. You'll want to put that in a .py file inside the ~/.ipython/profile_default/startup/ folder (there's a README there to help you).

For @wjakob:

%config TerminalInteractiveShell.cache_size = 0

That specific setting works only when using ipython in a terminal. The one you want to use in the notebook is the one that that applies to kernels -

%config ZMQInteractiveShell.cache_size = 0

[NightMachinery]

None of the proposed solutions seem to do anything for me. I have created a reproduction notebook:

https://colab.research.google.com/drive/1UpqpMbb6fpCZFDXNZ-Q5i72aAqn8R2cI?usp=sharing