"Memory leak" do to InteractiveShell.user_hidden_ns #10082

Open
Carreau opened this Issue Dec 7, 2016 · 8 comments

Projects

None yet

3 participants

@Carreau
Member
Carreau commented Dec 7, 2016

See following example:

In [1]: import os, sys, gc, weakref, numpy as np ; os.getpid()
Out[1]: 18719

In [2]: a = np.random.uniform( size=(16384,16384) )
# now use 2G of RAM

In [3]: w = weakref.ref(a) ; ip = get_ipython()
In [4]: a
Out[4]:
array([[ 0.80478205,  0.74083789,  0.68495306, ...,  0.63492685,
         0.36629052,  0.42683515],
       [ 0.7036597 ,  0.55800643,  0.84709347, ...,  0.33876275,
         0.50162708,  0.47421922],
       [ 0.69361104,  0.49171712,  0.32954206, ...,  0.62798451,
         0.93786031,  0.11439703],
       ...,
       [ 0.13777525,  0.66088395,  0.23408103, ...,  0.95339984,
         0.58267448,  0.63239088],
       [ 0.69385579,  0.80978902,  0.95708021, ...,  0.77515066,
         0.21577989,  0.00898049],
       [ 0.18237753,  0.20890975,  0.40356751, ...,  0.15583932,
         0.65681592,  0.64120292]])

In [5]: del a
   ...:

In [6]: del Out[4]; del _4

# still 2 GB of ram.... what's the last reference ? 

In [7]: guilty = {k for k,v in ip.user_ns_hidden.items() if w() is v}

In [8]: guilty
Out[8]: {'_4', '__'}

In [9]: for c in guilty:
    ...:     del ip.user_ns_hidden[c]

In [10]: gc.collect() # should be deleted if you do something else anyway.
Out[10]: 189

# yeah, back to reasonable amount. 

I think we do inject '_%s' % prompt_number in __main__ so we should likely not add them here.
Or, have del _N also remove from here.

(_, __, ___) will go away at some point, but not _4. We can also make this dict use weakrefs, maybe.

@takluyver
Member

Does %xdel a remove it? Once we have displayed something, you need our machinery to get rid of it.

@jboucasier
jboucasier commented Dec 7, 2016 edited

Yes it does. If I do a del a and after a %xdel _4, it works too.

(I am the guy who ask Carreau by mail about this behavior)

@Carreau
Member
Carreau commented Dec 7, 2016

It's still unclear to my why we keep track of _N in the hidden namespace. for _,__,___ it make sens as we shift them around. But I believe for _N it's not necessary. So I would lean toward this being a bug.

@takluyver
Member

The hidden namespace means things don't show up when you do %whos. I looked into making it a set years ago, but there was some issue - possibly that if you explicitly set a variable to a value, you want it to be visible, even if the name would normally be hidden.

@Carreau
Member
Carreau commented Dec 8, 2016

The hidden namespace means things don't show up when you do %whos. I looked into making it a set years ago, but there was some issue - possibly that if you explicitly set a variable to a value, you want it to be visible, even if the name would normally be hidden.

Well I get it's there to hide it. But if it's also in user_ns then user_ns_hidden would be a weakref dict, right ? is (AFAICU) still work, but don't keep object around if they get deleted

@takluyver
Member

I don't at the moment see any reason why a weakrefdict wouldn't work, but we should nonetheless check fairly carefully if you want to implement it.

@Carreau
Member
Carreau commented Dec 8, 2016

Hum... you can't weakref core objects like dicts and list (or actually any object that cannot define a __weakref__ attribute AFAICT from stackoverflow).

That's... problematic. That would require a bigger refactor.

I can see us "faking" a weakref by actually keeping an actual ref (that, we can) and for the pool of non-weakrefable objects, drop them when the count is 1. We can do that on every dict update.

The other annoying thing is that there are no method to check if an object is weakrefable. It just throw a TypeError.

@takluyver
Member

I can see us "faking" a weakref

I can see us dealing with weird, confusing bugs. ;-) I think the cure's worse than the disease here. We've long recommended using %xdel if you need to expunge all of IPython's internal references to an object.

@takluyver takluyver added this to the no action milestone Jan 12, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment