Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

%xdel magic #419

Merged
merged 4 commits into from

2 participants

Thomas Kluyver Fernando Perez
Thomas Kluyver
Owner

Fernando suggested this on issue #141. The %xdel magic hunts out the references to an object in the IPython machinery, removing them so that the object can be released from memory. I'm fairly satisfied with the implementation, although please tell me if I've overlooked somewhere that we can keep references.

The trickiest bit of this was writing the test. We have to eliminate extra references kept by the testing framework (fair enough). Then it still doesn't work unless I print the test function's frame's local variables before checking that the object was released. I've no idea what difference that makes (and calling gc.collect() doesn't work instead), but I've put it in until we can find a better way to make the test work. Obviously please make sure that this passes on other systems.

I've also manually checked it interactively, which works without any extra trickery.

Thomas Kluyver
Owner

OK, I've found a better way to make the test pass now, by modifying __delitem__ for the user namespace the test suite uses.

Fernando Perez fperez commented on the diff
IPython/core/interactiveshell.py
((5 lines not shown))
+
+ def del_var(self, varname, by_name=False):
+ """Delete a variable from the various namespaces, so that, as
+ far as possible, we're not keeping any hidden references to it.
+
+ Parameters
+ ----------
+ varname : str
+ The name of the variable to delete.
+ by_name : bool
+ If True, delete variables with the given name in each
+ namespace. If False (default), find the variable in the user
+ namespace, and delete references to it.
+ """
+ if varname in ('__builtin__', '__builtins__'):
+ raise ValueError("Refusing to delete %s" % varname)
Fernando Perez Owner
fperez added a note

I'd change the message to:

"Refusing to delete %s from builtins"

so it's clearer to the user what happened.

Thomas Kluyver Owner

But this is in case of someone doing xdel __builtins__, not trying to delete something from the builtins.

At present, trying to xdel something in the builtins just gives NameError: name 'eval' is not defined (because it's looking in the user namespace). We could check for that case separately.

Fernando Perez Owner
fperez added a note

Ah, sorry. Then:

  • you're right, the message is OK for this case
  • yes, I think we should do what I thought the code was doing. Deleting stuff from builtins is just going to blow lots of things up, so let's not do it in xdel.

Other than that, good to go, thanks!

Thomas Kluyver Owner

Sorry, I should have been clearer: xdel will not delete things from the builtins - it doesn't even look at them. The question is simply whether we want toc check that case and provide a specific error message, rather than the general NameError message it gives for any variable not in user_ns.

Fernando Perez Owner
fperez added a note

Ah, OK. Then I wouldn't worry about it, leave it as-is and merge away, it's such a rare edge case it's not worth losing more sleep over it. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Fernando Perez
Owner

Other than my minor comments, this looks great, thanks! I ran the tests and did some more interactive checking, and the functionality looks exactly like what we wanted. Awesome.

Once you make those tiny fixes, merge away!

Thomas Kluyver takluyver merged commit e971e2f into from
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on May 6, 2011
  1. Thomas Kluyver

    Implement deletion mechanism to remove references held by IPython beh…

    takluyver authored
    …ind the scenes. Exposed to the user as %xdel.
  2. Thomas Kluyver
Commits on May 9, 2011
  1. Thomas Kluyver
Commits on May 11, 2011
  1. Thomas Kluyver
This page is out of date. Refresh to see the latest.
44 IPython/core/interactiveshell.py
View
@@ -1079,7 +1079,49 @@ def reset(self, new_session=True):
# Clear out the namespace from the last %run
self.new_main_mod()
-
+
+ def del_var(self, varname, by_name=False):
+ """Delete a variable from the various namespaces, so that, as
+ far as possible, we're not keeping any hidden references to it.
+
+ Parameters
+ ----------
+ varname : str
+ The name of the variable to delete.
+ by_name : bool
+ If True, delete variables with the given name in each
+ namespace. If False (default), find the variable in the user
+ namespace, and delete references to it.
+ """
+ if varname in ('__builtin__', '__builtins__'):
+ raise ValueError("Refusing to delete %s" % varname)
Fernando Perez Owner
fperez added a note

I'd change the message to:

"Refusing to delete %s from builtins"

so it's clearer to the user what happened.

Thomas Kluyver Owner

But this is in case of someone doing xdel __builtins__, not trying to delete something from the builtins.

At present, trying to xdel something in the builtins just gives NameError: name 'eval' is not defined (because it's looking in the user namespace). We could check for that case separately.

Fernando Perez Owner
fperez added a note

Ah, sorry. Then:

  • you're right, the message is OK for this case
  • yes, I think we should do what I thought the code was doing. Deleting stuff from builtins is just going to blow lots of things up, so let's not do it in xdel.

Other than that, good to go, thanks!

Thomas Kluyver Owner

Sorry, I should have been clearer: xdel will not delete things from the builtins - it doesn't even look at them. The question is simply whether we want toc check that case and provide a specific error message, rather than the general NameError message it gives for any variable not in user_ns.

Fernando Perez Owner
fperez added a note

Ah, OK. Then I wouldn't worry about it, leave it as-is and merge away, it's such a rare edge case it's not worth losing more sleep over it. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ ns_refs = self.ns_refs_table + [self.user_ns,
+ self.user_global_ns, self._user_main_module.__dict__] +\
+ self._main_ns_cache.values()
+
+ if by_name: # Delete by name
+ for ns in ns_refs:
+ try:
+ del ns[varname]
+ except KeyError:
+ pass
+ else: # Delete by object
+ try:
+ obj = self.user_ns[varname]
+ except KeyError:
+ raise NameError("name '%s' is not defined" % varname)
+ # Also check in output history
+ ns_refs.append(self.history_manager.output_hist)
+ for ns in ns_refs:
+ to_delete = [n for n, o in ns.iteritems() if o is obj]
+ for name in to_delete:
+ del ns[name]
+
+ # displayhook keeps extra references, but not in a dictionary
+ for name in ('_', '__', '___'):
+ if getattr(self.displayhook, name) is obj:
+ setattr(self.displayhook, name, None)
+
def reset_selective(self, regex=None):
"""Clear selective variables from internal namespaces based on a
specified regular expression.
19 IPython/core/magic.py
View
@@ -1081,7 +1081,24 @@ def magic_reset_selective(self, parameter_s=''):
raise TypeError('regex must be a string or compiled pattern')
for i in self.magic_who_ls():
if m.search(i):
- del(user_ns[i])
+ del(user_ns[i])
+
+ def magic_xdel(self, parameter_s=''):
+ """Delete a variable, trying to clear it from anywhere that
+ IPython's machinery has references to it. By default, this uses
+ the identity of the named object in the user namespace to remove
+ references held under other names. The object is also removed
+ from the output history.
+
+ Options
+ -n : Delete the specified name from all namespaces, without
+ checking their identity.
+ """
+ opts, varname = self.parse_options(parameter_s,'n')
+ try:
+ self.shell.del_var(varname, ('n' in opts))
+ except (NameError, ValueError) as e:
+ print type(e).__name__ +": "+ str(e)
def magic_logstart(self,parameter_s=''):
"""Start logging anywhere in a session.
22 IPython/core/tests/test_magic.py
View
@@ -394,6 +394,28 @@ def __repr__(self):
nt.assert_equal(monitor, [])
_ip.magic_reset("-f")
nt.assert_equal(monitor, [1])
+
+class TestXdel(tt.TempFileMixin):
+ def test_xdel(self):
+ """Test that references from %run are cleared by xdel."""
+ src = ("class A(object):\n"
+ " monitor = []\n"
+ " def __del__(self):\n"
+ " self.monitor.append(1)\n"
+ "a = A()\n")
+ self.mktmp(src)
+ # %run creates some hidden references...
+ _ip.magic("run %s" % self.fname)
+ # ... as does the displayhook.
+ _ip.run_cell("a")
+
+ monitor = _ip.user_ns["A"].monitor
+ nt.assert_equal(monitor, [])
+
+ _ip.magic("xdel a")
+
+ # Check that a's __del__ method has been called.
+ nt.assert_equal(monitor, [1])
def doctest_who():
"""doctest for %who
10 IPython/testing/globalipapp.py
View
@@ -132,6 +132,16 @@ def update(self,other):
# correct for that ourselves, to ensure consitency with the 'real'
# ipython.
self['__builtins__'] = __builtin__
+
+ def __delitem__(self, key):
+ """Part of the test suite checks that we can release all
+ references to an object. So we need to make sure that we're not
+ keeping a reference in _savedict."""
+ dict.__delitem__(self, key)
+ try:
+ del self._savedict[key]
+ except KeyError:
+ pass
def get_ipython():
Something went wrong with that request. Please try again.