Skip to content
This repository

Update deepreload to use a rewritten knee.py. Fixes dreload(numpy). #1457

Merged
merged 7 commits into from about 2 years ago

2 participants

Bradley M. Froehle Fernando Perez
Bradley M. Froehle
Collaborator

knee.py, a Python re-implementation of hierarchical module import was
removed from the standard library because it no longer functioned properly.

deepreload.py is little more than a hacked version of knee.py which overrides
__builtin__.__import__ to ensure that each module is re-imported once (before
just referring to sys.modules as usual).

In addition, os.path was added to the default excluded modules, since
somehow it has an entry in sys.modules without `os' being a package.

I've posted my original re-written version of knee.py at https://gist.github.com/1944819. This would close gh-32.

Fernando Perez
Owner

@bfroehle, I'm really sorry for the long delay in reviewing this, but March was insane with travel and conferences. This is absolutely fantastic, thanks!!!

I'd only suggest that you also add a simple test that does import numpy; dreload(numpy), which would fail with current master and pass with your changes. That can be protected with a decorator so it only runs if people have numpy installed, we have such decorators already available, so you just have to do

from IPython.testing import decorators as dec
@dec.skipif_not_numpy
def test_deepreload():
  import numpy
  dreload(numpy)
added some commits February 29, 2012
Bradley M. Froehle Update deepreload to use a rewritten knee.py. Fixes dreload(numpy).
knee.py, a Python re-implementation of hierarchical module import was
removed from the standard library because it no longer functioned properly.

deepreload.py is little more than a hacked version of knee.py which overrides
__builtin__.__import__ to ensure that each module is re-imported once (before
just referring to sys.modules as usual).

In addition, `os.path` was added to the default excluded modules, since
somehow it has an entry in sys.modules without `os' being a package.
c0ad013
Bradley M. Froehle Keep original function names. 47c768c
Bradley M. Froehle Add deepreload unit test.
Thanks to Fernando for the suggestion and starting code.
b0f53e4
Bradley M. Froehle
Collaborator

Thanks for the unit test suggestion. The code provided worked, except that I needed to add 'unittest' to the list of excluded modules to prevent an error.

I suppose there should be a way to test that the deep reloading works, e.g.

def test_deepreload():
    import something_which_imports_string as m # What to choose here?
    x = string.Template('')

    dreload(m, exclude=['string'])
    nt.assert_is_instance(x, string.Template)

    dreload(m)
    nt.assert_not_is_instance(x, string.Template)
Fernando Perez
Owner

Yes, that would be possible, but I consider that to be above and beyond the expected call of duty here :) You'd have to create dynamically two modules in temporary files, modify the 2nd by rewriting the file, calling dreload() and then checking that the new values are correct. It's not really difficult, just some extra work. If you want to do that let me know and I'll wait, otherwise I think it's OK to proceed with merging.

Bradley M. Froehle
Collaborator

The unit test need not be that complicated. As I alluded to above, we could check verify that the module was reloaded by seeing if an instance of a class of the original module is no longer an instance of the class of the reloaded module.

--- A.py ---
class Object(object):
    pass

--- B.py ---
import A

--- test_deepreload.py ---
import nose.tools as nt
def test_deepreload():
    import A
    import B # B imports A
    obj = A.Object()

    dreload(B, exclude=['A'])
    nt.assert_is_instance(obj, A.Object)

    dreload(B)
    nt.assert_not_is_instance(obj, A.Object)

However I'm not convinced that it's worth creating this elaborate framework (of several modules) for this one test. So I think we can just go ahead with the merge.

Fernando Perez
Owner

good point. Since you have it basically written already (and making those two files just requires this code:

import tempfile
tmp = tempfile.mkdtemp()
import sys
from os.path import join
sys.path.insert(0, tmp)
with open(join(tmp, 'A.py'), 'w') as f:
    f.write('class Object(object):\n    pass\n')
with open(join(tmp, 'B.py'), 'w') as f:
    f.write('import A')
try:
    # do rest of test here
    import A # etc...
finally:
    import shutil
    shutil.rmtree(tmp)

I guess we might as well put it in :) Do you want to go ahead and finish it? It will mean that this code at least has one real test for it, which does help.

Fernando Perez
Owner

ps - the above is a quick draft, the stdlib imports should go atop the main file, as usual

Bradley M. Froehle
Collaborator

Okay, I think this is pretty much done. Do I need to cleanup the addition to sys.path? Other IPython tests are pretty inconsistent in this manner, some doing it, and others not.

Fernando Perez
Owner

Sorry, forgot to do that. Yes, it's probably a good idea to do undo the sys.path change. We should probably just have a context manager for that:

with temp_sys_path():
  sys.path.whatever()
  ...
# back to normal sys.path

but don't worry about it, just undo it with a sys.path.pop(0) in the finally clause.

added some commits April 17, 2012
Bradley M. Froehle Fix deepreload tests in Python 2.6.
Apparently assert_is_instance and assert_not_is_instance are not available
in Python 2.6.
807f609
Bradley M. Froehle Clean up sys.path entry. 2e6a444
Bradley M. Froehle
Collaborator

Apparently such a context manager exists: IPython.utils.syspathcontext.prepended_to_syspath. Unfortunately we cannot use a more compact syntax since we need Python 2.6 compatibility:

with TemporaryDirectory() as tmpdir, prepended_to_syspath(tmpdir):
    ...
Fernando Perez
Owner

Looks great, thanks! Merging now. Thanks a ton for your patience despite my delayed reply...

Fernando Perez fperez merged commit a206ee2 into from April 17, 2012
Fernando Perez fperez closed this April 17, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 7 unique commits by 1 author.

Apr 17, 2012
Bradley M. Froehle Update deepreload to use a rewritten knee.py. Fixes dreload(numpy).
knee.py, a Python re-implementation of hierarchical module import was
removed from the standard library because it no longer functioned properly.

deepreload.py is little more than a hacked version of knee.py which overrides
__builtin__.__import__ to ensure that each module is re-imported once (before
just referring to sys.modules as usual).

In addition, `os.path` was added to the default excluded modules, since
somehow it has an entry in sys.modules without `os' being a package.
c0ad013
Bradley M. Froehle Keep original function names. 47c768c
Bradley M. Froehle Add deepreload unit test.
Thanks to Fernando for the suggestion and starting code.
b0f53e4
Bradley M. Froehle Reformat test to a standard style. 27ee275
Bradley M. Froehle Add deepreload functionality test. 45df6ff
Bradley M. Froehle Fix deepreload tests in Python 2.6.
Apparently assert_is_instance and assert_not_is_instance are not available
in Python 2.6.
807f609
Bradley M. Froehle Clean up sys.path entry. 2e6a444
This page is out of date. Refresh to see the latest.
360  IPython/lib/deepreload.py
@@ -14,9 +14,9 @@
14 14
 
15 15
     __builtin__.dreload = deepreload.reload
16 16
 
17  
-This code is almost entirely based on knee.py from the standard library.
  17
+This code is almost entirely based on knee.py, which is a Python
  18
+re-implementation of hierarchical module import.
18 19
 """
19  
-
20 20
 #*****************************************************************************
21 21
 #       Copyright (C) 2001 Nathaniel Gray <n8gray@caltech.edu>
22 22
 #
@@ -28,135 +28,267 @@
28 28
 import imp
29 29
 import sys
30 30
 
31  
-# Replacement for __import__()
32  
-def deep_import_hook(name, globals=None, locals=None, fromlist=None, level=-1):
33  
-    # For now level is ignored, it's just there to prevent crash
34  
-    # with from __future__ import absolute_import
35  
-    parent = determine_parent(globals)
36  
-    q, tail = find_head_package(parent, name)
37  
-    m = load_tail(q, tail)
38  
-    if not fromlist:
39  
-        return q
40  
-    if hasattr(m, "__path__"):
41  
-        ensure_fromlist(m, fromlist)
42  
-    return m
  31
+from types import ModuleType
  32
+from warnings import warn
  33
+
  34
+def get_parent(globals, level):
  35
+    """
  36
+    parent, name = get_parent(globals, level)
  37
+
  38
+    Return the package that an import is being performed in.  If globals comes
  39
+    from the module foo.bar.bat (not itself a package), this returns the
  40
+    sys.modules entry for foo.bar.  If globals is from a package's __init__.py,
  41
+    the package's entry in sys.modules is returned.
  42
+
  43
+    If globals doesn't come from a package or a module in a package, or a
  44
+    corresponding entry is not found in sys.modules, None is returned.
  45
+    """
  46
+    orig_level = level
  47
+
  48
+    if not level or not isinstance(globals, dict):
  49
+        return None, ''
43 50
 
44  
-def determine_parent(globals):
45  
-    if not globals or  not globals.has_key("__name__"):
46  
-        return None
47  
-    pname = globals['__name__']
48  
-    if globals.has_key("__path__"):
49  
-        parent = sys.modules[pname]
50  
-        assert globals is parent.__dict__
51  
-        return parent
52  
-    if '.' in pname:
53  
-        i = pname.rfind('.')
54  
-        pname = pname[:i]
55  
-        parent = sys.modules[pname]
56  
-        assert parent.__name__ == pname
57  
-        return parent
58  
-    return None
59  
-
60  
-def find_head_package(parent, name):
61  
-    # Import the first
62  
-    if '.' in name:
63  
-        # 'some.nested.package' -> head = 'some', tail = 'nested.package'
64  
-        i = name.find('.')
65  
-        head = name[:i]
66  
-        tail = name[i+1:]
  51
+    pkgname = globals.get('__package__', None)
  52
+
  53
+    if pkgname is not None:
  54
+        # __package__ is set, so use it
  55
+        if not hasattr(pkgname, 'rindex'):
  56
+            raise ValueError('__package__ set to non-string')
  57
+        if len(pkgname) == 0:
  58
+            if level > 0:
  59
+                raise ValueError('Attempted relative import in non-package')
  60
+            return None, ''
  61
+        name = pkgname
67 62
     else:
68  
-        # 'packagename' -> head = 'packagename', tail = ''
69  
-        head = name
70  
-        tail = ""
71  
-    if parent:
72  
-        # If this is a subpackage then qname = parent's name + head
73  
-        qname = "%s.%s" % (parent.__name__, head)
  63
+        # __package__ not set, so figure it out and set it
  64
+        if '__name__' not in globals:
  65
+            return None, ''
  66
+        modname = globals['__name__']
  67
+
  68
+        if '__path__' in globals:
  69
+            # __path__ is set, so modname is already the package name
  70
+            globals['__package__'] = name = modname
  71
+        else:
  72
+            # Normal module, so work out the package name if any
  73
+            lastdot = modname.rfind('.')
  74
+            if lastdot < 0 and level > 0:
  75
+                raise ValueError("Attempted relative import in non-package")
  76
+            if lastdot < 0:
  77
+                globals['__package__'] = None
  78
+                return None, ''
  79
+            globals['__package__'] = name = modname[:lastdot]
  80
+
  81
+    dot = len(name)
  82
+    for x in xrange(level, 1, -1):
  83
+        try:
  84
+            dot = name.rindex('.', 0, dot)
  85
+        except ValueError:
  86
+            raise ValueError("attempted relative import beyond top-level "
  87
+                             "package")
  88
+    name = name[:dot]
  89
+
  90
+    try:
  91
+        parent = sys.modules[name]
  92
+    except:
  93
+        if orig_level < 1:
  94
+            warn("Parent module '%.200s' not found while handling absolute "
  95
+                 "import" % name)
  96
+            parent = None
  97
+        else:
  98
+            raise SystemError("Parent module '%.200s' not loaded, cannot "
  99
+                              "perform relative import" % name)
  100
+
  101
+    # We expect, but can't guarantee, if parent != None, that:
  102
+    # - parent.__name__ == name
  103
+    # - parent.__dict__ is globals
  104
+    # If this is violated...  Who cares?
  105
+    return parent, name
  106
+
  107
+def load_next(mod, altmod, name, buf):
  108
+    """
  109
+    mod, name, buf = load_next(mod, altmod, name, buf)
  110
+
  111
+    altmod is either None or same as mod
  112
+    """
  113
+
  114
+    if len(name) == 0:
  115
+        # completely empty module name should only happen in
  116
+        # 'from . import' (or '__import__("")')
  117
+        return mod, None, buf
  118
+
  119
+    dot = name.find('.')
  120
+    if dot == 0:
  121
+        raise ValueError('Empty module name')
  122
+
  123
+    if dot < 0:
  124
+        subname = name
  125
+        next = None
74 126
     else:
75  
-        qname = head
76  
-    q = import_module(head, qname, parent)
77  
-    if q: return q, tail
78  
-    if parent:
79  
-        qname = head
80  
-        parent = None
81  
-        q = import_module(head, qname, parent)
82  
-        if q: return q, tail
83  
-    raise ImportError, "No module named " + qname
84  
-
85  
-def load_tail(q, tail):
86  
-    m = q
87  
-    while tail:
88  
-        i = tail.find('.')
89  
-        if i < 0: i = len(tail)
90  
-        head, tail = tail[:i], tail[i+1:]
91  
-
92  
-        # fperez: fix dotted.name reloading failures by changing:
93  
-        #mname = "%s.%s" % (m.__name__, head)
94  
-        # to:
95  
-        mname = m.__name__
96  
-        # This needs more testing!!! (I don't understand this module too well)
97  
-
98  
-        #print '** head,tail=|%s|->|%s|, mname=|%s|' % (head,tail,mname)  # dbg
99  
-        m = import_module(head, mname, m)
100  
-        if not m:
101  
-            raise ImportError, "No module named " + mname
102  
-    return m
  127
+        subname = name[:dot]
  128
+        next = name[dot+1:]
  129
+
  130
+    if buf != '':
  131
+        buf += '.'
  132
+    buf += subname
  133
+
  134
+    result = import_submodule(mod, subname, buf)
  135
+    if result is None and mod != altmod:
  136
+        result = import_submodule(altmod, subname, subname)
  137
+        if result is not None:
  138
+            buf = subname
  139
+
  140
+    if result is None:
  141
+        raise ImportError("No module named %.200s" % name)
103 142
 
104  
-def ensure_fromlist(m, fromlist, recursive=0):
105  
-    for sub in fromlist:
106  
-        if sub == "*":
107  
-            if not recursive:
108  
-                try:
109  
-                    all = m.__all__
110  
-                except AttributeError:
111  
-                    pass
112  
-                else:
113  
-                    ensure_fromlist(m, all, 1)
114  
-            continue
115  
-        if sub != "*" and not hasattr(m, sub):
116  
-            subname = "%s.%s" % (m.__name__, sub)
117  
-            submod = import_module(sub, subname, m)
118  
-            if not submod:
119  
-                raise ImportError, "No module named " + subname
  143
+    return result, next, buf
120 144
 
121 145
 # Need to keep track of what we've already reloaded to prevent cyclic evil
122 146
 found_now = {}
123 147
 
124  
-def import_module(partname, fqname, parent):
  148
+def import_submodule(mod, subname, fullname):
  149
+    """m = import_submodule(mod, subname, fullname)"""
  150
+    # Require:
  151
+    # if mod == None: subname == fullname
  152
+    # else: mod.__name__ + "." + subname == fullname
  153
+
125 154
     global found_now
126  
-    if found_now.has_key(fqname):
  155
+    if fullname in found_now and fullname in sys.modules:
  156
+        m = sys.modules[fullname]
  157
+    else:
  158
+        print 'Reloading', fullname
  159
+        found_now[fullname] = 1
  160
+        oldm = sys.modules.get(fullname, None)
  161
+
  162
+        if mod is None:
  163
+            path = None
  164
+        elif hasattr(mod, '__path__'):
  165
+            path = mod.__path__
  166
+        else:
  167
+            return None
  168
+
127 169
         try:
128  
-            return sys.modules[fqname]
129  
-        except KeyError:
130  
-            pass
  170
+            fp, filename, stuff  = imp.find_module(subname, path)
  171
+        except ImportError:
  172
+            return None
  173
+
  174
+        try:
  175
+            m = imp.load_module(fullname, fp, filename, stuff)
  176
+        except:
  177
+            # load_module probably removed name from modules because of
  178
+            # the error.  Put back the original module object.
  179
+            if oldm:
  180
+                sys.modules[fullname] = oldm
  181
+            raise
  182
+        finally:
  183
+            if fp: fp.close()
  184
+
  185
+        add_submodule(mod, m, fullname, subname)
  186
+
  187
+    return m
  188
+
  189
+def add_submodule(mod, submod, fullname, subname):
  190
+    """mod.{subname} = submod"""
  191
+    if mod is None:
  192
+        return #Nothing to do here.
  193
+
  194
+    if submod is None:
  195
+        submod = sys.modules[fullname]
  196
+
  197
+    setattr(mod, subname, submod)
  198
+
  199
+    return
  200
+
  201
+def ensure_fromlist(mod, fromlist, buf, recursive):
  202
+    """Handle 'from module import a, b, c' imports."""
  203
+    if not hasattr(mod, '__path__'):
  204
+        return
  205
+    for item in fromlist:
  206
+        if not hasattr(item, 'rindex'):
  207
+            raise TypeError("Item in ``from list'' not a string")
  208
+        if item == '*':
  209
+            if recursive:
  210
+                continue # avoid endless recursion
  211
+            try:
  212
+                all = mod.__all__
  213
+            except AttributeError:
  214
+                pass
  215
+            else:
  216
+                ret = ensure_fromlist(mod, all, buf, 1)
  217
+                if not ret:
  218
+                    return 0
  219
+        elif not hasattr(mod, item):
  220
+            import_submodule(mod, item, buf + '.' + item)
  221
+
  222
+def deep_import_hook(name, globals=None, locals=None, fromlist=None, level=-1):
  223
+    """Replacement for __import__()"""
  224
+    parent, buf = get_parent(globals, level)
  225
+
  226
+    head, name, buf = load_next(parent, None if level < 0 else parent, name, buf)
  227
+
  228
+    tail = head
  229
+    while name:
  230
+        tail, name, buf = load_next(tail, tail, name, buf)
  231
+
  232
+    # If tail is None, both get_parent and load_next found
  233
+    # an empty module name: someone called __import__("") or
  234
+    # doctored faulty bytecode
  235
+    if tail is None:
  236
+        raise ValueError('Empty module name')
131 237
 
132  
-    print 'Reloading', fqname #, sys.excepthook is sys.__excepthook__, \
133  
-            #sys.displayhook is sys.__displayhook__
  238
+    if not fromlist:
  239
+        return head
  240
+
  241
+    ensure_fromlist(tail, fromlist, buf, 0)
  242
+    return tail
  243
+
  244
+modules_reloading = {}
  245
+
  246
+def deep_reload_hook(m):
  247
+    """Replacement for reload()."""
  248
+    if not isinstance(m, ModuleType):
  249
+        raise TypeError("reload() argument must be module")
  250
+
  251
+    name = m.__name__
  252
+
  253
+    if name not in sys.modules:
  254
+        raise ImportError("reload(): module %.200s not in sys.modules" % name)
  255
+
  256
+    global modules_reloading
  257
+    try:
  258
+        return modules_reloading[name]
  259
+    except:
  260
+        modules_reloading[name] = m
  261
+
  262
+    dot = name.rfind('.')
  263
+    if dot < 0:
  264
+        subname = name
  265
+        path = None
  266
+    else:
  267
+        try:
  268
+            parent = sys.modules[name[:dot]]
  269
+        except KeyError:
  270
+            modules_reloading.clear()
  271
+            raise ImportError("reload(): parent %.200s not in sys.modules" % name[:dot])
  272
+        subname = name[dot+1:]
  273
+        path = getattr(parent, "__path__", None)
134 274
 
135  
-    found_now[fqname] = 1
136 275
     try:
137  
-        fp, pathname, stuff = imp.find_module(partname,
138  
-                                              parent and parent.__path__)
139  
-    except ImportError:
140  
-        return None
  276
+        fp, filename, stuff  = imp.find_module(subname, path)
  277
+    finally:
  278
+        modules_reloading.clear()
141 279
 
142 280
     try:
143  
-        m = imp.load_module(fqname, fp, pathname, stuff)
  281
+        newm = imp.load_module(name, fp, filename, stuff)
  282
+    except:
  283
+         # load_module probably removed name from modules because of
  284
+         # the error.  Put back the original module object.
  285
+        sys.modules[name] = m
  286
+        raise
144 287
     finally:
145 288
         if fp: fp.close()
146 289
 
147  
-    if parent:
148  
-        setattr(parent, partname, m)
149  
-
150  
-    return m
151  
-
152  
-def deep_reload_hook(module):
153  
-    name = module.__name__
154  
-    if '.' not in name:
155  
-        return import_module(name, name, None)
156  
-    i = name.rfind('.')
157  
-    pname = name[:i]
158  
-    parent = sys.modules[pname]
159  
-    return import_module(name[i+1:], name, parent)
  290
+    modules_reloading.clear()
  291
+    return newm
160 292
 
161 293
 # Save the original hooks
162 294
 try:
@@ -165,7 +297,7 @@ def deep_reload_hook(module):
165 297
     original_reload = imp.reload    # Python 3
166 298
 
167 299
 # Replacement for reload()
168  
-def reload(module, exclude=['sys', '__builtin__', '__main__']):
  300
+def reload(module, exclude=['sys', 'os.path', '__builtin__', '__main__']):
169 301
     """Recursively reload all modules used in the given module.  Optionally
170 302
     takes a list of modules to exclude from reloading.  The default exclude
171 303
     list contains sys, __main__, and __builtin__, to prevent, e.g., resetting
53  IPython/lib/tests/test_deepreload.py
... ...
@@ -0,0 +1,53 @@
  1
+# -*- coding: utf-8 -*-
  2
+"""Test suite for the deepreload module."""
  3
+
  4
+#-----------------------------------------------------------------------------
  5
+# Imports
  6
+#-----------------------------------------------------------------------------
  7
+
  8
+import os
  9
+import sys
  10
+
  11
+import nose.tools as nt
  12
+
  13
+from IPython.testing import decorators as dec
  14
+from IPython.utils.syspathcontext import prepended_to_syspath
  15
+from IPython.utils.tempdir import TemporaryDirectory
  16
+from IPython.lib.deepreload import reload as dreload
  17
+
  18
+#-----------------------------------------------------------------------------
  19
+# Test functions begin
  20
+#-----------------------------------------------------------------------------
  21
+
  22
+@dec.skipif_not_numpy
  23
+def test_deepreload_numpy():
  24
+    "Test that NumPy can be deep reloaded."
  25
+    import numpy
  26
+    exclude = [
  27
+        # Standard exclusions:
  28
+        'sys', 'os.path', '__builtin__', '__main__',
  29
+        # Test-related exclusions:
  30
+        'unittest',
  31
+        ]
  32
+    dreload(numpy, exclude=exclude)
  33
+
  34
+def test_deepreload():
  35
+    "Test that dreload does deep reloads and skips excluded modules."
  36
+    with TemporaryDirectory() as tmpdir:
  37
+        with prepended_to_syspath(tmpdir):
  38
+            with open(os.path.join(tmpdir, 'A.py'), 'w') as f:
  39
+                f.write("class Object(object):\n    pass\n")
  40
+            with open(os.path.join(tmpdir, 'B.py'), 'w') as f:
  41
+                f.write("import A\n")
  42
+            import A
  43
+            import B
  44
+
  45
+            # Test that A is not reloaded.
  46
+            obj = A.Object()
  47
+            dreload(B, exclude=['A'])
  48
+            nt.assert_true(isinstance(obj, A.Object))
  49
+
  50
+            # Test that A is reloaded.
  51
+            obj = A.Object()
  52
+            dreload(B)
  53
+            nt.assert_false(isinstance(obj, A.Object))
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.