Skip to content
This repository

Usermod #648

Merged
merged 13 commits into from over 2 years ago

3 participants

Thomas Kluyver Fernando Perez Min RK
Thomas Kluyver
Collaborator

This is intended to supersede my PR #384, where I made it possible to pickle interactively defined objects. While I was working on that, I realised that our distinction between local and global namespaces is a bit unclear (because they're usually the same thing). This goes some way towards clearing that up.

A global namespace should always be tied to a module: pickle accesses classes via the module in which they're defined. So I've changed the arguments for instantiating an InteractiveShell to include user_module in place of user_global_ns. The global namespace simply becomes a reference to user_module.__dict__.

For instantiating InteractiveShell, there are four possibilities:

  • Neither user_ns nor user_module is given. A new (real) module is created named __main__, and its __dict__ becomes the global and local namespace. This is what happens when starting IPython normally.
  • Only user_module is given. Its __dict__ becomes the global and local namespace.
  • Both user_ns and user_module are given. user_module.__dict__ is the global namespace, and user_ns is the local namespace. Note that we can't interactively define closures over variables in the local namespace (this seems to be a limitation of Python).
  • Only user_ns is given. It is treated as the global and local namespace, and a DummyMod object is created to refer to it. This is intended as a convenience, especially for the test suite. The recommended way to pass in a single global namespace is as a reference to the module.

embed() digs out the locals and the module from the frame in which it's called.

While I was doing namespaces, I also re-included a reference to the InteractiveShell, as a weakref proxy, named _ipy.

Thomas Kluyver
Collaborator

This now works with embedding in a terminal program, but there's an odd exception. You can modify mutable values and define new local variables, but it doesn't rebind existing names. In the example below, doing b=77 in the embedded shell appears to work, and b becomes 77 for the lifetime of the shell. But when you leave the shell, the script still sees b as 66.

from IPython import embed

a = 12
def f():
    b = 66
    c = []
    embed()

    print locals()
    print globals()

f()
Thomas Kluyver
Collaborator

Also, if I define d in the interactive shell, it shows up when I do print locals(), but print d throws a NameError. I think this is because Python detects local variables when compiling the function, so it doesn't look for d in the locals.

Fernando Perez
Owner

Should we close #384 and focus only on this one then?

Thomas Kluyver
Collaborator

If you're happy with this approach, yes.

Fernando Perez
Owner

OK, I'm closing #384 then, and we'll focus on this one instead. I do like the idea of being more careful about how we handle namespaces, so I vote for this. Unfortunately I can't finish a full review of this one right now, but I'll try to get to it in the evening. Thanks!

Thomas Kluyver
Collaborator

OK, great. Actually, I'm just going to revert a couple of commits that I no longer like, so if you've checked this branch out, be ready to fetch it again.

Min RK
Owner

We had wanted this one to sit in master for a while before release, given its potential implications. Do we want to get on this, as we want 0.12 by December?

Thomas Kluyver
Collaborator

I'm not too concerned about it being deferred. I think the main visible benefit it gives is that you can pickle objects defined interactively (#29). It could potentially lead to some small API breakage, but I don't think most software that embeds it passes a user_global_ns, so it shouldn't be too drastic.

Fernando Perez
Owner

@takluyver, I see a merge conflict on this one now... If you have a chance to clean it up, I'd be happy to try and work on it so we can merge sooner rather than later...

Thomas Kluyver
Collaborator

Rebased and run the test suite on 2.7 and 3.2.

IPython/core/tests/test_interactiveshell.py
((16 lines not shown))
  168
+        # We need to swap in our main module - this is only necessary
  169
+        # inside the test framework, because IPython puts the interactive module
  170
+        # in place (but the test framework undoes this).
  171
+        _main = sys.modules['__main__']
  172
+        sys.modules['__main__'] = ip.user_module
  173
+        try:
  174
+            res = dumps(ip.user_ns["w"])
  175
+        finally:
  176
+            sys.modules['__main__'] = _main
  177
+        self.assertTrue(isinstance(res, bytes))
  178
+        
  179
+    def test_global_ns(self):
  180
+        "Code in functions must be able to access variables outside them."
  181
+        ip = get_ipython()
  182
+        ip.run_cell("a = 10")
  183
+        ip.run_cell(("def f(x):"
2
Fernando Perez Owner
fperez added a note

though this code is correct as-is, you probably meant to have a \n at the end here, no? It would make it more natural and in case anyone ever adds for some reason one more line, it would continue to work where as without newlines one more would be a problem.

Thomas Kluyver Collaborator

You're right - thanks. Sorted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Fernando Perez fperez merged commit a1e4911 into from
Fernando Perez fperez closed this
Fernando Perez fperez referenced this pull request from a commit in fperez/ipython
Fernando Perez Fix critical bug with pylab support inadvertently introduced in #648.
code used it as a dict.  Updated that code to handle a dict correctly,
and added tests to catch this issue in the future (also increases test
coverage of pylab code).
cd84e0e
Fernando Perez fperez referenced this pull request
Merged

Pylab fix #1052

Matthias Bussonnier Carreau referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
Fernando Perez fperez referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
Fernando Perez fperez referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
Fernando Perez fperez referenced this pull request from a commit
Commit has since been removed from the repository and is no longer available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
8  IPython/core/displayhook.py
@@ -265,16 +265,16 @@ def update_user_ns(self, result):
265 265
                 self.___ = self.__
266 266
                 self.__ = self._
267 267
                 self._ = result
268  
-                self.shell.user_ns.update({'_':self._,
269  
-                                           '__':self.__,
270  
-                                           '___':self.___})
  268
+                self.shell.push({'_':self._,
  269
+                                 '__':self.__,
  270
+                                '___':self.___}, interactive=False)
271 271
 
272 272
             # hackish access to top-level  namespace to create _1,_2... dynamically
273 273
             to_main = {}
274 274
             if self.do_full_cache:
275 275
                 new_result = '_'+`self.prompt_count`
276 276
                 to_main[new_result] = result
277  
-                self.shell.user_ns.update(to_main)
  277
+                self.shell.push(to_main, interactive=False)
278 278
                 self.shell.user_ns['_oh'][self.prompt_count] = result
279 279
 
280 280
     def log_output(self, format_dict):
3  IPython/core/history.py
@@ -559,7 +559,8 @@ def store_inputs(self, line_num, source, source_raw=None):
559 559
                    '_ii': self._ii,
560 560
                    '_iii': self._iii,
561 561
                    new_i : self._i00 }
562  
-        self.shell.user_ns.update(to_main)
  562
+
  563
+        self.shell.push(to_main, interactive=False)
563 564
 
564 565
     def store_output(self, line_num):
565 566
         """If database output logging is enabled, this saves all the
209  IPython/core/interactiveshell.py
@@ -29,6 +29,7 @@
29 29
 import sys
30 30
 import tempfile
31 31
 import types
  32
+
32 33
 try:
33 34
     from contextlib import nested
34 35
 except:
@@ -372,7 +373,7 @@ def profile(self):
372 373
     _post_execute = Instance(dict)
373 374
 
374 375
     def __init__(self, config=None, ipython_dir=None, profile_dir=None,
375  
-                 user_ns=None, user_global_ns=None,
  376
+                 user_module=None, user_ns=None,
376 377
                  custom_exceptions=((), None)):
377 378
 
378 379
         # This is where traits with a config_key argument are updated
@@ -387,7 +388,7 @@ def __init__(self, config=None, ipython_dir=None, profile_dir=None,
387 388
         self.init_environment()
388 389
 
389 390
         # Create namespaces (user_ns, user_global_ns, etc.)
390  
-        self.init_create_namespaces(user_ns, user_global_ns)
  391
+        self.init_create_namespaces(user_module, user_ns)
391 392
         # This has to be done after init_create_namespaces because it uses
392 393
         # something in self.user_ns, but before init_sys_modules, which
393 394
         # is the first thing to modify sys.
@@ -639,17 +640,14 @@ def init_reload_doctest(self):
639 640
     def save_sys_module_state(self):
640 641
         """Save the state of hooks in the sys module.
641 642
 
642  
-        This has to be called after self.user_ns is created.
  643
+        This has to be called after self.user_module is created.
643 644
         """
644 645
         self._orig_sys_module_state = {}
645 646
         self._orig_sys_module_state['stdin'] = sys.stdin
646 647
         self._orig_sys_module_state['stdout'] = sys.stdout
647 648
         self._orig_sys_module_state['stderr'] = sys.stderr
648 649
         self._orig_sys_module_state['excepthook'] = sys.excepthook
649  
-        try:
650  
-            self._orig_sys_modules_main_name = self.user_ns['__name__']
651  
-        except KeyError:
652  
-            pass
  650
+        self._orig_sys_modules_main_name = self.user_module.__name__
653 651
 
654 652
     def restore_sys_module_state(self):
655 653
         """Restore the state of the sys module."""
@@ -659,10 +657,7 @@ def restore_sys_module_state(self):
659 657
         except AttributeError:
660 658
             pass
661 659
         # Reset what what done in self.init_sys_modules
662  
-        try:
663  
-            sys.modules[self.user_ns['__name__']] = self._orig_sys_modules_main_name
664  
-        except (AttributeError, KeyError):
665  
-            pass
  660
+        sys.modules[self.user_module.__name__] = self._orig_sys_modules_main_name
666 661
 
667 662
     #-------------------------------------------------------------------------
668 663
     # Things related to hooks
@@ -860,7 +855,7 @@ def debugger(self,force=False):
860 855
     # Things related to IPython's various namespaces
861 856
     #-------------------------------------------------------------------------
862 857
 
863  
-    def init_create_namespaces(self, user_ns=None, user_global_ns=None):
  858
+    def init_create_namespaces(self, user_module=None, user_ns=None):
864 859
         # Create the namespace where the user will operate.  user_ns is
865 860
         # normally the only one used, and it is passed to the exec calls as
866 861
         # the locals argument.  But we do carry a user_global_ns namespace
@@ -894,26 +889,14 @@ def init_create_namespaces(self, user_ns=None, user_global_ns=None):
894 889
         # should start with "import __builtin__" (note, no 's') which will
895 890
         # definitely give you a module. Yeah, it's somewhat confusing:-(.
896 891
 
897  
-        # These routines return properly built dicts as needed by the rest of
898  
-        # the code, and can also be used by extension writers to generate
899  
-        # properly initialized namespaces.
900  
-        user_ns, user_global_ns = self.make_user_namespaces(user_ns,
901  
-                                                            user_global_ns)
  892
+        # These routines return a properly built module and dict as needed by
  893
+        # the rest of the code, and can also be used by extension writers to
  894
+        # generate properly initialized namespaces.
  895
+        self.user_module, self.user_ns = self.prepare_user_module(user_module, user_ns)
902 896
 
903  
-        # Assign namespaces
904  
-        # This is the namespace where all normal user variables live
905  
-        self.user_ns = user_ns
906  
-        self.user_global_ns = user_global_ns
907  
-
908  
-        # An auxiliary namespace that checks what parts of the user_ns were
909  
-        # loaded at startup, so we can list later only variables defined in
910  
-        # actual interactive use.  Since it is always a subset of user_ns, it
911  
-        # doesn't need to be separately tracked in the ns_table.
912  
-        self.user_ns_hidden = {}
913  
-
914  
-        # A namespace to keep track of internal data structures to prevent
915  
-        # them from cluttering user-visible stuff.  Will be updated later
916  
-        self.internal_ns = {}
  897
+        # A record of hidden variables we have added to the user namespace, so
  898
+        # we can list later only variables defined in actual interactive use.
  899
+        self.user_ns_hidden = set()
917 900
 
918 901
         # Now that FakeModule produces a real module, we've run into a nasty
919 902
         # problem: after script execution (via %run), the module where the user
@@ -946,78 +929,61 @@ def init_create_namespaces(self, user_ns=None, user_global_ns=None):
946 929
 
947 930
         # A table holding all the namespaces IPython deals with, so that
948 931
         # introspection facilities can search easily.
949  
-        self.ns_table = {'user':user_ns,
950  
-                         'user_global':user_global_ns,
951  
-                         'internal':self.internal_ns,
  932
+        self.ns_table = {'user_global':self.user_module.__dict__,
  933
+                         'user_local':user_ns,
952 934
                          'builtin':builtin_mod.__dict__
953 935
                          }
  936
+    
  937
+    @property
  938
+    def user_global_ns(self):
  939
+        return self.user_module.__dict__
954 940
 
955  
-        # Similarly, track all namespaces where references can be held and that
956  
-        # we can safely clear (so it can NOT include builtin).  This one can be
957  
-        # a simple list.  Note that the main execution namespaces, user_ns and
958  
-        # user_global_ns, can NOT be listed here, as clearing them blindly
959  
-        # causes errors in object __del__ methods.  Instead, the reset() method
960  
-        # clears them manually and carefully.
961  
-        self.ns_refs_table = [ self.user_ns_hidden,
962  
-                               self.internal_ns, self._main_ns_cache ]
963  
-
964  
-    def make_user_namespaces(self, user_ns=None, user_global_ns=None):
965  
-        """Return a valid local and global user interactive namespaces.
966  
-
967  
-        This builds a dict with the minimal information needed to operate as a
968  
-        valid IPython user namespace, which you can pass to the various
969  
-        embedding classes in ipython. The default implementation returns the
970  
-        same dict for both the locals and the globals to allow functions to
971  
-        refer to variables in the namespace. Customized implementations can
972  
-        return different dicts. The locals dictionary can actually be anything
973  
-        following the basic mapping protocol of a dict, but the globals dict
974  
-        must be a true dict, not even a subclass. It is recommended that any
975  
-        custom object for the locals namespace synchronize with the globals
976  
-        dict somehow.
977  
-
978  
-        Raises TypeError if the provided globals namespace is not a true dict.
  941
+    def prepare_user_module(self, user_module=None, user_ns=None):
  942
+        """Prepare the module and namespace in which user code will be run.
  943
+        
  944
+        When IPython is started normally, both parameters are None: a new module
  945
+        is created automatically, and its __dict__ used as the namespace.
  946
+        
  947
+        If only user_module is provided, its __dict__ is used as the namespace.
  948
+        If only user_ns is provided, a dummy module is created, and user_ns
  949
+        becomes the global namespace. If both are provided (as they may be
  950
+        when embedding), user_ns is the local namespace, and user_module
  951
+        provides the global namespace.
979 952
 
980 953
         Parameters
981 954
         ----------
982  
-        user_ns : dict-like, optional
983  
-            The current user namespace. The items in this namespace should
984  
-            be included in the output. If None, an appropriate blank
985  
-            namespace should be created.
986  
-        user_global_ns : dict, optional
987  
-            The current user global namespace. The items in this namespace
988  
-            should be included in the output. If None, an appropriate
989  
-            blank namespace should be created.
  955
+        user_module : module, optional
  956
+            The current user module in which IPython is being run. If None,
  957
+            a clean module will be created.
  958
+        user_ns : dict, optional
  959
+            A namespace in which to run interactive commands.
990 960
 
991 961
         Returns
992 962
         -------
993  
-            A pair of dictionary-like object to be used as the local namespace
994  
-            of the interpreter and a dict to be used as the global namespace.
  963
+        A tuple of user_module and user_ns, each properly initialised.
995 964
         """
996  
-
997  
-
  965
+        if user_module is None and user_ns is not None:
  966
+            user_ns.setdefault("__name__", "__main__")
  967
+            class DummyMod(object):
  968
+                "A dummy module used for IPython's interactive namespace."
  969
+                pass
  970
+            user_module = DummyMod()
  971
+            user_module.__dict__ = user_ns
  972
+            
  973
+        if user_module is None:
  974
+            user_module = types.ModuleType("__main__",
  975
+                doc="Automatically created module for IPython interactive environment")
  976
+        
998 977
         # We must ensure that __builtin__ (without the final 's') is always
999 978
         # available and pointing to the __builtin__ *module*.  For more details:
1000 979
         # http://mail.python.org/pipermail/python-dev/2001-April/014068.html
1001  
-
  980
+        user_module.__dict__.setdefault('__builtin__', builtin_mod)
  981
+        user_module.__dict__.setdefault('__builtins__', builtin_mod)
  982
+        
1002 983
         if user_ns is None:
1003  
-            # Set __name__ to __main__ to better match the behavior of the
1004  
-            # normal interpreter.
1005  
-            user_ns = {'__name__'     :'__main__',
1006  
-                       py3compat.builtin_mod_name: builtin_mod,
1007  
-                       '__builtins__' : builtin_mod,
1008  
-                      }
1009  
-        else:
1010  
-            user_ns.setdefault('__name__','__main__')
1011  
-            user_ns.setdefault(py3compat.builtin_mod_name,builtin_mod)
1012  
-            user_ns.setdefault('__builtins__',builtin_mod)
1013  
-
1014  
-        if user_global_ns is None:
1015  
-            user_global_ns = user_ns
1016  
-        if type(user_global_ns) is not dict:
1017  
-            raise TypeError("user_global_ns must be a true dict; got %r"
1018  
-                % type(user_global_ns))
  984
+            user_ns = user_module.__dict__
1019 985
 
1020  
-        return user_ns, user_global_ns
  986
+        return user_module, user_ns
1021 987
 
1022 988
     def init_sys_modules(self):
1023 989
         # We need to insert into sys.modules something that looks like a
@@ -1036,13 +1002,8 @@ def init_sys_modules(self):
1036 1002
         # embedded in).
1037 1003
 
1038 1004
         # This is overridden in the InteractiveShellEmbed subclass to a no-op.
1039  
-
1040  
-        try:
1041  
-            main_name = self.user_ns['__name__']
1042  
-        except KeyError:
1043  
-            raise KeyError('user_ns dictionary MUST have a "__name__" key')
1044  
-        else:
1045  
-            sys.modules[main_name] = FakeModule(self.user_ns)
  1005
+        main_name = self.user_module.__name__
  1006
+        sys.modules[main_name] = self.user_module
1046 1007
 
1047 1008
     def init_user_ns(self):
1048 1009
         """Initialize all user-visible namespaces to their minimum defaults.
@@ -1073,8 +1034,8 @@ def init_user_ns(self):
1073 1034
 
1074 1035
         # For more details:
1075 1036
         # http://mail.python.org/pipermail/python-dev/2001-April/014068.html
1076  
-        ns = dict(__builtin__ = builtin_mod)
1077  
-
  1037
+        ns = dict()
  1038
+        
1078 1039
         # Put 'help' in the user namespace
1079 1040
         try:
1080 1041
             from site import _Helper
@@ -1096,7 +1057,7 @@ def init_user_ns(self):
1096 1057
 
1097 1058
         # Store myself as the public api!!!
1098 1059
         ns['get_ipython'] = self.get_ipython
1099  
-
  1060
+        
1100 1061
         ns['exit'] = self.exiter
1101 1062
         ns['quit'] = self.exiter
1102 1063
 
@@ -1110,6 +1071,16 @@ def init_user_ns(self):
1110 1071
 
1111 1072
         # Finally, update the real user's namespace
1112 1073
         self.user_ns.update(ns)
  1074
+    
  1075
+    @property
  1076
+    def all_ns_refs(self):
  1077
+        """Get a list of references to all the namespace dictionaries in which
  1078
+        IPython might store a user-created object.
  1079
+        
  1080
+        Note that this does not include the displayhook, which also caches
  1081
+        objects from the output."""
  1082
+        return [self.user_ns, self.user_global_ns,
  1083
+                self._user_main_module.__dict__] + self._main_ns_cache.values()
1113 1084
 
1114 1085
     def reset(self, new_session=True):
1115 1086
         """Clear all internal namespaces, and attempt to release references to
@@ -1127,20 +1098,21 @@ def reset(self, new_session=True):
1127 1098
         if self.displayhook.do_full_cache:
1128 1099
             self.displayhook.flush()
1129 1100
 
1130  
-        # Restore the user namespaces to minimal usability
1131  
-        for ns in self.ns_refs_table:
1132  
-            ns.clear()
1133  
-
1134 1101
         # The main execution namespaces must be cleared very carefully,
1135 1102
         # skipping the deletion of the builtin-related keys, because doing so
1136 1103
         # would cause errors in many object's __del__ methods.
1137  
-        for ns in [self.user_ns, self.user_global_ns]:
1138  
-            drop_keys = set(ns.keys())
1139  
-            drop_keys.discard('__builtin__')
1140  
-            drop_keys.discard('__builtins__')
1141  
-            for k in drop_keys:
1142  
-                del ns[k]
1143  
-
  1104
+        if self.user_ns is not self.user_global_ns:
  1105
+            self.user_ns.clear()
  1106
+        ns = self.user_global_ns
  1107
+        drop_keys = set(ns.keys())
  1108
+        drop_keys.discard('__builtin__')
  1109
+        drop_keys.discard('__builtins__')
  1110
+        drop_keys.discard('__name__')
  1111
+        for k in drop_keys:
  1112
+            del ns[k]
  1113
+        
  1114
+        self.user_ns_hidden.clear()
  1115
+        
1144 1116
         # Restore the user namespaces to minimal usability
1145 1117
         self.init_user_ns()
1146 1118
 
@@ -1170,10 +1142,9 @@ def del_var(self, varname, by_name=False):
1170 1142
         """
1171 1143
         if varname in ('__builtin__', '__builtins__'):
1172 1144
             raise ValueError("Refusing to delete %s" % varname)
1173  
-        ns_refs = self.ns_refs_table + [self.user_ns,
1174  
-                self.user_global_ns, self._user_main_module.__dict__] +\
1175  
-                self._main_ns_cache.values()
1176 1145
 
  1146
+        ns_refs = self.all_ns_refs
  1147
+        
1177 1148
         if by_name:                    # Delete by name
1178 1149
             for ns in ns_refs:
1179 1150
                 try:
@@ -1214,7 +1185,7 @@ def reset_selective(self, regex=None):
1214 1185
                 raise TypeError('regex must be a string or compiled pattern')
1215 1186
             # Search for keys in each namespace that match the given regex
1216 1187
             # If a match is found, delete the key/value pair.
1217  
-            for ns in self.ns_refs_table:
  1188
+            for ns in self.all_ns_refs:
1218 1189
                 for var in ns:
1219 1190
                     if m.search(var):
1220 1191
                         del ns[var]
@@ -1260,13 +1231,11 @@ def push(self, variables, interactive=True):
1260 1231
         self.user_ns.update(vdict)
1261 1232
 
1262 1233
         # And configure interactive visibility
1263  
-        config_ns = self.user_ns_hidden
  1234
+        user_ns_hidden = self.user_ns_hidden
1264 1235
         if interactive:
1265  
-            for name, val in vdict.iteritems():
1266  
-                config_ns.pop(name, None)
  1236
+            user_ns_hidden.difference_update(vdict)
1267 1237
         else:
1268  
-            for name,val in vdict.iteritems():
1269  
-                config_ns[name] = val
  1238
+            user_ns_hidden.update(vdict)
1270 1239
     
1271 1240
     def drop_by_id(self, variables):
1272 1241
         """Remove a dict of variables from the user namespace, if they are the
@@ -1284,7 +1253,7 @@ def drop_by_id(self, variables):
1284 1253
         for name, obj in variables.iteritems():
1285 1254
             if name in self.user_ns and self.user_ns[name] is obj:
1286 1255
                 del self.user_ns[name]
1287  
-                self.user_ns_hidden.pop(name, None)
  1256
+                self.user_ns_hidden.discard(name)
1288 1257
 
1289 1258
     #-------------------------------------------------------------------------
1290 1259
     # Things related to object introspection
@@ -1308,7 +1277,7 @@ def _ofind(self, oname, namespaces=None):
1308 1277
             # Put them in a list. The order is important so that we
1309 1278
             # find things in the same order that Python finds them.
1310 1279
             namespaces = [ ('Interactive', self.user_ns),
1311  
-                           ('IPython internal', self.internal_ns),
  1280
+                           ('Interactive (global)', self.user_global_ns),
1312 1281
                            ('Python builtin', builtin_mod.__dict__),
1313 1282
                            ('Alias', self.alias_manager.alias_table),
1314 1283
                            ]
3  IPython/core/magic.py
@@ -748,11 +748,10 @@ def magic_who_ls(self, parameter_s=''):
748 748
         """
749 749
 
750 750
         user_ns = self.shell.user_ns
751  
-        internal_ns = self.shell.internal_ns
752 751
         user_ns_hidden = self.shell.user_ns_hidden
753 752
         out = [ i for i in user_ns
754 753
                 if not i.startswith('_') \
755  
-                and not (i in internal_ns or i in user_ns_hidden) ]
  754
+                and not i in user_ns_hidden ]
756 755
 
757 756
         typelist = parameter_s.split()
758 757
         if typelist:
2  IPython/core/prefilter.py
@@ -86,7 +86,7 @@ def is_shadowed(identifier, ip):
86 86
     than ifun, because it can not contain a '.' character."""
87 87
     # This is much safer than calling ofind, which can change state
88 88
     return (identifier in ip.user_ns \
89  
-            or identifier in ip.internal_ns \
  89
+            or identifier in ip.user_global_ns \
90 90
             or identifier in ip.ns_table['builtin'])
91 91
 
92 92
 
33  IPython/core/tests/test_interactiveshell.py
@@ -25,6 +25,7 @@
25 25
 import tempfile
26 26
 import unittest
27 27
 from os.path import join
  28
+import sys
28 29
 from StringIO import StringIO
29 30
 
30 31
 from IPython.testing import decorators as dec
@@ -128,6 +129,7 @@ def __repr__(self):
128 129
         f = IPython.core.formatters.PlainTextFormatter()
129 130
         f([Spam(),Spam()])
130 131
     
  132
+
131 133
     def test_future_flags(self):
132 134
         """Check that future flags are used for parsing code (gh-777)"""
133 135
         ip = get_ipython()
@@ -151,6 +153,37 @@ def test_future_unicode(self):
151 153
         finally:
152 154
             # Reset compiler flags so we don't mess up other tests.
153 155
             ip.compile.reset_compiler_flags()
  156
+    
  157
+    def test_can_pickle(self):
  158
+        "Can we pickle objects defined interactively (GH-29)"
  159
+        ip = get_ipython()
  160
+        ip.reset()
  161
+        ip.run_cell(("class Mylist(list):\n"
  162
+                     "    def __init__(self,x=[]):\n"
  163
+                     "        list.__init__(self,x)"))
  164
+        ip.run_cell("w=Mylist([1,2,3])")
  165
+        
  166
+        from cPickle import dumps
  167
+        
  168
+        # We need to swap in our main module - this is only necessary
  169
+        # inside the test framework, because IPython puts the interactive module
  170
+        # in place (but the test framework undoes this).
  171
+        _main = sys.modules['__main__']
  172
+        sys.modules['__main__'] = ip.user_module
  173
+        try:
  174
+            res = dumps(ip.user_ns["w"])
  175
+        finally:
  176
+            sys.modules['__main__'] = _main
  177
+        self.assertTrue(isinstance(res, bytes))
  178
+        
  179
+    def test_global_ns(self):
  180
+        "Code in functions must be able to access variables outside them."
  181
+        ip = get_ipython()
  182
+        ip.run_cell("a = 10")
  183
+        ip.run_cell(("def f(x):\n"
  184
+                     "    return x + a"))
  185
+        ip.run_cell("b = f(12)")
  186
+        self.assertEqual(ip.user_ns["b"], 22)
154 187
 
155 188
     def test_bad_custom_tb(self):
156 189
         """Check that InteractiveShell is protected from bad custom exception handlers"""
16  IPython/core/tests/test_iplib.py
@@ -28,12 +28,8 @@
28 28
 # Test functions
29 29
 #-----------------------------------------------------------------------------
30 30
 
31  
-@dec.parametric
32 31
 def test_reset():
33 32
     """reset must clear most namespaces."""
34  
-    # The number of variables in the private user_ns_hidden is not zero, but it
35  
-    # should be constant regardless of what we do
36  
-    nvars_config_ns = len(ip.user_ns_hidden)
37 33
 
38 34
     # Check that reset runs without error
39 35
     ip.reset()
@@ -41,6 +37,7 @@ def test_reset():
41 37
     # Once we've reset it (to clear of any junk that might have been there from
42 38
     # other tests, we can count how many variables are in the user's namespace
43 39
     nvars_user_ns = len(ip.user_ns)
  40
+    nvars_hidden = len(ip.user_ns_hidden)
44 41
 
45 42
     # Now add a few variables to user_ns, and check that reset clears them
46 43
     ip.user_ns['x'] = 1
@@ -49,15 +46,8 @@ def test_reset():
49 46
     
50 47
     # Finally, check that all namespaces have only as many variables as we
51 48
     # expect to find in them:
52  
-    for ns in ip.ns_refs_table:
53  
-        if ns is ip.user_ns:
54  
-            nvars_expected = nvars_user_ns
55  
-        elif ns is ip.user_ns_hidden:
56  
-            nvars_expected = nvars_config_ns
57  
-        else:
58  
-            nvars_expected = 0
59  
-            
60  
-        yield nt.assert_equals(len(ns), nvars_expected)
  49
+    nt.assert_equals(len(ip.user_ns), nvars_user_ns)
  50
+    nt.assert_equals(len(ip.user_ns_hidden), nvars_hidden)
61 51
 
62 52
 
63 53
 # Tests for reporting of exceptions in various modes, handling of SystemExit,
12  IPython/core/tests/test_run.py
@@ -228,3 +228,15 @@ def test_tclass(self):
228 228
         else:
229 229
             err = None
230 230
         tt.ipexec_validate(self.fname, out, err)
  231
+
  232
+    def test_run_i_after_reset(self):
  233
+        """Check that %run -i still works after %reset (gh-693)"""
  234
+        src = "yy = zz\n"
  235
+        self.mktmp(src)
  236
+        _ip.run_cell("zz = 23")
  237
+        _ip.magic('run -i %s' % self.fname)
  238
+        tt.assert_equals(_ip.user_ns['yy'], 23)
  239
+        _ip.magic('reset -f')
  240
+        _ip.run_cell("zz = 23")
  241
+        _ip.magic('run -i %s' % self.fname)
  242
+        tt.assert_equals(_ip.user_ns['yy'], 23)
58  IPython/frontend/terminal/embed.py
@@ -72,13 +72,13 @@ class InteractiveShellEmbed(TerminalInteractiveShell):
72 72
     display_banner = CBool(True)
73 73
 
74 74
     def __init__(self, config=None, ipython_dir=None, user_ns=None,
75  
-                 user_global_ns=None, custom_exceptions=((),None),
  75
+                 user_module=None, custom_exceptions=((),None),
76 76
                  usage=None, banner1=None, banner2=None,
77 77
                  display_banner=None, exit_msg=u''):
78 78
 
79 79
         super(InteractiveShellEmbed,self).__init__(
80 80
             config=config, ipython_dir=ipython_dir, user_ns=user_ns,
81  
-            user_global_ns=user_global_ns, custom_exceptions=custom_exceptions,
  81
+            user_module=user_module, custom_exceptions=custom_exceptions,
82 82
             usage=usage, banner1=banner1, banner2=banner2,
83 83
             display_banner=display_banner
84 84
         )
@@ -95,7 +95,7 @@ def __init__(self, config=None, ipython_dir=None, user_ns=None,
95 95
     def init_sys_modules(self):
96 96
         pass
97 97
 
98  
-    def __call__(self, header='', local_ns=None, global_ns=None, dummy=None,
  98
+    def __call__(self, header='', local_ns=None, module=None, dummy=None,
99 99
                  stack_depth=1):
100 100
         """Activate the interactive interpreter.
101 101
 
@@ -140,14 +140,14 @@ def __call__(self, header='', local_ns=None, global_ns=None, dummy=None,
140 140
 
141 141
         # Call the embedding code with a stack depth of 1 so it can skip over
142 142
         # our call and get the original caller's namespaces.
143  
-        self.mainloop(local_ns, global_ns, stack_depth=stack_depth)
  143
+        self.mainloop(local_ns, module, stack_depth=stack_depth)
144 144
 
145 145
         self.banner2 = self.old_banner2
146 146
 
147 147
         if self.exit_msg is not None:
148 148
             print self.exit_msg
149 149
 
150  
-    def mainloop(self, local_ns=None, global_ns=None, stack_depth=0,
  150
+    def mainloop(self, local_ns=None, module=None, stack_depth=0,
151 151
                  display_banner=None):
152 152
         """Embeds IPython into a running python program.
153 153
 
@@ -172,32 +172,37 @@ def mainloop(self, local_ns=None, global_ns=None, stack_depth=0,
172 172
         there is no fundamental reason why it can't work perfectly."""
173 173
 
174 174
         # Get locals and globals from caller
175  
-        if local_ns is None or global_ns is None:
  175
+        if local_ns is None or module is None:
176 176
             call_frame = sys._getframe(stack_depth).f_back
177 177
 
178 178
             if local_ns is None:
179 179
                 local_ns = call_frame.f_locals
180  
-            if global_ns is None:
  180
+            if module is None:
181 181
                 global_ns = call_frame.f_globals
182  
-
  182
+                module = sys.modules[global_ns['__name__']]
  183
+        
  184
+        # Save original namespace and module so we can restore them after 
  185
+        # embedding; otherwise the shell doesn't shut down correctly.
  186
+        orig_user_module = self.user_module
  187
+        orig_user_ns = self.user_ns
  188
+        
183 189
         # Update namespaces and fire up interpreter
184  
-
  190
+        
185 191
         # The global one is easy, we can just throw it in
186  
-        self.user_global_ns = global_ns
  192
+        self.user_module = module
187 193
 
188  
-        # but the user/local one is tricky: ipython needs it to store internal
189  
-        # data, but we also need the locals.  We'll copy locals in the user
190  
-        # one, but will track what got copied so we can delete them at exit.
191  
-        # This is so that a later embedded call doesn't see locals from a
192  
-        # previous call (which most likely existed in a separate scope).
193  
-        local_varnames = local_ns.keys()
194  
-        self.user_ns.update(local_ns)
195  
-        #self.user_ns['local_ns'] = local_ns  # dbg
  194
+        # But the user/local one is tricky: ipython needs it to store internal
  195
+        # data, but we also need the locals. We'll throw our hidden variables
  196
+        # like _ih and get_ipython() into the local namespace, but delete them
  197
+        # later.
  198
+        self.user_ns = local_ns
  199
+        self.init_user_ns()
196 200
 
197 201
         # Patch for global embedding to make sure that things don't overwrite
198 202
         # user globals accidentally. Thanks to Richard <rxe@renre-europe.com>
199 203
         # FIXME. Test this a bit more carefully (the if.. is new)
200  
-        if local_ns is None and global_ns is None:
  204
+        # N.B. This can't now ever be called. Not sure what it was for.
  205
+        if local_ns is None and module is None:
201 206
             self.user_global_ns.update(__main__.__dict__)
202 207
 
203 208
         # make sure the tab-completer has the correct frame information, so it
@@ -206,13 +211,14 @@ def mainloop(self, local_ns=None, global_ns=None, stack_depth=0,
206 211
 
207 212
         with nested(self.builtin_trap, self.display_trap):
208 213
             self.interact(display_banner=display_banner)
209  
-
210  
-            # now, purge out the user namespace from anything we might have added
211  
-            # from the caller's local namespace
212  
-            delvar = self.user_ns.pop
213  
-            for var in local_varnames:
214  
-                delvar(var,None)
215  
-
  214
+        
  215
+        # now, purge out the local namespace of IPython's hidden variables.
  216
+        for name in self.user_ns_hidden:
  217
+            local_ns.pop(name, None)
  218
+        
  219
+        # Restore original namespace so shell can shut down when we exit.
  220
+        self.user_module = orig_user_module
  221
+        self.user_ns = orig_user_ns
216 222
 
217 223
 _embedded_shell = None
218 224
 
4  IPython/frontend/terminal/interactiveshell.py
@@ -172,13 +172,13 @@ class TerminalInteractiveShell(InteractiveShell):
172 172
     )
173 173
 
174 174
     def __init__(self, config=None, ipython_dir=None, profile_dir=None, user_ns=None,
175  
-                 user_global_ns=None, custom_exceptions=((),None),
  175
+                 user_module=None, custom_exceptions=((),None),
176 176
                  usage=None, banner1=None, banner2=None,
177 177
                  display_banner=None):
178 178
 
179 179
         super(TerminalInteractiveShell, self).__init__(
180 180
             config=config, profile_dir=profile_dir, user_ns=user_ns,
181  
-            user_global_ns=user_global_ns, custom_exceptions=custom_exceptions
  181
+            user_module=user_module, custom_exceptions=custom_exceptions
182 182
         )
183 183
         # use os.system instead of utils.process.system by default,
184 184
         # because piped system doesn't make sense in the Terminal:
4  IPython/testing/globalipapp.py
@@ -190,7 +190,6 @@ def start_ipython():
190 190
     # Create and initialize our test-friendly IPython instance.
191 191
     shell = TerminalInteractiveShell.instance(config=config,
192 192
                                               user_ns=ipnsdict(),
193  
-                                              user_global_ns={}
194 193
                                               )
195 194
 
196 195
     # A few more tweaks needed for playing nicely with doctests...
@@ -206,8 +205,7 @@ def start_ipython():
206 205
     # can capture subcommands and print them to Python's stdout, otherwise the
207 206
     # doctest machinery would miss them.
208 207
     shell.system = py3compat.MethodType(xsys, shell)
209  
-
210  
-
  208
+    
211 209
     shell._showtraceback = py3compat.MethodType(_showtraceback, shell)
212 210
 
213 211
     # IPython is ready, now clean up some global state...
4  IPython/testing/plugin/ipdoctest.py
@@ -271,6 +271,8 @@ def setUp(self):
271 271
             # for IPython examples *only*, we swap the globals with the ipython
272 272
             # namespace, after updating it with the globals (which doctest
273 273
             # fills with the necessary info from the module being tested).
  274
+            self.user_ns_orig = {}
  275
+            self.user_ns_orig.update(_ip.user_ns)
274 276
             _ip.user_ns.update(self._dt_test.globs)
275 277
             self._dt_test.globs = _ip.user_ns
276 278
             # IPython must protect the _ key in the namespace (it can't exist)
@@ -286,6 +288,8 @@ def tearDown(self):
286 288
         # teardown doesn't destroy the ipython namespace
287 289
         if isinstance(self._dt_test.examples[0],IPExample):
288 290
             self._dt_test.globs = self._dt_test_globs_ori
  291
+            _ip.user_ns.clear()
  292
+            _ip.user_ns.update(self.user_ns_orig)
289 293
             # Restore the behavior of the '_' key in the user namespace to
290 294
             # normal after each doctest, so that unittests behave normally
291 295
             _ip.user_ns.protect_underscore = False
10  docs/source/whatsnew/development.txt
@@ -97,6 +97,10 @@ Major Bugs fixed
97 97
 * IPython no longer crashes when started on recent versions of Python 3 in
98 98
   Windows (:ghissue:`737`).
99 99
 
  100
+* Instances of classes defined interactively can now be pickled (:ghissue:`29`;
  101
+  :ghpull:`648`). Note that pickling saves a reference to the class definition,
  102
+  so unpickling the instances will only work where the class has been defined.
  103
+
100 104
 .. * use bullet list
101 105
 
102 106
 Backwards incompatible changes
@@ -132,4 +136,10 @@ Backwards incompatible changes
132 136
   The full path will still work, and is necessary for using custom launchers not in
133 137
   IPython's launcher module.
134 138
 
  139
+* For embedding a shell, note that the parameter ``user_global_ns`` has been
  140
+  replaced by ``user_module``, and expects a module-like object, rather than
  141
+  a namespace dict. The ``user_ns`` parameter works the same way as before, and
  142
+  calling :func:`~IPython.frontend.terminal.embed.embed` with no arguments still
  143
+  works the same way.
  144
+
135 145
 .. * use bullet list
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.