cPickle works in standard interpreter, but not in IPython #29

Closed
ipython opened this Issue May 10, 2010 · 17 comments

Comments

Projects
None yet
5 participants
@ghost
Collaborator

ghost commented May 10, 2010

Original Launchpad bug 363115: https://bugs.launchpad.net/ipython/+bug/363115
Reported by: reckoner (reckoner).

For example in IPython:

class Mylist(list):
     def __init__(self,x=[]):
          list.__init__(self,x)

from cPickle import dumps
w=Mylist([1,2,3])
dumps(w)

PicklingError: Can't pickle <class '__main__.Mylist'>: attribute
lookup __main__.Mylist failed

However, using the standard Python interpreter:

 Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32
 Type "help", "copyright", "credits" or "license" for more information.
 >>> from cPickle import dumps
 >>> class Mylist(list):
 ...         def __init__(self,x=[]):
 ...             list.__init__(self,x)
 ...
 >>> w=Mylist([1,2,3])
 >>> dumps(w)
 'ccopy_reg\n_reconstructor\np1\n(c__main__\nMylist\np2\nc__builtin__\nlist\np3\n
 (lp4\nI1\naI2\naI3\natRp5\n.'
 >>>
@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost May 10, 2010

Collaborator

[ LP comment 1 by: avdd, on 2009-07-16 03:48:57.797382+00:00 ]

Has nothing to do with pickle, but the FakeModule that is registered as main:

$ python
Python 2.5.2 (r252:60911, Oct 5 2008, 19:24:49)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import sys
sys.modules[name].foo = 1
foo
1

a$ ipython

In [1]: import sys

In [2]: sys.modules[name].foo = 1

In [3]: foo

NameError Traceback (most recent call last)

/home/avdd/ in ()

NameError: name 'foo' is not defined

In [4]:

Collaborator

ghost commented May 10, 2010

[ LP comment 1 by: avdd, on 2009-07-16 03:48:57.797382+00:00 ]

Has nothing to do with pickle, but the FakeModule that is registered as main:

$ python
Python 2.5.2 (r252:60911, Oct 5 2008, 19:24:49)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import sys
sys.modules[name].foo = 1
foo
1

a$ ipython

In [1]: import sys

In [2]: sys.modules[name].foo = 1

In [3]: foo

NameError Traceback (most recent call last)

/home/avdd/ in ()

NameError: name 'foo' is not defined

In [4]:

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost May 10, 2010

Collaborator

[ LP comment 2 by: Matt Hickford, on 2009-07-18 19:36:43.821085+00:00 ]

I have problem that neither pickle nor cPickle work in ipython (but work in python proper).

In my case pickle.dump works but pickle.

This is Jaunty, Python 2.6

My error:
AttributeError: 'FakeModule' object has no attribute 'Dictionary'

(I have a custom class inherited from list called Dictionary)

Collaborator

ghost commented May 10, 2010

[ LP comment 2 by: Matt Hickford, on 2009-07-18 19:36:43.821085+00:00 ]

I have problem that neither pickle nor cPickle work in ipython (but work in python proper).

In my case pickle.dump works but pickle.

This is Jaunty, Python 2.6

My error:
AttributeError: 'FakeModule' object has no attribute 'Dictionary'

(I have a custom class inherited from list called Dictionary)

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost May 10, 2010

Collaborator

[ LP comment 3 by: Ville M. Vainio, on 2009-07-18 20:11:20+00:00 ]

On Sat, Jul 18, 2009 at 10:36 PM, Matt
Hickfordmatt.hickford+launchpad@gmail.com wrote:

I have problem that neither pickle nor cPickle work in ipython (but work
in python proper).

In my case pickle.dump works but pickle.

I think you'll note that pickle works when the class of the pickleable
object is not declared interactively, but rather imported from module.

Ville M. Vainio
http://tinyurl.com/vainio

Collaborator

ghost commented May 10, 2010

[ LP comment 3 by: Ville M. Vainio, on 2009-07-18 20:11:20+00:00 ]

On Sat, Jul 18, 2009 at 10:36 PM, Matt
Hickfordmatt.hickford+launchpad@gmail.com wrote:

I have problem that neither pickle nor cPickle work in ipython (but work
in python proper).

In my case pickle.dump works but pickle.

I think you'll note that pickle works when the class of the pickleable
object is not declared interactively, but rather imported from module.

Ville M. Vainio
http://tinyurl.com/vainio

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost May 10, 2010

Collaborator

[ LP comment 4 by: reckoner, on 2009-07-19 23:50:39+00:00 ]

If you get the class using the %run magic instead of importing, you DO
get the pickle failure. I don't know if doing %run is the same as
defining a class interactively, though.

Any guidance appreciated.

thanks.

On Sat, Jul 18, 2009 at 1:11 PM, Ville M. Vainiovivainio@gmail.com wrote:

On Sat, Jul 18, 2009 at 10:36 PM, Matt
Hickfordmatt.hickford+launchpad@gmail.com wrote:

I have problem that neither pickle nor cPickle work in ipython (but work
in python proper).

In my case pickle.dump works but pickle.

I think you'll note that pickle works when the class of the pickleable
object is not declared interactively, but rather imported from module.

Ville M. Vainio
http://tinyurl.com/vainio

cPickle works in standard interpreter, but not in IPython
https://bugs.launchpad.net/bugs/363115
You received this bug notification because you are a direct subscriber
of the bug.

Status in IPython - Enhanced Interactive Python: Confirmed

Bug description:

for example:

In IPython:

class Mylist(list):
      def init(self,x=[]):
          list.init(self,x)

from cPickle import dumps
w=Mylist([1,2,3])
dumps(w)

PicklingError: Can't pickle <class 'main.Mylist'>: attribute
lookup main.Mylist failed

However, using the standard Python interpreter:

Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.

from cPickle import dumps
class Mylist(list):
...         def init(self,x=[]):
...             list.init(self,x)
...
w=Mylist([1,2,3])
dumps(w)
'ccopy_reg\n_reconstructor\np1\n(c__main__\nMylist\np2\nc__builtin__\nlist\np3\n
(lp4\nI1\naI2\naI3\natRp5\n.'

Collaborator

ghost commented May 10, 2010

[ LP comment 4 by: reckoner, on 2009-07-19 23:50:39+00:00 ]

If you get the class using the %run magic instead of importing, you DO
get the pickle failure. I don't know if doing %run is the same as
defining a class interactively, though.

Any guidance appreciated.

thanks.

On Sat, Jul 18, 2009 at 1:11 PM, Ville M. Vainiovivainio@gmail.com wrote:

On Sat, Jul 18, 2009 at 10:36 PM, Matt
Hickfordmatt.hickford+launchpad@gmail.com wrote:

I have problem that neither pickle nor cPickle work in ipython (but work
in python proper).

In my case pickle.dump works but pickle.

I think you'll note that pickle works when the class of the pickleable
object is not declared interactively, but rather imported from module.

Ville M. Vainio
http://tinyurl.com/vainio

cPickle works in standard interpreter, but not in IPython
https://bugs.launchpad.net/bugs/363115
You received this bug notification because you are a direct subscriber
of the bug.

Status in IPython - Enhanced Interactive Python: Confirmed

Bug description:

for example:

In IPython:

class Mylist(list):
      def init(self,x=[]):
          list.init(self,x)

from cPickle import dumps
w=Mylist([1,2,3])
dumps(w)

PicklingError: Can't pickle <class 'main.Mylist'>: attribute
lookup main.Mylist failed

However, using the standard Python interpreter:

Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.

from cPickle import dumps
class Mylist(list):
...         def init(self,x=[]):
...             list.init(self,x)
...
w=Mylist([1,2,3])
dumps(w)
'ccopy_reg\n_reconstructor\np1\n(c__main__\nMylist\np2\nc__builtin__\nlist\np3\n
(lp4\nI1\naI2\naI3\natRp5\n.'

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost May 10, 2010

Collaborator

[ LP comment 5 by: eteq, on 2009-12-17 01:34:20.367155+00:00 ]

I've encountered this in the past - I always just define any class I want to pickle in a file and then reload the module as necessary instead of pickling outputs from run or interactively-defined objects. That seems safer anyway - otherwise if you restart the interpreter, the methods on the object are forgotten.

It seems to work fine if you pickle from a file and then load that file with an interactively-defined or defined in "run" version, though.

Collaborator

ghost commented May 10, 2010

[ LP comment 5 by: eteq, on 2009-12-17 01:34:20.367155+00:00 ]

I've encountered this in the past - I always just define any class I want to pickle in a file and then reload the module as necessary instead of pickling outputs from run or interactively-defined objects. That seems safer anyway - otherwise if you restart the interpreter, the methods on the object are forgotten.

It seems to work fine if you pickle from a file and then load that file with an interactively-defined or defined in "run" version, though.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost May 10, 2010

Collaborator

[ LP comment 6 by: Zak Stone, on 2009-12-17 03:02:24.672555+00:00 ]

For me, a similar problem arises when I am simply unpickling dictionaries in ipython -- no custom classes involved.

Collaborator

ghost commented May 10, 2010

[ LP comment 6 by: Zak Stone, on 2009-12-17 03:02:24.672555+00:00 ]

For me, a similar problem arises when I am simply unpickling dictionaries in ipython -- no custom classes involved.

@minrk

This comment has been minimized.

Show comment
Hide comment
@minrk

minrk Mar 23, 2011

Member

This means that interactively defined functions cannot be used with multiprocessing, as described in #86

Member

minrk commented Mar 23, 2011

This means that interactively defined functions cannot be used with multiprocessing, as described in #86

@minrk

This comment has been minimized.

Show comment
Hide comment
@minrk

minrk Mar 28, 2011

Member

The problem appears to be that the FakeModule used for sys.modules['__main__'] is updated once from user_ns, but it is not linked. If anyone knows of an easy way to effectively do:
FakeModule.dict = user_ns
Then we should be set. Otherwise, I can get pickling to work with the following unattractive hack:

In [1]: from IPython.core.fakemodule import init_fakemod_dict
In [2]: ip = get_ipython()
In [3]: fakemain = sys.modules['__main__']
In [4]: ip.register_post_execute(lambda : init_fakemod_dict(fakemain, ip.user_ns))

This manually syncs the FakeModule in __main__ after each execution. It is expensive and ugly because it involves a clear+update of the entire user_ns after each execution, but for users who need to pickle interactively defined objects before we figure out a clean solution, it should work as a band-aid solution.

Member

minrk commented Mar 28, 2011

The problem appears to be that the FakeModule used for sys.modules['__main__'] is updated once from user_ns, but it is not linked. If anyone knows of an easy way to effectively do:
FakeModule.dict = user_ns
Then we should be set. Otherwise, I can get pickling to work with the following unattractive hack:

In [1]: from IPython.core.fakemodule import init_fakemod_dict
In [2]: ip = get_ipython()
In [3]: fakemain = sys.modules['__main__']
In [4]: ip.register_post_execute(lambda : init_fakemod_dict(fakemain, ip.user_ns))

This manually syncs the FakeModule in __main__ after each execution. It is expensive and ugly because it involves a clear+update of the entire user_ns after each execution, but for users who need to pickle interactively defined objects before we figure out a clean solution, it should work as a band-aid solution.

@fperez

This comment has been minimized.

Show comment
Hide comment
@fperez

fperez Apr 8, 2011

Member

Thanks for the hack, Min. This is a really important one to fix, though I'm reluctant to make it a blocker because it's hard work that may delay us for a while, and it's such an old bug that we've obviously managed to live with it.

But I'd be thrilled to see a proper fix for this.

Member

fperez commented Apr 8, 2011

Thanks for the hack, Min. This is a really important one to fix, though I'm reluctant to make it a blocker because it's hard work that may delay us for a while, and it's such an old bug that we've obviously managed to live with it.

But I'd be thrilled to see a proper fix for this.

@minrk

This comment has been minimized.

Show comment
Hide comment
@minrk

minrk Apr 8, 2011

Member

@fperez do you know if there is a way for us to just have the FM.dict softlink to user_ns, rather than doing a one-time copy?

I couldn't figure out a way in my relatively brief exploration of this, but it seems like that would resolve the whole problem.

Member

minrk commented Apr 8, 2011

@fperez do you know if there is a way for us to just have the FM.dict softlink to user_ns, rather than doing a one-time copy?

I couldn't figure out a way in my relatively brief exploration of this, but it seems like that would resolve the whole problem.

@fperez

This comment has been minimized.

Show comment
Hide comment
@fperez

fperez Apr 8, 2011

Member

On Fri, Apr 8, 2011 at 9:49 AM, minrk
reply@reply.github.com
wrote:

@fperez do you know if there is a way for us to just have the FM.dict softlink to user_ns, rather than doing a one-time copy?

I couldn't figure out a way in my relatively brief exploration of this, but it seems like that would resolve the whole problem.

My concern would be the fact that this would couple a module (a global
object) to the running namespace. That makes a mess with embedding,
multiple shells instantiated in the test suite, etc.

The FakeModule thing is something I wrote years ago, at the very
beginning of ipython, to get some namespace issues working. Perhaps
we should just revisit the whole question fresh, and see if we can get
%run and other namespace-intensive things working, but with a
different approach that has a better profile regarding pickling...

Member

fperez commented Apr 8, 2011

On Fri, Apr 8, 2011 at 9:49 AM, minrk
reply@reply.github.com
wrote:

@fperez do you know if there is a way for us to just have the FM.dict softlink to user_ns, rather than doing a one-time copy?

I couldn't figure out a way in my relatively brief exploration of this, but it seems like that would resolve the whole problem.

My concern would be the fact that this would couple a module (a global
object) to the running namespace. That makes a mess with embedding,
multiple shells instantiated in the test suite, etc.

The FakeModule thing is something I wrote years ago, at the very
beginning of ipython, to get some namespace issues working. Perhaps
we should just revisit the whole question fresh, and see if we can get
%run and other namespace-intensive things working, but with a
different approach that has a better profile regarding pickling...

@minrk

This comment has been minimized.

Show comment
Hide comment
@minrk

minrk Apr 8, 2011

Member

Right, but it's specifically the __main__ module, which is the running namespace.

I certainly don't know enough about what __main__ is used for, and what we should do about it.

Member

minrk commented Apr 8, 2011

Right, but it's specifically the __main__ module, which is the running namespace.

I certainly don't know enough about what __main__ is used for, and what we should do about it.

@fperez

This comment has been minimized.

Show comment
Hide comment
@fperez

fperez Apr 8, 2011

Member

On Fri, Apr 8, 2011 at 10:46 AM, minrk
reply@reply.github.com
wrote:

Right, but it's specifically the __main__ module, which is the running namespace.

I certainly don't know enough about what __main__ is used for, and what we should do about it.

That's why I'm reluctant to make this one a blocker right now: it's
very delicate and affects a lot of things in how everything operates.
The best might be to come up with a few tests around all of this, add
them including failing ones as KnownFailure, and at least we'll have
coverage of the issue from the test perspective. Then post 0.11, we
can take a stab at actually fixing this, with some time to think
carefully about the entire issue without the rush of a release.

Cheers,

f

Member

fperez commented Apr 8, 2011

On Fri, Apr 8, 2011 at 10:46 AM, minrk
reply@reply.github.com
wrote:

Right, but it's specifically the __main__ module, which is the running namespace.

I certainly don't know enough about what __main__ is used for, and what we should do about it.

That's why I'm reluctant to make this one a blocker right now: it's
very delicate and affects a lot of things in how everything operates.
The best might be to come up with a few tests around all of this, add
them including failing ones as KnownFailure, and at least we'll have
coverage of the issue from the test perspective. Then post 0.11, we
can take a stab at actually fixing this, with some time to think
carefully about the entire issue without the rush of a release.

Cheers,

f

@dwf

This comment has been minimized.

Show comment
Hide comment
@dwf

dwf Apr 9, 2011

Contributor

The exact same issue causes #131. This should also get a "FakeModule" label, but I can't figure out how to add it myself (maybe only organization members can do that).

Contributor

dwf commented Apr 9, 2011

The exact same issue causes #131. This should also get a "FakeModule" label, but I can't figure out how to add it myself (maybe only organization members can do that).

@fperez

This comment has been minimized.

Show comment
Hide comment
@fperez

fperez Apr 9, 2011

Member

On Sat, Apr 9, 2011 at 1:31 PM, dwf
reply@reply.github.com
wrote:

The exact same issue causes #131. This should also get a "FakeModule" label, but I can't figure out how to add it myself (maybe only organization members can do that).

Done, thanks! Unfortunately github only allows repo owners to set labels...

Member

fperez commented Apr 9, 2011

On Sat, Apr 9, 2011 at 1:31 PM, dwf
reply@reply.github.com
wrote:

The exact same issue causes #131. This should also get a "FakeModule" label, but I can't figure out how to add it myself (maybe only organization members can do that).

Done, thanks! Unfortunately github only allows repo owners to set labels...

@takluyver

This comment has been minimized.

Show comment
Hide comment
@takluyver

takluyver Apr 15, 2011

Member

I've had a go at this, if people want to test my pickle-interactive branch. See pull request #384 for more explanation.

Member

takluyver commented Apr 15, 2011

I've had a go at this, if people want to test my pickle-interactive branch. See pull request #384 for more explanation.

@fperez

This comment has been minimized.

Show comment
Hide comment
@fperez

fperez Apr 15, 2011

Member

Thanks, Thomas. BTW, let's not forget to close this, #131 and anything else on this front when we merge #384 (which I think we will shortly).

Member

fperez commented Apr 15, 2011

Thanks, Thomas. BTW, let's not forget to close this, #131 and anything else on this front when we merge #384 (which I think we will shortly).

@ghost ghost assigned takluyver May 11, 2011

@takluyver takluyver referenced this issue Oct 14, 2011

Merged

Usermod #648

@fperez fperez closed this in a1e4911 Nov 27, 2011

stefanv pushed a commit to stefanv/ipython that referenced this issue Nov 30, 2011

Use user_ns as global namespace if it is passed without user_module, …
…and add test for pickling interactively defined objects.


Closes gh-29

minrk pushed a commit to minrk/ipython that referenced this issue Jul 1, 2013

Merge pull request #29 from ivanov/write-py
added 'py' format, which is cleaner than PyWriter

mattvonrocketstein pushed a commit to mattvonrocketstein/ipython that referenced this issue Nov 3, 2014

Use user_ns as global namespace if it is passed without user_module, …
…and add test for pickling interactively defined objects.


Closes gh-29

mattvonrocketstein pushed a commit to mattvonrocketstein/ipython that referenced this issue Nov 3, 2014

Merge pull request #648 from takluyver/usermod
Clean up handling of global namespaces with the proper semantics.

A global namespace should always be tied to a module: pickle accesses classes via the module in which they're defined. So I've changed the arguments for instantiating an InteractiveShell to include `user_module` in place of `user_global_ns`. The global namespace simply becomes a reference to `user_module.__dict__`.

For instantiating InteractiveShell, there are four possibilities:

* Neither `user_ns` nor `user_module` is given. A new (real) module is created named `__main__`, and its `__dict__` becomes the global and local namespace. This is what happens when starting IPython normally.
* Only `user_module` is given. Its `__dict__` becomes the global and local namespace.
* Both `user_ns` and `user_module` are given. `user_module.__dict__` is the global namespace, and `user_ns` is the local namespace. Note that we can't interactively define closures over variables in the local namespace (this seems to be a limitation of Python).
* Only `user_ns` is given. It is treated as the global and local namespace, and a `DummyMod` object is created to refer to it. This is intended as a convenience, especially for the test suite. The recommended way to pass in a single global namespace is as a reference to the module.

`embed()` digs out the locals and the module from the frame in which it's called.

Closes gh-29, closes gh-693.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment