Skip to content
This repository

cPickle works in standard interpreter, but not in IPython #29

Closed
ipython opened this Issue May 10, 2010 · 17 comments

5 participants

IPython: interactive computing in Python Thomas Kluyver Fernando Perez Min RK David Warde-Farley
IPython: interactive computing in Python
Owner
ipython commented May 10, 2010

Original Launchpad bug 363115: https://bugs.launchpad.net/ipython/+bug/363115
Reported by: reckoner (reckoner).

For example in IPython:

class Mylist(list):
     def __init__(self,x=[]):
          list.__init__(self,x)

from cPickle import dumps
w=Mylist([1,2,3])
dumps(w)

PicklingError: Can't pickle <class '__main__.Mylist'>: attribute
lookup __main__.Mylist failed

However, using the standard Python interpreter:

 Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32
 Type "help", "copyright", "credits" or "license" for more information.
 >>> from cPickle import dumps
 >>> class Mylist(list):
 ...         def __init__(self,x=[]):
 ...             list.__init__(self,x)
 ...
 >>> w=Mylist([1,2,3])
 >>> dumps(w)
 'ccopy_reg\n_reconstructor\np1\n(c__main__\nMylist\np2\nc__builtin__\nlist\np3\n
 (lp4\nI1\naI2\naI3\natRp5\n.'
 >>>
IPython: interactive computing in Python
Owner
ipython commented May 10, 2010

[ LP comment 1 by: avdd, on 2009-07-16 03:48:57.797382+00:00 ]

Has nothing to do with pickle, but the FakeModule that is registered as main:

$ python
Python 2.5.2 (r252:60911, Oct 5 2008, 19:24:49)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import sys
sys.modules[name].foo = 1
foo
1

a$ ipython

In [1]: import sys

In [2]: sys.modules[name].foo = 1

In [3]: foo

NameError Traceback (most recent call last)

/home/avdd/ in ()

NameError: name 'foo' is not defined

In [4]:

IPython: interactive computing in Python
Owner
ipython commented May 10, 2010

[ LP comment 2 by: Matt Hickford, on 2009-07-18 19:36:43.821085+00:00 ]

I have problem that neither pickle nor cPickle work in ipython (but work in python proper).

In my case pickle.dump works but pickle.

This is Jaunty, Python 2.6

My error:
AttributeError: 'FakeModule' object has no attribute 'Dictionary'

(I have a custom class inherited from list called Dictionary)

IPython: interactive computing in Python
Owner
ipython commented May 10, 2010

[ LP comment 3 by: Ville M. Vainio, on 2009-07-18 20:11:20+00:00 ]

On Sat, Jul 18, 2009 at 10:36 PM, Matt
Hickfordmatt.hickford+launchpad@gmail.com wrote:

I have problem that neither pickle nor cPickle work in ipython (but work
in python proper).

In my case pickle.dump works but pickle.

I think you'll note that pickle works when the class of the pickleable
object is not declared interactively, but rather imported from module.

Ville M. Vainio
http://tinyurl.com/vainio

IPython: interactive computing in Python
Owner
ipython commented May 10, 2010

[ LP comment 4 by: reckoner, on 2009-07-19 23:50:39+00:00 ]

If you get the class using the %run magic instead of importing, you DO
get the pickle failure. I don't know if doing %run is the same as
defining a class interactively, though.

Any guidance appreciated.

thanks.

On Sat, Jul 18, 2009 at 1:11 PM, Ville M. Vainiovivainio@gmail.com wrote:

On Sat, Jul 18, 2009 at 10:36 PM, Matt
Hickfordmatt.hickford+launchpad@gmail.com wrote:

I have problem that neither pickle nor cPickle work in ipython (but work
in python proper).

In my case pickle.dump works but pickle.

I think you'll note that pickle works when the class of the pickleable
object is not declared interactively, but rather imported from module.

Ville M. Vainio
http://tinyurl.com/vainio

cPickle works in standard interpreter, but not in IPython
https://bugs.launchpad.net/bugs/363115
You received this bug notification because you are a direct subscriber
of the bug.

Status in IPython - Enhanced Interactive Python: Confirmed

Bug description:

for example:

In IPython:

class Mylist(list):
      def init(self,x=[]):
          list.init(self,x)

from cPickle import dumps
w=Mylist([1,2,3])
dumps(w)

PicklingError: Can't pickle : attribute
lookup main.Mylist failed

However, using the standard Python interpreter:

Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.

from cPickle import dumps
class Mylist(list):
...         def init(self,x=[]):
...             list.init(self,x)
...
w=Mylist([1,2,3])
dumps(w)
'ccopy_reg\n_reconstructor\np1\n(c__main__\nMylist\np2\nc__builtin__\nlist\np3\n
(lp4\nI1\naI2\naI3\natRp5\n.'

IPython: interactive computing in Python
Owner
ipython commented May 10, 2010

[ LP comment 5 by: eteq, on 2009-12-17 01:34:20.367155+00:00 ]

I've encountered this in the past - I always just define any class I want to pickle in a file and then reload the module as necessary instead of pickling outputs from run or interactively-defined objects. That seems safer anyway - otherwise if you restart the interpreter, the methods on the object are forgotten.

It seems to work fine if you pickle from a file and then load that file with an interactively-defined or defined in "run" version, though.

IPython: interactive computing in Python
Owner
ipython commented May 10, 2010

[ LP comment 6 by: Zak Stone, on 2009-12-17 03:02:24.672555+00:00 ]

For me, a similar problem arises when I am simply unpickling dictionaries in ipython -- no custom classes involved.

Min RK
Owner
minrk commented March 22, 2011

This means that interactively defined functions cannot be used with multiprocessing, as described in #86

Min RK
Owner
minrk commented March 28, 2011

The problem appears to be that the FakeModule used for sys.modules['__main__'] is updated once from user_ns, but it is not linked. If anyone knows of an easy way to effectively do:
FakeModule.dict = user_ns
Then we should be set. Otherwise, I can get pickling to work with the following unattractive hack:

In [1]: from IPython.core.fakemodule import init_fakemod_dict
In [2]: ip = get_ipython()
In [3]: fakemain = sys.modules['__main__']
In [4]: ip.register_post_execute(lambda : init_fakemod_dict(fakemain, ip.user_ns))

This manually syncs the FakeModule in __main__ after each execution. It is expensive and ugly because it involves a clear+update of the entire user_ns after each execution, but for users who need to pickle interactively defined objects before we figure out a clean solution, it should work as a band-aid solution.

Fernando Perez
Owner

Thanks for the hack, Min. This is a really important one to fix, though I'm reluctant to make it a blocker because it's hard work that may delay us for a while, and it's such an old bug that we've obviously managed to live with it.

But I'd be thrilled to see a proper fix for this.

Min RK
Owner
minrk commented April 08, 2011

@fperez do you know if there is a way for us to just have the FM.dict softlink to user_ns, rather than doing a one-time copy?

I couldn't figure out a way in my relatively brief exploration of this, but it seems like that would resolve the whole problem.

Fernando Perez
Owner
Min RK
Owner
minrk commented April 08, 2011

Right, but it's specifically the __main__ module, which is the running namespace.

I certainly don't know enough about what __main__ is used for, and what we should do about it.

Fernando Perez
Owner
David Warde-Farley
dwf commented April 09, 2011

The exact same issue causes #131. This should also get a "FakeModule" label, but I can't figure out how to add it myself (maybe only organization members can do that).

Fernando Perez
Owner
Thomas Kluyver
Collaborator

I've had a go at this, if people want to test my pickle-interactive branch. See pull request #384 for more explanation.

Fernando Perez
Owner

Thanks, Thomas. BTW, let's not forget to close this, #131 and anything else on this front when we merge #384 (which I think we will shortly).

Thomas Kluyver takluyver referenced this issue from a commit July 30, 2011
Commit has since been removed from the repository and is no longer available.
Thomas Kluyver takluyver referenced this issue from a commit July 30, 2011
Commit has since been removed from the repository and is no longer available.
Thomas Kluyver takluyver referenced this issue from a commit October 04, 2011
Commit has since been removed from the repository and is no longer available.
Thomas Kluyver takluyver referenced this issue October 14, 2011
Merged

Usermod #648

Fernando Perez fperez closed this in a1e4911 November 26, 2011
Stefan van der Walt stefanv referenced this issue from a commit in stefanv/ipython July 30, 2011
Thomas Kluyver Use user_ns as global namespace if it is passed without user_module, …
…and add test for pickling interactively defined objects.

Closes gh-29
b460945
Brian E. Granger ellisonbg referenced this issue from a commit January 10, 2012
Commit has since been removed from the repository and is no longer available.
Fernando Perez fperez referenced this issue from a commit January 10, 2012
Commit has since been removed from the repository and is no longer available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.