Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

cPickle works in standard interpreter, but not in IPython #29

Closed
ipython opened this Issue · 17 comments

5 participants

IPython: interactive computing in Python Min RK Fernando Perez David Warde-Farley Thomas Kluyver
IPython: interactive computing in Python

Original Launchpad bug 363115: https://bugs.launchpad.net/ipython/+bug/363115
Reported by: reckoner (reckoner).

For example in IPython:

class Mylist(list):
     def __init__(self,x=[]):
          list.__init__(self,x)

from cPickle import dumps
w=Mylist([1,2,3])
dumps(w)

PicklingError: Can't pickle <class '__main__.Mylist'>: attribute
lookup __main__.Mylist failed

However, using the standard Python interpreter:

 Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32
 Type "help", "copyright", "credits" or "license" for more information.
 >>> from cPickle import dumps
 >>> class Mylist(list):
 ...         def __init__(self,x=[]):
 ...             list.__init__(self,x)
 ...
 >>> w=Mylist([1,2,3])
 >>> dumps(w)
 'ccopy_reg\n_reconstructor\np1\n(c__main__\nMylist\np2\nc__builtin__\nlist\np3\n
 (lp4\nI1\naI2\naI3\natRp5\n.'
 >>>
IPython: interactive computing in Python

[ LP comment 1 by: avdd, on 2009-07-16 03:48:57.797382+00:00 ]

Has nothing to do with pickle, but the FakeModule that is registered as main:

$ python
Python 2.5.2 (r252:60911, Oct 5 2008, 19:24:49)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import sys
sys.modules[name].foo = 1
foo
1

a$ ipython

In [1]: import sys

In [2]: sys.modules[name].foo = 1

In [3]: foo

NameError Traceback (most recent call last)

/home/avdd/ in ()

NameError: name 'foo' is not defined

In [4]:

IPython: interactive computing in Python

[ LP comment 2 by: Matt Hickford, on 2009-07-18 19:36:43.821085+00:00 ]

I have problem that neither pickle nor cPickle work in ipython (but work in python proper).

In my case pickle.dump works but pickle.

This is Jaunty, Python 2.6

My error:
AttributeError: 'FakeModule' object has no attribute 'Dictionary'

(I have a custom class inherited from list called Dictionary)

IPython: interactive computing in Python

[ LP comment 3 by: Ville M. Vainio, on 2009-07-18 20:11:20+00:00 ]

On Sat, Jul 18, 2009 at 10:36 PM, Matt
Hickford wrote:

I have problem that neither pickle nor cPickle work in ipython (but work
in python proper).

In my case pickle.dump works but pickle.

I think you'll note that pickle works when the class of the pickleable
object is not declared interactively, but rather imported from module.

Ville M. Vainio
http://tinyurl.com/vainio

IPython: interactive computing in Python

[ LP comment 4 by: reckoner, on 2009-07-19 23:50:39+00:00 ]

If you get the class using the %run magic instead of importing, you DO
get the pickle failure. I don't know if doing %run is the same as
defining a class interactively, though.

Any guidance appreciated.

thanks.

On Sat, Jul 18, 2009 at 1:11 PM, Ville M. Vainiovivainio@gmail.com wrote:

On Sat, Jul 18, 2009 at 10:36 PM, Matt
Hickford wrote:

I have problem that neither pickle nor cPickle work in ipython (but work
in python proper).

In my case pickle.dump works but pickle.

I think you'll note that pickle works when the class of the pickleable
object is not declared interactively, but rather imported from module.

Ville M. Vainio
http://tinyurl.com/vainio

cPickle works in standard interpreter, but not in IPython
https://bugs.launchpad.net/bugs/363115
You received this bug notification because you are a direct subscriber
of the bug.

Status in IPython - Enhanced Interactive Python: Confirmed

Bug description:

for example:

In IPython:

class Mylist(list):
      def init(self,x=[]):
          list.init(self,x)

from cPickle import dumps
w=Mylist([1,2,3])
dumps(w)

PicklingError: Can't pickle : attribute
lookup main.Mylist failed

However, using the standard Python interpreter:

Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.

from cPickle import dumps
class Mylist(list):
...         def init(self,x=[]):
...             list.init(self,x)
...
w=Mylist([1,2,3])
dumps(w)
'ccopy_reg\n_reconstructor\np1\n(c__main__\nMylist\np2\nc__builtin__\nlist\np3\n
(lp4\nI1\naI2\naI3\natRp5\n.'

IPython: interactive computing in Python

[ LP comment 5 by: eteq, on 2009-12-17 01:34:20.367155+00:00 ]

I've encountered this in the past - I always just define any class I want to pickle in a file and then reload the module as necessary instead of pickling outputs from run or interactively-defined objects. That seems safer anyway - otherwise if you restart the interpreter, the methods on the object are forgotten.

It seems to work fine if you pickle from a file and then load that file with an interactively-defined or defined in "run" version, though.

IPython: interactive computing in Python

[ LP comment 6 by: Zak Stone, on 2009-12-17 03:02:24.672555+00:00 ]

For me, a similar problem arises when I am simply unpickling dictionaries in ipython -- no custom classes involved.

Min RK
Owner

This means that interactively defined functions cannot be used with multiprocessing, as described in #86

Min RK
Owner

The problem appears to be that the FakeModule used for sys.modules['__main__'] is updated once from user_ns, but it is not linked. If anyone knows of an easy way to effectively do:
FakeModule.dict = user_ns
Then we should be set. Otherwise, I can get pickling to work with the following unattractive hack:

In [1]: from IPython.core.fakemodule import init_fakemod_dict
In [2]: ip = get_ipython()
In [3]: fakemain = sys.modules['__main__']
In [4]: ip.register_post_execute(lambda : init_fakemod_dict(fakemain, ip.user_ns))

This manually syncs the FakeModule in __main__ after each execution. It is expensive and ugly because it involves a clear+update of the entire user_ns after each execution, but for users who need to pickle interactively defined objects before we figure out a clean solution, it should work as a band-aid solution.

Fernando Perez
Owner

Thanks for the hack, Min. This is a really important one to fix, though I'm reluctant to make it a blocker because it's hard work that may delay us for a while, and it's such an old bug that we've obviously managed to live with it.

But I'd be thrilled to see a proper fix for this.

Min RK
Owner

@fperez do you know if there is a way for us to just have the FM.dict softlink to user_ns, rather than doing a one-time copy?

I couldn't figure out a way in my relatively brief exploration of this, but it seems like that would resolve the whole problem.

Fernando Perez
Owner
Min RK
Owner

Right, but it's specifically the __main__ module, which is the running namespace.

I certainly don't know enough about what __main__ is used for, and what we should do about it.

Fernando Perez
Owner
David Warde-Farley
dwf commented

The exact same issue causes #131. This should also get a "FakeModule" label, but I can't figure out how to add it myself (maybe only organization members can do that).

Fernando Perez
Owner
Thomas Kluyver
Owner

I've had a go at this, if people want to test my pickle-interactive branch. See pull request #384 for more explanation.

Fernando Perez
Owner

Thanks, Thomas. BTW, let's not forget to close this, #131 and anything else on this front when we merge #384 (which I think we will shortly).

Thomas Kluyver takluyver was assigned
Thomas Kluyver takluyver referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
Thomas Kluyver takluyver referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
Thomas Kluyver takluyver referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
Thomas Kluyver takluyver referenced this issue
Merged

Usermod #648

Fernando Perez fperez closed this in a1e4911
Stefan van der Walt stefanv referenced this issue from a commit in stefanv/ipython
Thomas Kluyver takluyver Use user_ns as global namespace if it is passed without user_module, …
…and add test for pickling interactively defined objects.

Closes gh-29
b460945
Brian E. Granger ellisonbg referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
Fernando Perez fperez referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
matthew von rocketstein mattvonrocketstein referenced this issue from a commit in mattvonrocketstein/ipython
Thomas Kluyver takluyver Use user_ns as global namespace if it is passed without user_module, …
…and add test for pickling interactively defined objects.

Closes gh-29
1eb56f8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.