cPickle works in standard interpreter, but not in IPython

[ipython]

Original Launchpad bug 363115: https://bugs.launchpad.net/ipython/+bug/363115
Reported by: reckoner (reckoner).

For example in IPython:

class Mylist(list):
     def __init__(self,x=[]):
          list.__init__(self,x)

from cPickle import dumps
w=Mylist([1,2,3])
dumps(w)

PicklingError: Can't pickle <class '__main__.Mylist'>: attribute
lookup __main__.Mylist failed

However, using the standard Python interpreter:

 Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on win32
 Type "help", "copyright", "credits" or "license" for more information.
 >>> from cPickle import dumps
 >>> class Mylist(list):
 ...         def __init__(self,x=[]):
 ...             list.__init__(self,x)
 ...
 >>> w=Mylist([1,2,3])
 >>> dumps(w)
 'ccopy_reg\n_reconstructor\np1\n(c__main__\nMylist\np2\nc__builtin__\nlist\np3\n
 (lp4\nI1\naI2\naI3\natRp5\n.'
 >>>

[ipython]

[ LP comment 1 by: avdd, on 2009-07-16 03:48:57.797382+00:00 ]

Has nothing to do with pickle, but the FakeModule that is registered as main:

$ python
Python 2.5.2 (r252:60911, Oct 5 2008, 19:24:49)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import sys
sys.modules[name].foo = 1
foo
1

a$ ipython

In [1]: import sys

In [2]: sys.modules[name].foo = 1

In [3]: foo

NameError Traceback (most recent call last)

/home/avdd/ in ()

NameError: name 'foo' is not defined

In [4]:

[ipython]

[ LP comment 2 by: Matt Hickford, on 2009-07-18 19:36:43.821085+00:00 ]

I have problem that neither pickle nor cPickle work in ipython (but work in python proper).

In my case pickle.dump works but pickle.

This is Jaunty, Python 2.6

My error:
AttributeError: 'FakeModule' object has no attribute 'Dictionary'

(I have a custom class inherited from list called Dictionary)

[ipython]

[ LP comment 3 by: Ville M. Vainio, on 2009-07-18 20:11:20+00:00 ]

On Sat, Jul 18, 2009 at 10:36 PM, Matt
Hickfordmatt.hickford+launchpad@gmail.com wrote:

I have problem that neither pickle nor cPickle work in ipython (but work
in python proper).

In my case pickle.dump works but pickle.

I think you'll note that pickle works when the class of the pickleable
object is not declared interactively, but rather imported from module.

Ville M. Vainio
http://tinyurl.com/vainio

[ipython]

[ LP comment 4 by: reckoner, on 2009-07-19 23:50:39+00:00 ]

If you get the class using the %run magic instead of importing, you DO
get the pickle failure. I don't know if doing %run is the same as
defining a class interactively, though.

Any guidance appreciated.

thanks.

On Sat, Jul 18, 2009 at 1:11 PM, Ville M. Vainiovivainio@gmail.com wrote:

On Sat, Jul 18, 2009 at 10:36 PM, Matt
Hickfordmatt.hickford+launchpad@gmail.com wrote:

I have problem that neither pickle nor cPickle work in ipython (but work
in python proper).

In my case pickle.dump works but pickle.

I think you'll note that pickle works when the class of the pickleable
object is not declared interactively, but rather imported from module.

Ville M. Vainio
http://tinyurl.com/vainio

cPickle works in standard interpreter, but not in IPython
https://bugs.launchpad.net/bugs/363115
You received this bug notification because you are a direct subscriber
of the bug.

Status in IPython - Enhanced Interactive Python: Confirmed

Bug description:

for example:

In IPython:

class Mylist(list):
      def init(self,x=[]):
          list.init(self,x)

from cPickle import dumps
w=Mylist([1,2,3])
dumps(w)

PicklingError: Can't pickle <class 'main.Mylist'>: attribute
lookup main.Mylist failed

However, using the standard Python interpreter:

Python 2.5.2 (r252:60911, Feb 21 2008, 13:11:45) [MSC v.1310 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.

from cPickle import dumps
class Mylist(list):
...         def init(self,x=[]):
...             list.init(self,x)
...
w=Mylist([1,2,3])
dumps(w)
'ccopy_reg\n_reconstructor\np1\n(c__main__\nMylist\np2\nc__builtin__\nlist\np3\n
(lp4\nI1\naI2\naI3\natRp5\n.'

[ipython]

[ LP comment 5 by: eteq, on 2009-12-17 01:34:20.367155+00:00 ]

I've encountered this in the past - I always just define any class I want to pickle in a file and then reload the module as necessary instead of pickling outputs from run or interactively-defined objects. That seems safer anyway - otherwise if you restart the interpreter, the methods on the object are forgotten.

It seems to work fine if you pickle from a file and then load that file with an interactively-defined or defined in "run" version, though.

[ipython]

[ LP comment 6 by: Zak Stone, on 2009-12-17 03:02:24.672555+00:00 ]

For me, a similar problem arises when I am simply unpickling dictionaries in ipython -- no custom classes involved.

[minrk]

This means that interactively defined functions cannot be used with multiprocessing, as described in #86

[minrk]

The problem appears to be that the FakeModule used for sys.modules['__main__'] is updated once from user_ns, but it is not linked. If anyone knows of an easy way to effectively do:
FakeModule.dict = user_ns
Then we should be set. Otherwise, I can get pickling to work with the following unattractive hack:

In [1]: from IPython.core.fakemodule import init_fakemod_dict
In [2]: ip = get_ipython()
In [3]: fakemain = sys.modules['__main__']
In [4]: ip.register_post_execute(lambda : init_fakemod_dict(fakemain, ip.user_ns))

This manually syncs the FakeModule in __main__ after each execution. It is expensive and ugly because it involves a clear+update of the entire user_ns after each execution, but for users who need to pickle interactively defined objects before we figure out a clean solution, it should work as a band-aid solution.

[fperez]

Thanks for the hack, Min. This is a really important one to fix, though I'm reluctant to make it a blocker because it's hard work that may delay us for a while, and it's such an old bug that we've obviously managed to live with it.

But I'd be thrilled to see a proper fix for this.

[minrk]

@fperez do you know if there is a way for us to just have the FM.dict softlink to user_ns, rather than doing a one-time copy?

I couldn't figure out a way in my relatively brief exploration of this, but it seems like that would resolve the whole problem.

[fperez]

On Fri, Apr 8, 2011 at 9:49 AM, minrk
reply@reply.github.com
wrote:

@fperez do you know if there is a way for us to just have the FM.dict softlink to user_ns, rather than doing a one-time copy?

I couldn't figure out a way in my relatively brief exploration of this, but it seems like that would resolve the whole problem.

My concern would be the fact that this would couple a module (a global
object) to the running namespace. That makes a mess with embedding,
multiple shells instantiated in the test suite, etc.

The FakeModule thing is something I wrote years ago, at the very
beginning of ipython, to get some namespace issues working. Perhaps
we should just revisit the whole question fresh, and see if we can get
%run and other namespace-intensive things working, but with a
different approach that has a better profile regarding pickling...

[minrk]

Right, but it's specifically the __main__ module, which is the running namespace.

I certainly don't know enough about what __main__ is used for, and what we should do about it.

[fperez]

On Fri, Apr 8, 2011 at 10:46 AM, minrk
reply@reply.github.com
wrote:

Right, but it's specifically the __main__ module, which is the running namespace.

I certainly don't know enough about what __main__ is used for, and what we should do about it.

That's why I'm reluctant to make this one a blocker right now: it's
very delicate and affects a lot of things in how everything operates.
The best might be to come up with a few tests around all of this, add
them including failing ones as KnownFailure, and at least we'll have
coverage of the issue from the test perspective. Then post 0.11, we
can take a stab at actually fixing this, with some time to think
carefully about the entire issue without the rush of a release.

Cheers,

f

[dwf]

The exact same issue causes #131. This should also get a "FakeModule" label, but I can't figure out how to add it myself (maybe only organization members can do that).

[fperez]

On Sat, Apr 9, 2011 at 1:31 PM, dwf
reply@reply.github.com
wrote:

The exact same issue causes #131. This should also get a "FakeModule" label, but I can't figure out how to add it myself (maybe only organization members can do that).

Done, thanks! Unfortunately github only allows repo owners to set labels...

[takluyver]

I've had a go at this, if people want to test my pickle-interactive branch. See pull request #384 for more explanation.

[fperez]

Thanks, Thomas. BTW, let's not forget to close this, #131 and anything else on this front when we merge #384 (which I think we will shortly).