Serialize methods and closures

[mrocklin]

IPython+Pickle can't serialize class methods.

>>> from IPython.parallel import Client
>>> view = Client()[:]
>>> view.map(str.split, ['hello world', 'foo bar'])

PicklingError: Can't pickle <type 'method_descriptor'>: attribute lookup __builtin__.method_descriptor failed

IPython+Pickle can't serialze closures

/home/mrocklin/Software/anaconda/lib/python2.7/site-packages/IPython/utils/codeutil.pyc in reduce_code(co)
     32 def reduce_code(co):
     33     if co.co_freevars or co.co_cellvars:
---> 34         raise ValueError("Sorry, cannot pickle code objects with closures")
     35     args =  [co.co_argcount, co.co_nlocals, co.co_stacksize,
     36             co.co_flags, co.co_code, co.co_consts, co.co_names,

ValueError: Sorry, cannot pickle code objects with closures

The dill project (CC @mmckerns) can serialize code in both of these cases. Is there a way to leverage dill here?

One somewhat hacky option would be to fall back on pickle rather than cPickle in case of failure. Dill is able to register its serialization routines with the pure Python implementation.

In [1]: import pickle
In [2]: pickle.dumps(str.split)
TypeError: can't pickle method_descriptor objects

In [3]: import dill
In [4]: pickle.dumps(str.split)  # Note: still using pickle interface
Out[4]: 'cdill.dill\n_getattr\np0\n(cdill.dill\n_load_type\np1\n(S\'StringType\'\np2\ntp3\nRp4\nS\'split\'\np5\nS"<method \'split\' of \'str\' objects>"\np6\ntp7\nRp8\n.'

[minrk]

Yes, there is. Since IPython (mostly) just uses pickle, import dill gets you 99% there. I've illustrated this here. I've been meaning to add a public use_dill API that will enable this cleanly and clearly.

[mrocklin]

Great! Thanks for the pointer @minrk . A use_dill routine in some utils file would be appreciated. While the process outlined in your notebook is straightforward it's not the sort of thing I'll want to remember each time I have to do it.

[minrk]

I'll assign this issue to me, so that I remember to make that use_dill entry point

[minrk]

PR #4552 lets you do rc[:].use_dill() to enable dill everywhere.

[mmckerns]

@minrk: awesome.

[mrocklin]

What remains to be done on this?

[minrk]

Nothing, just waiting for someone to sign off on #4552.

[mmckerns]

@minrk: I was thinking this also might be useful to you guys.

Use dill to get the source code of interactively defined functions, classes, class methods, lambdas, and the like. Then can serialize to a stream or temp file… or just display the source. The beginnings of cracking open a live object and then editing the source and repacking it. like smalltalk. it pretty much works ok on a lot of stuff.

http://stackoverflow.com/questions/427453/how-can-i-get-the-source-code-of-a-python-function/21339166#21339166
https://github.com/uqfoundation/dill/blob/master/tests/test_source.py
https://github.com/uqfoundation/dill/blob/master/tests/test_source.py