add pickleutil.use_dill

[minrk]

and DirectView.use_dill to call it locally and on all engines

example notebook added. I tried to include tests, but something about the test environment causes problems with dill.

closes #4551

[mrocklin]

I've also had issues with dill and testing. See uqfoundation/dill#16

[Carreau]

I thought cPickle was the default on Py3, does this work than ?

[takluyver]

To expand: import pickle on Python 3 will try to load the C pickle implementation, and fall back to the Python implementation if that fails for some reason.

Possibly we should be setting dill itself as the pickle implementation? @mmckerns, any tips?

[mmckerns]

To take advantage of dill, mpi4py abstracts the pickle import, so you can do this:

>>> from mpi4py import MPI
>>> MPI._p_pickle.dumps = dill.dumps
>>> MPI._p_pickle.loads = dill.loads

It requires the user to import dill on their own first.

So you might be able to make a singleton somewhere with the module or with the dumps/loads functions.

[minrk]

This singleton does already exist (well, in two places, so 'doubleton'?). I just need to change the mod.pickle = pickle to mod.pickle = dill

[minrk]

Will merge soon unless anyone wants to do further review.

[mrocklin]

One could generalize this away from dill by instead making a serialize_with(...) method that took a serialization library, e.g.

import dill
view.serialize_with(dill)

But, given the small nature of this fix and the small number of fully functional serializers out there, this may be overly farsighted.

[Carreau]

+1