CannedArray fails to pickle due to memoryview

[jakirkham]

Have seen this once in code that normally doesn't produce this. Using Jupyter/iPython cluster. Related question ( http://stackoverflow.com/q/27596463 ).

[mmckerns]

Can you post a reduced sample of your code, and the traceback, as well as the versions of python and modules you are using? I don't see the error you are linking to, but something else. For example:

>>> import numpy as np
>>> import dill
>>> from ipykernel.pickleutil import CannedArray
>>> a = np.array([1,2,3,4])
>>> ca = CannedArray(a)
>>> _ca = dill.dumps(ca)
>>> _ca
'\x80\x02cipykernel.pickleutil\nCannedArray\nq\x00)\x81q\x01}q\x02(U\x05dtypeq\x03U\x03<i8q\x04U\x05shapeq\x05K\x04\x85q\x06U\x07pickledq\x07\x89U\x07buffersq\x08]q\tcdill.dill\n_load_type\nq\nU\nBufferTypeq\x0b\x85q\x0cRq\r)\x81q\x0eaub.'
>>> ca_ = dill.loads(_ca)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mmckerns/lib/python2.7/site-packages/dill-0.2.5.dev0-py2.7.egg/dill/dill.py", line 214, in loads
    return load(file)
  File "/Users/mmckerns/lib/python2.7/site-packages/dill-0.2.5.dev0-py2.7.egg/dill/dill.py", line 204, in load
    obj = pik.load()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1083, in load_newobj
    obj = cls.__new__(cls, *args)
TypeError: buffer() takes at least 1 argument (0 given)

This is using:

>>> sys.version
'2.7.10 (default, Sep  2 2015, 17:36:25) \n[GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)]'
>>> np.__version__
'1.9.2'
>>> IPython.__version__
'4.0.0'
>>> dill.__version__
'0.2.5.dev0'

I get this same error regardless of the dill.settings I have used.

However, I'm not sure this is as much of a dill issues as it might be a ipython.parallel issue. In the link you gave, I don't even see dill used.

[mmckerns]

This isn't a bug, it's at the very least a feature request.

[jakirkham]

You are correct that this comes up when using ipyparallel. I am using the use_dill method on all of the the engines. So pickling should be with dill.

Unfortunately coming up with a small reproducible example will be tricky. Part of the problem is this seems to be a sporadic bug that I run into with a somewhat low frequency of occurrence. As you have already pointed out it is probably buried in the interaction between ipyparallel and dill. However, the code that sometimes runs into this error is using ipyparallel at a very high level and is only encouraging the use of dill. It never explicitly creates a CannedArray, but engines do return numpy.ndarrays.

I have not had enough time to explore the potential causes. However, the fact that it isn't a consistent failure made me think it was a bug and that it may be due to a race case. The last part makes me think that this is an ipyparallel bug.

[jakirkham]

I have opened an issue on ipyparallel ( ipython/ipyparallel#65 ).

[mmckerns]

If it's an issue that comes up sporadically and infrequently, then it's very likely not a dill issue, as dill is quite deterministic. If you look at a CannedArray, it does have an attribute .pickled, and if it's been added dynamically, then it might get stripped off under the wrong kind of pickling. I'll leave this open, and keep an eye on ipython/ipyparallel#65.

[mmckerns]

Well... looks like ipyparallel closed the above ticket due to it being stale. In going through this again, with updated versions... it seems like the pickling issue has changed. I'm going to assume that since ipyparallel closed the issue that the missing attribute is resolved. However, you can see what happens when I try to pickle a CannedArray...

Python 3.7.15 (default, Oct 12 2022, 04:11:53) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> import dill
>>> from ipykernel.pickleutil import CannedArray
__main__:1: DeprecationWarning: ipykernel.pickleutil is deprecated. It has moved to ipyparallel.
>>> a = np.array([1,2,3,4])
>>> ca = CannedArray(a)
>>> _ca = dill.dumps(ca)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 263, in dumps
    dump(obj, file, protocol, byref, fmode, recurse, **kwds)#, strictio)
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 235, in dump
    Pickler(file, protocol, **_kwds).dump(obj)
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 394, in dump
    StockPickler.dump(self, obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 437, in dump
    self.save(obj)
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
    self.save_reduce(obj=obj, *rv)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
    save(state)
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 1186, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 859, in save_dict
    self._batch_setitems(obj.items())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 885, in _batch_setitems
    save(v)
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
    f(self, obj) # Call unbound method with explicit self
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 819, in save_list
    self._batch_appends(obj)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 846, in _batch_appends
    save(tmp[0])
  File "/Users/mmckerns/lib/python3.7/site-packages/dill/_dill.py", line 388, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 524, in save
    rv = reduce(self.proto)
TypeError: can't pickle memoryview objects
>>> np.__version__
'1.21.6'
>>> dill.__version__
'0.3.6.dev0'
>>> import ipyparallel
>>> ipyparallel.__version__
'7.0.1'

so... as a CannedArray is not a super common item, but there's currently not a ticket open for a memoryview object... I'm going to leave this open (with a slight change in focus).