-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Python] Canonical fixed-shape tensor extension array/type is not picklable. #35599
Comments
Thank you for the issue @clarkzinzow! Are you interested in making a PR with the proposed solution and a test for it? |
We might even be able to put this on the base ExtensionType class, so that every extension type implementation automatically has this implemented (since this should be generic). The reason for the current error message is that it falls back to the base class DataType reducer which essentially pickles the string repr of the type (which is overridden by many type subclasses): arrow/python/pyarrow/types.pxi Lines 225 to 226 in f6e4479
|
…ovariance matrices as ListColumns (until apache/arrow#35599 is fixed).
…ovariance matrices as ListColumns (until apache/arrow#35599 is fixed).
…ovariance matrices as ListColumns (until apache/arrow#35599 is fixed)
…ovariance matrices as ListColumns (until apache/arrow#35599 is fixed)
Describe the bug, including details regarding any error messages, version, and platform.
The fixed-shape tensor extension type does not appear to be picklable. Given that pickling Arrow data is supported in general and is used in Python-centric systems such as Ray, supporting pickling for canonical extension types/arrays seems reasonable.
Reproduction
Extension Type
raises the error:
Extension Array
raises the ~same error:
Environment
Possible Solution
It seems like we might be able to implement
__reduce__
onFixedShapeTensorType
such that it uses the__arrow_ext_serialize__
serialization protocol? E.g.Component(s)
Python
The text was updated successfully, but these errors were encountered: