[Python] Devise a means to serialize arrays of arbitrary Python objects in Arrow IPC messages

[asfimport]

Practically speaking, this would involve a "custom" logical type that is "pyobject", represented physically as an array of 64-bit pointers. On serialization, this would need to be converted to a BinaryArray containing pickled objects as binary values

At the moment, we don't yet have the machinery to deal with "custom" types where the in-memory representation is different from the on-wire representation. This would be a useful use case to work through the design issues

Interestingly, if done properly, this would enable other Arrow implementations to manipulate (filter, etc.) serialized Python objects as binary blobs.

Reporter: Wes McKinney / @wesm

_{Note: This issue was originally created as ARROW-823. Please see the migration documentation for further details.}

[asfimport]

Uwe Korn / @xhochy:
For me this issue depends also on how we represent custom/shared Pandas/Python metadata in Arrow and Parquet. While this will not fully solves the issue, I think that discussion is one of the requirements.

[asfimport]

Wes McKinney / @wesm:
This can be implemented now with ExtensionType

[asfimport]

Antoine Pitrou / @pitrou:
cc @jorisvandenbossche

[asfimport]

Antoine Pitrou / @pitrou:
I think we should directly people towards the standard pickle module. It's not obvious that putting Python objects in an Arrow array would bring a tangible improvement.