[Python] Provide Python API for creating user-defined data types that can survive Arrow IPC

[asfimport]

The user will provide:

• Data type subclass that can indicate the physical storage type
• "get state" and "set state" functions for serializing custom metadata to bytes
• An optional function for "boxing" scalar values from the physical array storage

Internally, this will build on an analogous C++ API for defining user data types

Reporter: Wes McKinney / @wesm
Assignee: Antoine Pitrou / @pitrou

Related issues:

• [Python] Define extension type API in Python to "receive" or "send" a foreign extension type (is related to)
• [Python] Add API to map Arrow types (including extension types) to pandas ExtensionArray instances for to_pandas conversions (is depended upon by)

PRs and other links:

• GitHub Pull Request #4532

_{Note: This issue was originally created as ARROW-840. Please see the migration documentation for further details.}

[asfimport]

Robert Nishihara / @robertnishihara:
Is "user-defined" data types intended to include arbitrary custom Python classes?

class Foo(object):
  def __init__(self):
    ...

For example, if we wish to pass a Foo object through Arrow IPC, what would the custom metadata be? Would it be the pickled class definition? Would it be something like the string "Foo"?

[asfimport]

Wes McKinney / @wesm:
Yeah, the idea is to embed basically anything that can be serialized to and from a bytestring. So you could use pickle to do the metadata serialization, but it would go into the IPC metadata as opaque bytes in https://github.com/apache/arrow/blob/master/format/Schema.fbs#L250

[asfimport]

Wes McKinney / @wesm:
This is within reach now because of the C++ extension types

[asfimport]

Wes McKinney / @wesm:
@jorisvandenbossche @pitrou what do you think is the path ahead for this project? There are two distinct areas of work:

• Defining extension types in Python and providing Python exposure for already C++-defined types

• Bridging between extension types (which may be C++ or Python-defined) and pandas

  This issue ARROW-840 covers the former but not the latter. It would be nice to have this feature available in 0.14.0. I'm concerned from today's date (June 11) that we may miss the window for 0.14.0 though

[asfimport]

Antoine Pitrou / @pitrou:
It looks the former should be reasonably doable. I have no idea about the latter.

[asfimport]

Wes McKinney / @wesm:
I think it becomes a pure Python exercise at that point, shouldn't really be any C++ development required. It looks like it's already ARROW-2428

[asfimport]

Wes McKinney / @wesm:
Actually, I'm wrong about that, because the mapped types have to be serialized in arrow_to_pandas.cc, so there is some C++ development work to do there

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
So the first bullet point (enabling "defining extension types in Python) requires to implement a C++ PythonExtensionType that can translate python function callbacks to the actual ExtensionType methods?

I looked into that a bit some time ago, and I think it is above my current C++ skill level (at least to start it). @pitrou is that something that you might want to look at?
Once the basics are there, I am very much interested to further help with this and do further work to enable pandas ExtensionArray interaction.

[asfimport]

Antoine Pitrou / @pitrou:
Just a heads up that I'm working on the first point.

[asfimport]

Wes McKinney / @wesm:
Cool – that's this JIRA so I assigned it to you

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 4532
#4532