[Python] Create UnionArray from mixed-type pandas categorical

[asfimport]

While troublehsooting ARROW-2966 I updated my pandas dataframe with more type information. Specifically, I changed some mixed type columns to categorical instead of object. I assumed that the Table.from_pandas() would inspect the pandas type information and respect that when converting it over to a table. It doesn't seem to.

For instance, I expected this code to work, but it throws the same ArrowTypeError as ARROW-2966.

 

import pandas as pd
import pyarrow as pa
import numpy as np
df=pd.DataFrame.from_dict({"col":[0,1,2,3,""]},dtype="category")
tb = pa.Table.from_pandas(df, columns=["col"])

 

Reporter: Christopher Brooks

Related issues:

• [Python] Implement pa.array() with type=union type (is related to)

_{Note: This issue was originally created as ARROW-3030. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
Thought we had implemented this. Confirmed it's an issue; fixing should not be too difficult

[asfimport]

Wes McKinney / @wesm:
PRs welcome of course

[asfimport]

Krisztian Szucs / @kszucs:
The example can be reduced to:

pa.array([0, 1, 2, 3, ''])

Creating unions is not implemented yet, reference https://github.com/apache/arrow/blob/master/cpp/src/arrow/python/inference.cc#L377

[asfimport]

Krisztian Szucs / @kszucs:
Type inference may involve optional checks whether the candidate types are castable to a single one rather than creating a union of them.
OTOH In certain cases cast checks requires the value too, which can make the inference expensive.

What would the expected type outcome for the following cases:

a1 = [1.2, 3.4, "4.6"]
a2 = [b"binary", "string"]
a3 = [[1, 2, 3], [4.0]]
a4 = ["2018-01-05", datetime.data(2018, 01, 05)]

[asfimport]

Wes McKinney / @wesm:
Moved to 0.12. Dealing with unions is out of scope for 0.11, but maybe we can look into it for 0.12 or 0.13

[asfimport]

Wes McKinney / @wesm:
Moved out of 0.12. We don't have much support for unions yet, which is what would be needed here. That's a bigger project

In the meantime I suggest homogenizing the types of your data to not have a mix of integers and strings

[asfimport]

Antoine Pitrou / @pitrou:
This sounds undesirable to do by default for the same reason as for ARROW-2774.

[asfimport]

Antoine Pitrou / @pitrou:
cc @jorisvandenbossche

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
I think the same discussion as in ARROW-2774 (supporting or not? optional?) for general conversion applies to Categorical as well, so let's close this referring to ARROW-2774