[Python] Incorrect inferred schema from pandas dataframe with length 0.

[asfimport]

We use pandas(with pyarrow engine) to write out parquet files and those outputs will be consumed by other applications such as Java apps using org.apache.parquet.hadoop.ParquetFileReader. We found that some empty dataframes would get incorrect schema for string columns in other applications. After some investigation, we narrow down the issue to the schema inference by pyarrow:

In [1]: import pandas as pd
In [2]: df = pd.DataFrame([['a', 1, 1.0]], columns=['a', 'b', 'c'])
In [3]: import pyarrow as pa
In [4]: pa.Schema.from_pandas(df)
 Out[4]:
 a: string
 b: int64
 c: double
 -- schema metadata --
 pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' + 562
In [5]: pa.Schema.from_pandas(df.head(0))
 Out[5]:
 a: null
 b: int64
 c: double
 -- schema metadata --
 pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' + 560
In [6]: pa._version_
 Out[6]: '5.0.0'

 As you can see, the column 'a' which should be string type if inferred as null type and is converted to int32 while writing to parquet files.

Is this an expected behavior? Or do we have any workaround for this issue? Could anyone take a look please. Thanks!

Environment: OS: Windows 10, CentOS 7
Reporter: Yuan Zhou

_{Note: This issue was originally created as ARROW-14488. Please see the migration documentation for further details.}

[asfimport]

Alenka Frim / @AlenkaF:
Hi [~zijie0] , thank you for reporting! And sorry for a late reply.

I think this may be a bug on Arrow side: when constructing metadata in dataframe_to_types ({}pandas_compat.py{}) the conversion from empty Pandas series to pa.array is wrong in the case of a string dtype. Here is an example:

>>> import pandas as pd
>>> import pyarrow as pa
>>> df = pd.DataFrame([['a', 1, 1.0]], columns=['a', 'b', 'c'])
>>> df
   a  b    c
0  a  1  1.0
>>> df["a"]
0    a
Name: a, dtype: object

# Non-empty dataframe
>>> pa.array(df["a"], from_pandas=True) # Works for non-empty dataframe
<pyarrow.lib.StringArray object at 0x12462ed00>
[
  "a"
]
>>> pa.array(df["a"], from_pandas=True).type
DataType(string)

# Empty dataframe
>>> pa.array(df["a"].head(0), from_pandas=True) # Becomes NullArray with no dtype in case of string/object
<pyarrow.lib.NullArray object at 0x12462eac0>
0 nulls
>>> pa.array(df["a"].head(0), from_pandas=True).type
DataType(null)

but that doesn't happen for integer or double:

>>> df["b"]
0    1
Name: b, dtype: int64

>>> pa.array(df["b"], from_pandas=True)
<pyarrow.lib.Int64Array object at 0x12462eac0>
[
  1
]

>>> pa.array(df["b"].head(0), from_pandas=True)
<pyarrow.lib.Int64Array object at 0x12462ea60>
[]
>>> pa.array(df["b"].head(0), from_pandas=True).type
DataType(int64)

@jorisvandenbossche what do you think?

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:

the conversion from empty Pandas series to pa.array is wrong in the case of a string dtype.

The main problem is that the example code is not using a "string dtype". By default, pandas uses the generic "object" dtype to store strings. But this data type basically means that it can hold any Python object. So it is not guaranteed to be strings (eg it could also be decimals, bytes, .., for some python types that pyarrow also infers).

As long as the array is not empty, the conversion to a pyarrow array will try to infer the appropriate type based on the values in the input array (eg in case of an object dtype array with strings, it will indeed convert that to a pa.string() type). But if the array is empty, there are no values to infer the type from. And that is the reason why pyarrow defaults to use the generic "null" data type for such array (or column in a DataFrame).

If you know that you have strings for a certain column, and want the pandas->pyarrow conversion to robustly work (regardless of having empty dataframes/arrays), the from_pandas method has a schema argument, and this way you can specific a schema to use (and so pyarrow will not try to infer the types based on the values in the array). You will have to construct this schema manually, though, in this case.

[asfimport]

Alenka Frim / @AlenkaF:
Thank you Joris!

An example would be:

>>> import pandas as pd
>>> df = pd.DataFrame([['a', 1, 1.0]], columns=['a', 'b', 'c'])
>>> import pyarrow as pa
>>> 
>>> schema = pa.schema([
...    pa.field('a', pa.string()),
...    pa.field('b', pa.int64()),
...    pa.field('c', pa.float64())])
>>> 
>>> pa.Table.from_pandas(df, schema=schema)
pyarrow.Table
a: string
b: int64
c: double
----
a: [["a"]]
b: [[1]]
c: [[1]]
>>> pa.Table.from_pandas(df.head(0), schema=schema)
pyarrow.Table
a: string
b: int64
c: double
----
a: [[]]
b: [[]]
c: [[]]

[thisisnic]

Closing this as it is now fixed as of PyArrow 23.0.1 - the empty string column now correctly is inferred to be a string and not null.