[Python] unpickling a pyarrow table with dictionary fields crashes

[asfimport]

The following code crashes on this check:

F1120 07:51:37.523720 12432 array.cc:773]  Check failed: (data->dictionary) != (nullptr) 

 

import cPickle as pickle
import pandas as pd
import pyarrow as pa

df = pd.DataFrame([{"cat": "a", "val":1},{"cat": "b", "val":2} ])
df["cat"] = df["cat"].astype('category')index_table = pa.Table.from_pandas(df, preserve_index=False)

with open('/tmp/zz.pickle', 'wb') as f:
    pickle.dump(index_table, f, protocol=2)

with open('/tmp/zz.pickle', 'rb') as f:
   index_table = pickle.load(f)

 

Used Python2 with the following environment:

Package         Version
--------------- -------
enum34          1.1.6  
futures         3.3.0  
numpy           1.16.5 
pandas          0.24.2 
pip             19.3.1 
pyarrow         0.14.1 (0.14.0 and up suffer from this issue)
python-dateutil 2.8.1  
pytz            2019.3 
setuptools      41.6.0 
six             1.13.0 

Reporter: Yevgeni Litvin
Assignee: Joris Van den Bossche / @jorisvandenbossche

PRs and other links:

• GitHub Pull Request #5871

_{Note: This issue was originally created as ARROW-7214. Please see the migration documentation for further details.}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
[~selitvin] Thanks for the report! I can confirm this crash with latest arrow.

[asfimport]

Antoine Pitrou / @pitrou:
Issue resolved by pull request 5871
#5871