[Python] Pandas roundtrip with object-dtype column labels with integer values: data type "integer" not understood

[asfimport]

The following will fail the roundtrip since the column indexes' pandas_type is converted from int64 to integer when an additional column is introduced and subsequently moved to the index:

 

df = pd.DataFrame(np.ones((3,1), index=[[1,2,3]])
df['foo'] = np.arange(3)
df = df.set_index('foo', append=True)
table = pyarrow.Table.from_pandas(df)
table.to_pandas()  # Errors

Reporter: Richard Wu
Assignee: Andrew Wieteska / @arw2019

PRs and other links:

• GitHub Pull Request #7822

_{Note: This issue was originally created as ARROW-9096. Please see the migration documentation for further details.}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Thanks for the report. A smaller reproducer:

df = pd.DataFrame(np.random.randn(5, 1), columns=pd.Index([1], dtype=object))   
table = pa.Table.from_pandas(df)  
table.to_pandas() 

so what triggers this is to have an object-dtype index with integers as the column labels. We try to preserve the dtype of the column labels on roundtrip (that's why we store this in the pandas metadata), but this case is clearly not covered.

Always welcome to take a look.

[asfimport]

Andrew Wieteska / @arw2019:
ARROW-3651 is related, I think

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 7822
#7822