[Python] Table.to_pandas fails to convert index dtype with a custom type mapper

[j-bennet]

Describe the bug, including details regarding any error messages, version, and platform.

When providing a custom type mapper to pyarrow.Table.to_pandas, column dtypes are converted, but not index dtypes.

Example:

import pyarrow as pa
import pandas as pd

df = pd.DataFrame({"s1": ["a", "c", "b"], "i1": [1, 2, 3], "s2": ["d", "e", "f"]}).set_index('s2')
tbl = pa.Table.from_pandas(df)
mapper = {pa.string(): pd.StringDtype("pyarrow")}.get
df2 = tbl.to_pandas(types_mapper=mapper)
print(f"{df2.dtypes}")
print(f"{df2.index.dtype = }")

This prints:

s1    string[pyarrow]
i1              int64
dtype: object
df2.index.dtype = dtype('O')

Column s1 was mapped to string[pyarrow] dtype, however, the index dtype remained object. I'd expect the type mapper being used to convert the index dtype as well.

Platform: macOS
Pyarrow version: 11.0.0
Build: py310h89f3c6b_2_cpu
Channel: conda-forge

Component(s)

Python

[AlenkaF]

Yes, I think that the dtype of an index is never converted according to the types_mapper keyword when converting pa.Table or an array for that matter.

A single array gets converted to pandas series in _array_like_to_pandas with pd.Series which doesn't take into account the dtype of the index.

arrow/python/pyarrow/array.pxi

Line 1649 in 958f635

result = pandas_api.series(arr, dtype=dtype, name=name)

This would be a good add on. What would be needed is to use pandas api to reset the series index with .astype() method after the code line linked above in arrow/python/pyarrow/array.pxi.

As for the Table, it gets converted with the use of pandas BlockManager and I am not sure how the desired dtype could be passed to the BlockManager axes in that case:

arrow/python/pyarrow/table.pxi

Lines 4001 to 4008 in 7828165

	def _to_pandas(self, options, categories=None, ignore_metadata=False,
	types_mapper=None):
	from pyarrow.pandas_compat import table_to_blockmanager
	mgr = table_to_blockmanager(
	options, self, categories,
	ignore_metadata=ignore_metadata,
	types_mapper=types_mapper)
	return pandas_api.data_frame(mgr)

arrow/python/pyarrow/pandas_compat.py

Lines 797 to 823 in 45918a9

	def table_to_blockmanager(options, table, categories=None,
	ignore_metadata=False, types_mapper=None):
	from pandas.core.internals import BlockManager
	all_columns = []
	column_indexes = []
	pandas_metadata = table.schema.pandas_metadata
	if not ignore_metadata and pandas_metadata is not None:
	all_columns = pandas_metadata['columns']
	column_indexes = pandas_metadata.get('column_indexes', [])
	index_descriptors = pandas_metadata['index_columns']
	table = _add_any_metadata(table, pandas_metadata)
	table, index = _reconstruct_index(table, index_descriptors,
	all_columns)
	ext_columns_dtypes = _get_extension_dtypes(
	table, all_columns, types_mapper)
	else:
	index = _pandas_api.pd.RangeIndex(table.num_rows)
	ext_columns_dtypes = _get_extension_dtypes(table, [], types_mapper)
	_check_data_column_metadata_consistency(all_columns)
	columns = _deserialize_column_index(table, all_columns, column_indexes)
	blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
	axes = [columns, index]
	return BlockManager(blocks, axes)

[jorisvandenbossche]

Issue resolved by pull request 34445
#34445