BUG: Regression for `pd.read_parquet` from pandas 1.5.0 ?

[itholic]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
pdf = pd.DataFrame({"index": [0, 1, 2], "A": [1, 2, 3]})
pdf = pdf.set_index("index", append=True)
pdf.to_parquet("pandas_parquet")
pd.read_parquet("pandas_parquet")

Issue Description

The Reproducible Example is not working with pandas 1.5.0.

Rather it raises KeyError below:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../pandas/io/parquet.py", line 501, in read_parquet
    return impl.read(
  File "/.../pandas/io/parquet.py", line 249, in read
    result = self.api.parquet.read_table(
  File "pyarrow/array.pxi", line 823, in pyarrow.lib._PandasConvertible.to_pandas
  File "pyarrow/table.pxi", line 3913, in pyarrow.lib.Table._to_pandas
  File "/.../pyarrow/pandas_compat.py", line 808, in table_to_blockmanager
    table, index = _reconstruct_index(table, index_descriptors,
  File "/.../pyarrow/pandas_compat.py", line 959, in _reconstruct_index
    result_table, index_level, index_name = _extract_index_level(
  File "/.../pyarrow/pandas_compat.py", line 997, in _extract_index_level
    logical_name = field_name_to_metadata[field_name]['name']
KeyError: 'index'

Maybe it's a regression in pandas 1.5.0 or expected behavior?

Expected Behavior

It works fine with pandas 1.4.0:

>>> import pandas as pd
>>> pdf = pd.DataFrame({"index": [0, 1, 2], "A": [1, 2, 3]})
>>> pdf = pdf.set_index("index", append=True)
>>> pdf.to_parquet("pandas_parquet")
>>> pd.read_parquet("pandas_parquet")
         A
  index
0 0      1
1 1      2
2 2      3

Installed Versions

INSTALLED VERSIONS

commit : 87cfe4e
python : 3.9.11.final.0
python-bits : 64
OS : Darwin
OS-release : 21.6.0
Version : Darwin Kernel Version 21.6.0: Wed Aug 10 14:25:27 PDT 2022; root:xnu-8020.141.5~2/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.UTF-8

pandas : 1.5.0
numpy : 1.22.3
pytz : 2021.3
dateutil : 2.8.2
setuptools : 61.2.0
pip : 22.2.2
Cython : 0.29.24
pytest : 6.2.5
hypothesis : None
sphinx : 3.0.4
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.3
IPython : 7.29.0
pandas_datareader: None
bs4 : 4.10.0
bottleneck : 1.3.4
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.5.1
numba : None
numexpr : 2.8.1
odfpy : None
openpyxl : 3.0.10
pandas_gbq : None
pyarrow : 9.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.7.2
snappy : None
sqlalchemy : 1.4.26
tables : None
tabulate : 0.8.9
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : None

[phofl]

e915b0a40a9599f46569f8f92a2eb2d3d05c02fa is the first bad commit
commit e915b0a40a9599f46569f8f92a2eb2d3d05c02fa
Author: Matthew Roeschke <emailformattr@gmail.com>
Date:   Fri Jul 8 16:58:31 2022 -0700

    ENH: RangeIndex._shallow_copy can return RangeIndex (#47557)

cc @mroeschke

[phofl]

Just some additional info: The to_parquet call is buggy, saving the file with 1.4.4 and opening it with 1.5.0 works

[debnathshoham]

Is this bug in pandas or pyarrow? 🤔

import pyarrow as pa
import pandas as pd
pdf = pd.DataFrame({"index": [0, 1, 2], "A": [1, 2, 3]})
pdf = pdf.set_index("index", append=True)

pa.Table.from_pandas(pdf).to_pandas()
Traceback (most recent call last):
  File "/Users/debnathshoham/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3378, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-4-2b79ebdcdf1b>", line 1, in <module>
    pa.Table.from_pandas(pdf).to_pandas()
  File "pyarrow/array.pxi", line 823, in pyarrow.lib._PandasConvertible.to_pandas
  File "pyarrow/table.pxi", line 3913, in pyarrow.lib.Table._to_pandas
  File "/Users/debnathshoham/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/pyarrow/pandas_compat.py", line 808, in table_to_blockmanager
    table, index = _reconstruct_index(table, index_descriptors,
  File "/Users/debnathshoham/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/pyarrow/pandas_compat.py", line 959, in _reconstruct_index
    result_table, index_level, index_name = _extract_index_level(
  File "/Users/debnathshoham/miniconda3/envs/pandas-dev/lib/python3.8/site-packages/pyarrow/pandas_compat.py", line 997, in _extract_index_level
    logical_name = field_name_to_metadata[field_name]['name']
KeyError: 'index'

[debnathshoham]

Reported Bug in arrow.