Reading a Parquet file saved with a categorical index using Pyarrow raises an exception.

[FilippoBovo]

What happened:
Reading a Parquet file saved with a categorical index using Pyarrow raises the following exception:

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/Users/user/.virtualenvs/price-maker-us/lib/python3.8/site-packages/dask/dataframe/io/parquet/core.py", line 231, in read_parquet
    meta, statistics, parts, index = engine.read_metadata(
  File "/Users/user/.virtualenvs/price-maker-us/lib/python3.8/site-packages/dask/dataframe/io/parquet/arrow.py", line 660, in read_metadata
    meta, index_cols, categories, index = _generate_dd_meta(
  File "/Users/user/.virtualenvs/price-maker-us/lib/python3.8/site-packages/dask/dataframe/io/parquet/arrow.py", line 317, in _generate_dd_meta
    meta = clear_known_categories(meta, cols=categories)
  File "/Users/user/.virtualenvs/price-maker-us/lib/python3.8/site-packages/dask/dataframe/utils.py", line 264, in clear_known_categories
    elif not mask.loc[cols].all():
  File "/Users/user/.virtualenvs/price-maker-us/lib/python3.8/site-packages/pandas/core/indexing.py", line 879, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
  File "/Users/user/.virtualenvs/price-maker-us/lib/python3.8/site-packages/pandas/core/indexing.py", line 1099, in _getitem_axis
    return self._getitem_iterable(key, axis=axis)
  File "/Users/user/.virtualenvs/price-maker-us/lib/python3.8/site-packages/pandas/core/indexing.py", line 1037, in _getitem_iterable
    keyarr, indexer = self._get_listlike_indexer(key, axis, raise_missing=False)
  File "/Users/user/.virtualenvs/price-maker-us/lib/python3.8/site-packages/pandas/core/indexing.py", line 1254, in _get_listlike_indexer
    self._validate_read_indexer(keyarr, indexer, axis, raise_missing=raise_missing)
  File "/Users/user/.virtualenvs/price-maker-us/lib/python3.8/site-packages/pandas/core/indexing.py", line 1315, in _validate_read_indexer
    raise KeyError(
KeyError: "Passing list-likes to .loc or [] with any missing labels is no longer supported. The following labels were missing: Index(['__null_dask_index__'], dtype='object'). See https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#deprecate-loc-reindex-listlike"

What you expected to happen:

I expected Dask to read the Parquet file without the exception and return a Dask DataFrame where the index has dtype category.

Minimal Complete Verifiable Example:

import dask.dataframe as dd
import pandas as pd

ddf = dd.from_pandas(
    pd.DataFrame(data={"A": [1, 2, 3], "B": ["a", "a", "b"]}, index=["x", "y", "y"]),
    npartitions=1
)
ddf = ddf.categorize(columns=["B"])
ddf.to_parquet("file.parquet")
ddf_from_parquet = dd.read_parquet("file.parquet")

Anything else we need to know?:

I was using Pyarrow 2.0.0 and Pandas 1.1.4.

Environment:

• Dask version: 2.30.0
• Python version: Python 3.8.5 from Pyenv.
• Operating System: Mac OS Big Sur (version 11.0.1 (20B50))
• Install method (conda, pip, source): Pip

[rjzamora]

Thanks for raising @FilippoBovo! #6887 should address this.

[FilippoBovo]

Thanks, @rjzamora. Do you happen to know when your pull request incorporate in a PyPi release?

[jsignell]

@FilippoBovo the next dask release will likely be within a month or so. If you'd like to install the unreleased changes, you can pip install from the git repo: pip install git+https://github.com/dask/dask.git --no-deps