BUG: Fix read_parquet not working with data type pyarrow list (#57411) #58156
Conversation
| df = pd.DataFrame(s, columns=["col"]) | ||
| df.to_parquet("ex.parquet") | ||
|
|
||
| result = read_parquet(path="ex.parquet", dtype_backend="pyarrow") |
There was a problem hiding this comment.
This seems off to me - so even with dtype_backend="pyarrow" we are trying to convert it to an object type? I think this should maintain the pyarrow type all the way through
There was a problem hiding this comment.
Hi William.
I do agree with you and I might be wrong.
But at least for this specific case, the object type is preserved.
Nevertheless, I will try to work on a better solution.
|
Thanks for the PR |
WillAyd
added
Dtype Conversions
| @@ -1623,6 +1623,8 @@ def pandas_dtype(dtype) -> DtypeObj: | |||
| return dtype.dtype | |||
| elif isinstance(dtype, (np.dtype, ExtensionDtype)): | |||
| return dtype | |||
| elif "list" in str(dtype) and "pyarrow" in str(dtype): | |||
There was a problem hiding this comment.
This fix is too specific and probably doesn't fix the core issue
There was a problem hiding this comment.
@mroeschke I second this. A more general fix is needed.
|
Thanks for the feedback. |
|
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen. |
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.When writing a DataFrame to a Parquet file, if the data type of the columns of the DataFrame is a PyArrow list, an error is raised when reading it from the Parquet file.
The error occurs because when reading, pandas tries to use NumPy dtypes which do not support PyArrow lists (in the function pandas_dtype in the file ./pandas/core/dtypes/common.py).
To solve this issue, we add an if statement to check if the column's data type is a PyArrow list. If so, we return the data type without converting it into a NumPy dtype (which does not support PyArrow lists).