You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The issue is due to the extension array in this example not having to_pandas_dtype method implemented. In this case ext_columns does not get populated in _get_extension_dtypes method with the name of the column with an extension type:
…orage type if to_pandas_dtype is not implemented (#34559)
### Rationale for this change
Method `to_pandas` fails with `KeyError` if a table has an extension array as a column with extension dtype not having `to_pandas_dtype` defined. In this cases we should fall back to storage type of the extension array.
### What changes are included in this PR?
Changes in `arrow_to_pandas.cc` at:
- `GetBlockType` for `ConvertTableToPandas`
- `ConvertChunkedArrayToPandas`
* Closes: #34165
Lead-authored-by: Alenka Frim <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
ArgusLi
pushed a commit
to Bit-Quill/arrow
that referenced
this issue
May 15, 2023
…the storage type if to_pandas_dtype is not implemented (apache#34559)
### Rationale for this change
Method `to_pandas` fails with `KeyError` if a table has an extension array as a column with extension dtype not having `to_pandas_dtype` defined. In this cases we should fall back to storage type of the extension array.
### What changes are included in this PR?
Changes in `arrow_to_pandas.cc` at:
- `GetBlockType` for `ConvertTableToPandas`
- `ConvertChunkedArrayToPandas`
* Closes: apache#34165
Lead-authored-by: Alenka Frim <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
rtpsw
pushed a commit
to rtpsw/arrow
that referenced
this issue
May 16, 2023
…the storage type if to_pandas_dtype is not implemented (apache#34559)
### Rationale for this change
Method `to_pandas` fails with `KeyError` if a table has an extension array as a column with extension dtype not having `to_pandas_dtype` defined. In this cases we should fall back to storage type of the extension array.
### What changes are included in this PR?
Changes in `arrow_to_pandas.cc` at:
- `GetBlockType` for `ConvertTableToPandas`
- `ConvertChunkedArrayToPandas`
* Closes: apache#34165
Lead-authored-by: Alenka Frim <frim.alenka@gmail.com>
Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Describe the bug, including details regarding any error messages, version, and platform.
When working on the extension type for tensors in PyArrow I came across a behaviour of the conversion to pandas that could be improved.
Creating an extension array (fixed shape tensor in this case) and converting it to pandas works well
But creating a table with an extension array and then converting it to pandas fails:
The issue is due to the extension array in this example not having
to_pandas_dtype
method implemented. In this caseext_columns
does not get populated in_get_extension_dtypes
method with the name of the column with an extension type:arrow/python/pyarrow/pandas_compat.py
Lines 870 to 879 in 0368e41
It would be good if it would, in case
to_pandas_dtype
method is not implemented, convert the storage arrayarrow/python/pyarrow/pandas_compat.py
Line 776 in 0368e41
arrow/python/pyarrow/array.pxi
Lines 2888 to 2889 in 925cbd8
Component(s)
Python
The text was updated successfully, but these errors were encountered: